Senior Data Engineer focusing on Databricks in a market - leading company. Designing data architectures and optimizing data workflows in a collaborative environment.
About the role
- Data Engineer I developing data services with Azure technology for global risk management insights. Collaborating with teams to optimize data processes and ensure quality standards.
Responsibilities
- Design and implement data pipelines to efficiently extract, transform, and load (ETL) data from various sources into a central data lake using Azure Data Factory and Azure Databricks
- Transform and process data using Python, PySpark, and SQL to create valuable insights
- Collaborate with peers and senior team members to optimize data processes and solve complex challenges
- Ensure data quality by performing thorough checks and troubleshooting issues
- Apply best practices in data engineering for reliability and performance
- Optimize pipelines for maximum efficiency and cost-effectiveness
- Document comprehensively and maintain data mapping across multiple systems
- Provide hands-on troubleshooting and consulting for data challenges when needed
- Perform basic unit testing to maintain data quality standards
Requirements
- Bachelor's or master's degree in computer science, information technology, or a related field
- Approximately 2-3 years of experience in a data-related role, preferably data engineering
- Hands-on expertise with programming languages such as Python, PySpark, and SQL
- Working knowledge of data processing frameworks or ETL tools like Azure Data Factory and Azure Databricks
- Experience with Azure DevOps or similar CI/CD tools
- Experience in agile software engineering methodologies (nice to have)
- Strong analytical and problem-solving skills to tackle complex data challenges (nice to have)
- Excellent interpersonal and communication skills, both written and verbal (nice to have)
- Meticulous attention to detail and commitment to data accuracy (nice to have)
- Knowledge of AI tools like CoPilot, ChatGPT, etc. (nice to have)
- Eagerness to learn and collaborate with team members (nice to have)
Benefits
- hybrid work model where expectation is to be in office at least three days per week
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior Data Architect Lead at Leidos developing enterprise data and analytics solutions for the Department of War. Collaborating with teams to implement data strategies and governance frameworks.
Senior Data Engineer Lead at Leidos, enhancing enterprise data solutions for DoD organizations. Collaborating with teams to deliver scalable data analytics and AI capabilities.
Data Engineer building advanced technology solutions for clients. Organizing and making disparate data meaningful to impact missions in fraud detection, cancer research, and national intelligence.
Staff Data Engineer on Real World Evidence team driving large - scale data initiatives. Collaborating with cross - functional teams to optimize data pipelines and improve healthcare outcomes.
Data Engineer Trainee developing expertise in Python, SQL, and modern cloud technologies. Innovating in a dynamic team environment with a focus on practical application.
AWS Data Engineer designing and maintaining scalable cloud - based data platforms within Banking and Financial Services at EXL. Collaborating across teams to enable data - driven decision - making through cloud technologies.
Senior Data Engineer designing and scaling advanced data platforms for 5G and future 6G innovations at Nokia. Building robust data pipelines while collaborating with Data Scientists and Engineers in a hybrid setting.
Senior Data Engineering Consultant at Gradion transforming data infrastructure for global clients. Modernizing legacy systems and operationalizing AI/ML solutions with cloud and AI consultants.
Data Engineer ETL responsible for AI model data cleaning and development. Leading projects in a hybrid work environment in Singapore.