Senior Data Engineer designing and optimizing data platforms for clients using Microsoft Azure, Microsoft Fabric, Power BI, and Databricks. Working closely with clients to deliver scalable solutions.
About the role
- Data Engineer optimizing data ingestion and transformation pipelines for seamless data flow. Collaborating with cross-functional teams using Databricks and other cloud services in a hybrid work setting.
Responsibilities
- Ingest data from a variety of sources such as Azure SQL DB, Google Analytics, Google Play Store, Apple App Store, Salesforce, and others.
- Develop and optimize ETL/ELT pipelines to transform data from CSV, JSON, SQL tables, and APIs into usable formats.
- Work with REST APIs to pull data from various external sources and integrate it into our data ecosystem.
- Design and implement efficient data transformation processes to cleanse, aggregate, and enrich data.
- Apply industry best practices for data modeling to ensure scalability, performance, and data integrity.
- Collaborate with data analysts and data scientists to provide clean, high-quality datasets for reporting and analysis.
- Utilize Databricks for data processing, transformation, and orchestration tasks.
- Manage and optimize Databricks clusters for performance, reliability, and cost-effectiveness.
- Implement Databricks workflows to automate and streamline data pipelines.
- Use Unity Catalog for data governance and metadata management, ensuring compliance and data access control.
Requirements
- 5+ years of hands-on experience in data engineering or a related field.
- Proven experience with Databricks and Databricks workflows, including cluster management and data pipeline orchestration.
- Strong experience in data ingestion from SQL databases (Azure SQL DB), APIs (Google Analytics, Google Play Store, Apple App Store, Salesforce), and file-based sources (CSV, JSON).
- Proficiency in SQL for data manipulation and transformation.
- Experience with Python or Scala for writing and managing data workflows.
- Working knowledge of REST APIs for data integration.
- Experience in data transformation using Apache Spark, Delta Lake, or similar technologies.
- Knowledge of cloud platforms such as Azure, with a focus on Azure SQL DB.
- Familiarity with Unity Catalog for metadata management and governance.
- Understanding of data architecture, data pipelines, and the ETL/ELT process.
- Experience in data modeling, optimizing queries, and working with large datasets.
- Familiar with data governance, metadata management, and data access controls.
- Knowledge of Apache Kafka or other real-time streaming technologies (optional).
- Experience with Data Lake or Data Warehouse technologies (optional).
- Familiarity with additional data transformation tools such as Apache Airflow or dbt (optional).
- Understanding of machine learning workflows and data pipelines (optional).
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer providing technical expertise on mission - critical NAVSUP OIS program. Work involves data architecture and database management in AWS GovCloud environments.
Senior Data Engineer focusing on data infrastructure for an AI - driven insurtech startup based in Nepal. Collaborating with teams to optimize data models and maintain data quality.
Senior Professional Consultant leading architecture and design for SAP BW and SAC solutions at Freudenberg. Collaborating with stakeholders and optimizing performance of data landscapes.
Senior Data Engineer designing and managing data architectures to transform large - scale data into insights for Humana. Involves leading technical discussions and implementing best data practices.
Data Engineer II at Early Warning Services developing data science tools and infrastructure. Collaborating on software enhancements and mentoring interns in a hybrid work environment.
ウーブン・バイ・トヨタのデータプラットフォーム実現に情熱を持つエンジニアを募集中。スケーラブルなデータアーキテクチャの設計・実装を担当します。
Senior Data Architect responsible for optimizing data architecture and supporting data - driven business decisions at TruStage. Leading technical guidance for data architecture and cross - functional team collaboration.
Senior Data Architect developing data architecture plans at The Hartford, collaborating with internal teams to align data standards and practices. Leading complex solutions with a focus on operational effectiveness.
Senior Solution Architect defining architecture framework for SA‑CCR in regulatory risk. Collaborating with stakeholders to ensure compliance and efficient data governance.