Data Engineer role specializing in Azure & Snowflake at InfoCentric. Leading design and delivery of enterprise - scale data platforms for large organizations.
About the role
- Lead Azure Data Engineer designing and optimizing data ecosystems on Microsoft Cloud. Responsible for building scalable data platforms and pipelines for analytics and reporting.
Responsibilities
- Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
- Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines.
- Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data.
- Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.
- Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics.
- Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments.
- Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.
- Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders.
- Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution).
- Oversee governance, access controls, metadata management, and lineage using Unity Catalog.
- Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality.
- Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions.
- Partner with stakeholders to define data strategy, standards, and architectural roadmaps.
- Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance.
- Implement secure, compliant data access and lineage frameworks across cloud data platforms.
- Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows.
Requirements
- 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric.
- Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming.
- Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage.
- Solid knowledge of Unity Catalog for data governance, security, and access management.
- Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning.
- Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions.
- Excellent leadership, cross-team collaboration, and communication skills.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Principal Data Architect at PointClickCare ensuring coherent and scalable data architecture. Driving unified data direction while collaborating with Engineering Architecture team for AI enablement.
Data Engineer Tech Lead developing data solutions at Carelon. Leading a cross - functional team to optimize data workflows and maintain data integrity.
Lead Data Engineer responsible for evolving Manna’s data infrastructure for drone delivery. Overseeing data architecture and analytics while building scalable data pipelines.
Data Engineer designing, implementing, and optimizing data pipelines for DeepLight AI. Collaborating closely with a multidisciplinary team to analyze large - scale data.
Data Engineer designing and maintaining scalable ETL pipelines at Satori Analytics. Collaborating with teams to deliver high - quality analytics solutions across various industries.
Data Architect responsible for defining enterprise data architecture on AWS and Databricks Lakehouse platforms. Enabling scalable data lakes and enterprise analytics for financial services organizations.
Data Platform Operations Support leading data engineering strategy across projects for EXL. Driving innovation and optimization while collaborating with various teams in the organization.
Manager II leading data engineering projects at Navy Federal Credit Union. Overseeing data governance and quality initiatives while managing engineering teams in a hybrid work environment.
Principal Data Engineer responsible for architecting scalable data pipelines and building high - quality data foundations. Collaborating closely with experts to ensure data readiness for advanced analytics.