Manager of Data Engineering developing and optimizing data platforms for enterprise analytics. Leading a team and collaborating with business stakeholders for high-quality data solutions.
Responsibilities
Lead, mentor, and grow a team of Data Engineers, including recruitment, onboarding, performance management, and continuous skill development.
Oversee the design, development, and optimization of data pipelines, architectures, and data sets for batch and streaming use cases.
Collaborate with business units and analysts to gather requirements and translate them into scalable data solutions.
Drive best practices in data modeling (conceptual, logical, and physical) to support analytics and reporting.
Establish and maintain metadata standards, data cataloging, and lineage to improve data discovery and governance.
Ensure timely delivery of projects by managing priorities, scope, and resources in an Agile framework.
Partner with vendors and managed service providers as required to support delivery and operations.
Define, track, and report on team KPIs such as data pipeline uptime, data freshness, and adoption metrics.
Contribute to capacity planning, cloud cost optimization, and vendor performance review.
Promote standardization and reusability of data models, pipelines, and integration approaches.
Serve as a subject matter expert, communicating technical solutions to both technical and non-technical audiences.
Maintain compliance with organizational standards for security, architecture, and development practices.
Encourage experimentation with automation, AI/ML integration, and modern data engineering practices.
Provide input into enterprise architecture and align data initiatives.
Requirements
Bachelor’s degree in computer science, Engineering, or related technical discipline.
5+ years of experience in data engineering, including 2–3 years in a leadership or managerial role.
Proficiency in Python, SQL, and Spark (batch and streaming).
Experience developing and maintaining data lakes and cloud-based platforms (preferably Azure/Databricks).
Strong background in data architecture, including data modeling (conceptual, logical, dimensional) and relational databases.
Familiarity with data mesh or federated architectures for distributed data ownership.
Demonstrated ability to quantify business impact of data initiatives (e.g., cost savings, reduced downtime, increased adoption).
Knowledge of data governance, data quality, and master data management practices.
Proven ability to mentor and develop technical talent.
Hands-on experience with Agile/Scrum methodologies.
Excellent communication, problem-solving, and collaboration skills with a strong delivery-focused mindset.
Knowledge of orchestration tools (SSIS, ADF, DBT or equivalent).
Familiarity with DevOps/DataOps practices—CI/CD, version control, automated testing for data pipelines.
Exposure to streaming technologies like Kafka/Event Hubs beyond Spark Streaming.
Understanding of security and compliance frameworks (PII, GDPR, SOC2, HIPAA depending on industry).
Technical Lead for data engineering and reporting in healthcare technology at Dedalus. Shaping innovative software solutions and leading cross - functional technical teams in Australia.
Senior ML Data Engineer working on data pipeline curation for Mobileye's autonomous vehicle dataset. Collaborating across teams to enhance ML engineering and vision model applications.
Data Engineer managing customer datasets to enhance industrial research and development. Responsible for ETL pipelines and data ingestion for the Uncountable Web Platform.
Data Engineer designing and maintaining scalable data solutions on Databricks for clinical trials. Collaborating with teams to overcome data challenges and ensure the smooth logistics of clinical supplies.
Senior Manager leading a team of database engineers to manage CCC's data platform. Overseeing mission - critical applications and collaborating with cross - functional teams in a hybrid environment.
As a Principal Data Architect at Solstice, lead the design and implementation of data architecture solutions. Ensure data integrity, security, and accessibility to meet strategic organizational goals.
Data Platform Specialist overseeing data workflows and enhancing data quality for Stackgini's AI - driven IT solutions. Collaborating with teams to drive improvements and stakeholder support.
Data Engineer designing data pipelines in Python for a major railway industry client. Collaborate with Data Scientists and ensure code quality with agile methodologies.
Senior Data Engineer responsible for building and optimizing data pipelines for banking analytics initiatives. Collaborating with data teams to ensure data quality and readiness for enterprise use.
Senior Data Engineer developing scalable data solutions on Databricks for analytics and operational workloads. Collaborating with cross - functional teams to modernize the data ecosystem.