Data Engineer designing data pipelines in Python for a major railway industry client. Collaborate with Data Scientists and ensure code quality with agile methodologies.
About the role
- Seeking a Data Architect experienced in designing scalable cloud data platforms and modern data ecosystems at Reply. Collaboration with customers and engineering teams for robust data solutions is key.
Responsibilities
- Design and lead end-to-end architecture for complex cloud-based data ecosystems, including lakehouse and enterprise data platforms.
- Translate business requirements into scalable architectural designs, roadmaps, and technical specifications, including advanced scenarios, such as establishing common data models across development teams, sharing multi-tenant system data directly to customers, integrations between Databricks and external data platforms, and handling RLS from Databricks source data to analytics and reporting tools.
- Architect and implement Databricks-based solutions, including Unity Catalog, Delta Lake, Databricks SQL, Workflows, and governance frameworks. Establish data governance policies in addition to technical solutions.
- Define and enforce data modelling standards for relational, dimensional, and lakehouse structures, including common data models across global systems.
- Architect and oversee development of ETL/ELT frameworks, source-to-target mappings, and reusable transformation standards, focusing on meta-data solutions.
- Establish best practices for data ingestion, curation, cataloging, lineage, quality, and MDM across the data ecosystem. Establish MDM solutions, preferably with Profisee.
- Partner with cross-functional engineering teams to ensure architectural consistency, performance optimization, and security compliance.
- Mentor and lead junior engineers, contributing to technical direction, design reviews, and architectural decision-making.
- Develop cloud-native reference architectures leveraging Azure Data Factory, Azure SQL, Synapse Analytics, Azure Data Lake, Stream Analytics, and other modern Azure services.
- Collaborate with executive and architect stakeholders to define data governance standards, taxonomy structures, and metadata strategies. Explain and defend architecture decisions to customers.
Requirements
- Bachelor’s Degree in Computer Science, Engineering, MIS, or a related field.
- 12+ years of experience in total along with strong data engineering or data platform development background and with at least 3+ years in data architecture roles.
- 3+ years of experience with Databricks, including Unity Catalog, Delta Lake, Databricks SQL, and Workflow orchestration.
- Strong proficiency with Python, Apache Spark, and distributed data processing frameworks.
- Advanced SQL expertise, including performance tuning, indexing, and optimization for large datasets.
- Proven experience designing and implementing lakehouse architectures and cloud data ecosystems.
- Hands-on experience with Azure Data Services: ADF, ADLS, Azure SQL, Synapse Analytics, Stream Analytics or Fabric equivalents.
- Strong understanding of data modelling principles (3NF, dimensional modelling, Kimball, Inmon) and enterprise data warehouse concepts.
- Prior consulting experience delivering analytics or data platform solutions to enterprise clients.
- Familiarity with CI/CD pipelines and IaC tools (Terraform, ARM, Bicep).
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior Data Engineer responsible for building and optimizing data pipelines for banking analytics initiatives. Collaborating with data teams to ensure data quality and readiness for enterprise use.
Senior Data Engineer developing scalable data solutions on Databricks for analytics and operational workloads. Collaborating with cross - functional teams to modernize the data ecosystem.
Data Engineer focused on analytics and data pipeline development for network optimisation. Collaborating with teams to deliver high - quality data solutions with Python and SQL.
Senior Product Manager defining platform capabilities for Data Cloud in Salesforce. Collaborating with R&D teams while shaping product strategy for Data 360 integration.
Senior Data Engineer at Goodwin enhancing data platforms and fostering data - driven culture across teams. Collaborating with IT and Finance on technology solutions and data governance practices.
Director, Data Platform Design and Strategy at MedImpact leading data platform and AI innovations to enhance healthcare services. Overseeing enterprise projects and managing teams to meet strategic goals.
Data Engineer delivering AI - and data - driven solutions for Honeywell’s industrial customers. Architecting and implementing scalable data pipelines and platforms focused on IoT and real - time data processing.
Data Engineering Associate focusing on data quality control and management for distribution platform. Collaborates on large scale data projects to ensure data accuracy and availability for users.
Data Architect managing enterprise data platform built on Microsoft Fabric at Johnstone Supply. Leading architectural standards and collaborating with business and IT leaders for strategic data - driven insights.