Seeking a Senior/Lead Platform Engineer responsible for architecting and implementing scalable data and ML platforms. Focusing on AWS and Databricks, while leading DevSecOps practices.
Responsibilities
Architect and implement end-to-end data and ML platforms: data lakes, warehouses, streaming and batch pipelines, model training/deployment infrastructure, on AWS + Databricks.
Lead DevSecOps and DataOps practices: infrastructure as code (IaC), CI/CD pipelines for data & ML workflows, secure multi-account/multi-region cloud operations.
Integrate AWS services (e.g., S3, Redshift, Kinesis, Lambda, EKS/ECS) with Databricks runtime, Delta Lake, Unity Catalog etc to build scalable, performant pipelines.
Build and operate ML infrastructure: training clusters, model versioning, MLOps toolchain (e.g., MLflow), model monitoring and observability, automatic retraining workflows.
Establish data governance, lineage, quality, observability standards across data pipelines and ML workflows.
Mentor engineering teams, define architectural best practices and guide implementation of high-scale data/ML systems.
Optimize system performance, cost and scalability; diagnose and resolve large-scale production issues.
Continuously evaluate new tools and technologies in the areas of cloud, data platform, DevSecOps, ML infrastructure and apply them to drive innovation.
Requirements
7+ years of experience in data platform architecture, cloud/ML infrastructure engineering or related roles.
Deep technical expertise in **Databricks and AWS**: demonstrated ability to design, integrate and operate solutions spanning both platforms.
Strong hands-on implementation skills: you will not just design but build, deploy and operate the platform.
Proven track record of building and operating scalable ML/AI platforms in production (model training & deployment).
Expertise in Apache Spark, Delta Lake, modern data pipeline frameworks (batch + streaming).
Strong background in infrastructure as code (Terraform, CloudFormation), CI/CD for data/ML, and DevSecOps practices.
Proficiency in Python and SQL; familiarity with Scala or equivalent is a plus.
Experience with data governance, data lineage, observability and MLOps frameworks (e.g., MLflow, Airflow, dbt).
Bonus: Experience in fintech, regulated industries or high-security environments.
Smarsh seeks a Platform Engineer I to design AWS cloud infrastructure for digital communications risk management. Collaborate with teams on infrastructure and customer onboarding efforts.
Software engineer at Uncountable focusing on Generative AI deployment in software. Building AI - powered search tools and developing LLM stack for scientific research.
Lead Platform Engineer at TD Securities, developing a high performing Trading Risk Warehouse platform. Responsible for ensuring stability and scalability, while managing underlying infrastructure and supporting development teams.
Lead Platform Engineer at Capital One driving transformation in technology and solutions with Agile practices and DevOps tools. Collaborating on complex technical problems in a fast - paced environment.
Data Platform Engineer managing daily operations of data platforms for a global cybersecurity company. Collaborating with teams to ensure platform reliability and performance.
Senior Platform Engineer focused on building internal platform capabilities for developer tooling and experience at MONY Group. Collaborating with teams to enhance platform engineering and software delivery.
Databricks Platform Engineer working on AWS ecosystem design, build, and optimization. Responsible for implementing scalable pipeline solutions across data platforms.
Senior Data & Platform Support Engineer supporting Oracle databases at the Federal Reserve Bank. Collaborating with teams to ensure operability of payment systems and enhance business outcomes.