Director of Engineering overseeing development of AI - driven data platforms at LVT. Leading teams to transform sensor data into actionable insights using modern architecture and technologies.
About the role
- Data Engineer developing scalable data lake solutions and optimizing data pipelines at U.S. Bank. Collaborating with teams to manage data governance and cloud migration activities.
Responsibilities
- Design and implement scalable data lake solutions using Snowflake and Databricks
- Develop and optimize data pipelines for ingestion, transformation, and storage
- Manage data governance, quality, and security across cloud environments and implement performance tuning, automation, and CI/CD for data workflows
- Collaborate with cross-functional teams to support cloud migration activities
- Tune Hadoop, Hive, and Spark jobs and configurations for optimal performance, efficiency, and resource utilization
- Diagnose and resolve issues related to Linux servers, networks, cluster health, job failures, and performance bottlenecks
- Provide on-call support and collaborate with other teams to ensure smooth operations
- Implement and manage security measures within the Cloudera environment, including Kerberos, Apache Ranger, and Atlas, to ensure data governance and compliance
- Setup and manage HashiCorp Vault for secure keys and secrets management
- Migrate Datastage ETL jobs to Azure cloud services such as Azure Synapse Analytics, Azure Databricks, or Snowflake
- Develop scripts (e.g., shell, Ansible, Python) for automating administrative tasks, deployments, and monitoring
- Create and maintain documentation for system configurations, operational procedures, and troubleshooting knowledge bases
- Work closely with the vendor to stay current with the latest releases, perform upgrades, and address vulnerabilities
Requirements
- Bachelor’s degree, or equivalent work experience
- Three to five years of relevant experience
- Deep expertise in Data Engineering and Management technologies, synthetic data, automation, advanced analytics
- Ability to do on-call rotation once a month
- Very strong customer-centric focus
- 6 - 8 years of hands-on experience in Data engineering, Cloud platform management, and performance optimization
- Very strong Azure Data Factory tools experience
- Excellent SQL Experience, including performance tuning and optimization
- Hands-on experience with Hadoop, Hive, Spark, and migration of Big Data into Azure cloud services
- DataStage experience for conversion of ETL jobs to Pyspark ETL pipelines
- Working with offshore teams
- Working knowledge and hands-on experience in Data Integration and Data Lake Architectures with Databricks and Snowflake platforms
- Working knowledge of Microsoft Azure cloud and big data migration to cloud platforms
- Proficiency in Linux, clustering, and distributed systems
- Expertise in Hive and Spark for data processing and analytics
- Expertise in Hadoop ecosystem components such as HDFS, YARN, Hive, Spark, and Sqoop
- Proficiency in languages such as shell, Python, Pyspark for automating workflows, deployments, and monitoring
- Expertise in Linux, Network, Python scripting, DNS, Kerberos, LDAP/AD, JupyterHub
- Experience in creating and maintaining documentation for system configurations, operational procedures, and troubleshooting knowledge bases
- Strong problem-solving skills and the ability to diagnose and resolve system failures and performance bottlenecks
- Excellent communication and collaboration skills to work effectively with cross-functional teams
Benefits
- Healthcare (medical, dental, vision)
- Basic term and optional term life insurance
- Short-term and long-term disability
- Pregnancy disability and parental leave
- 401(k) and employer-funded retirement plan
- Paid vacation (from two to five weeks depending on salary grade and tenure)
- Up to 11 paid holiday opportunities
- Adoption assistance
- Sick and Safe Leave accruals of one hour for every 30 worked, up to 80 hours per calendar year unless otherwise provided by law
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Senior Data Engineer at Independence Pet Holdings shaping data ecosystem by building platforms and pipelines. Collaborating with teams to enhance data analytics and operational insights.
Senior Data Engineer designing and developing scalable data pipelines for fintech company. Collaborating with stakeholders to ensure analytics - ready data formats and supporting batch and streaming processes.
Senior Data Engineer at Vancity designing, building, and optimizing scalable data pipelines. Collaborating closely with analytics and business teams to deliver trusted data products while ensuring high standards of data quality.
Principal Software Engineer shaping and evolving Mastercard’s global enterprise data architecture. Leading modernization of data platforms across hybrid cloud and on - premises environments.
Senior Data Engineer leading Snowflake implementation and defining modern data architecture. Mentoring internal team while working on strategic projects.
Data Engineer at Booz Allen creating advanced technology solutions and managing data engineering activities for mission - driven projects. Collaborating with a multi - disciplinary team in a fast - paced environment.
Lead Platform Data Engineer focusing on data architecture and integration at Allegion, enhancing security solutions. Collaborate on data strategy and mentor engineering teams for data standardization and quality.
Data Engineer building and optimising the Azure - based data platform at Castle Trust Bank. Collaborating to deliver scalable, reliable solutions empowering decision - making across the Bank.
Lead Data Engineer at Castle Trust Bank, owning the Azure - Databricks platform and SQL infrastructure. Delivering scalable and reliable solutions that drive strategic goals across the organization.