Software Developer in Test working on cloud - based data platform at Tecsys. Ensuring quality and reliability of data pipelines and transformations using automation frameworks.
CS
Hybrid Senior Cloud Data Engineer
Posted 2 weeks ago
About the role
- Senior Cloud Data Engineer responsible for operating and optimizing cloud-based data environments. Collaborating with analytics teams and specializing in AWS, Databricks, and Spark technologies.
Responsibilities
- Responsible for operating and optimizing our cloud-based data processing environment.
- Work with Databricks, AWS services, Spark, Unity Catalog, and Delta Lake to ensure efficient, secure, and reliable data pipelines and analytics workloads.
- Refines data transformations using PySpark and Spark SQL within notebooks.
- Leverages orchestration tools like Apache Airflow to automate data workflows.
- Participates in code reviews, testing, and documentation.
- Supports and troubleshoots Databricks jobs, Spark workloads, and AWS-based data processes
- Optimizes Databricks clusters and jobs for performance and cost.
- Works closely with data engineering and analytics teams to improve data quality.
Requirements
- Bachelor’s degree in Computer Science, Information Systems, Data Engineering, or similar
- Master’s Degree will be considered as an asset
- 5+ years of experience in big data operations or cloud-based data engineering.
- Strong hands-on experience with AWS, Databricks, Delta Lake, and Apache Spark
- Proficient in Python, SQL, and PySpark
- Experience with CI/CD, version control, and release processes (AWS CodePipeline, Git)
- Experience with monitoring, debugging, and optimizing ETL/ELT and Spark workloads
- Knowledge of data governance frameworks and exposure to enterprise security or regulated environments will be considered as an asset
- **Competencies**
- Excellent problem-solving skills and attention to detail
- Strong communication skills and the ability to work collaboratively in a team environment
- Effective time management with ability to multi-task and prioritize work
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer responsible for designing, building, and optimizing data pipelines and architectures in a tech environment. Requires extensive experience with modern data warehousing and cloud platforms.
Lead Data Engineer role at Brillio focusing on AI & Data Engineering with expertise in Azure and MS Fabric. Collaborate within the Data Engineering team in Pune, Maharashtra, India.
Data Architect at Whiteshield designing scalable, secure data architectures for national and enterprise transformation programs. Architecting modern data platforms to support analytics, AI and operational use cases.
Data Engineer managing scalable data ecosystems for actionable business intelligence and cross - functional stakeholder collaboration. Optimizing ETL/ELT pipelines and ensuring data integrity and security.
Data Engineer specializing in data architecture and solutions for a banking environment, driving value for customers through innovative engineering practices and technologies in data management.
Technical Lead for data engineering and reporting in healthcare technology at Dedalus. Shaping innovative software solutions and leading cross - functional technical teams in Australia.
Senior ML Data Engineer working on data pipeline curation for Mobileye's autonomous vehicle dataset. Collaborating across teams to enhance ML engineering and vision model applications.
Data Engineer managing customer datasets to enhance industrial research and development. Responsible for ETL pipelines and data ingestion for the Uncountable Web Platform.
Data Engineer designing and maintaining scalable data solutions on Databricks for clinical trials. Collaborating with teams to overcome data challenges and ensure the smooth logistics of clinical supplies.