Data Solutions Architect leading business intelligence solutions and analytics at Crowe. Overseeing data pipelines and analytics frameworks to drive decision - making and compliance.
About the role
- AWS Glue Data Engineer at DeepLight AI responsible for data ingestion and pipeline performance optimisation. Collaborate with teams to build scalable solutions in a hybrid work environment.
Responsibilities
- ***Your responsibilities as the AWS Glue Data Engineer will include:***
- **Data Ingestion Development**
- Building and implementing AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
- Implementing correct loading methods based on source requirements (CDC, full load, delta, snapshot).
- Designing and executing historical loading mechanisms to bring legacy data into the Lakehouse.
- **Performance Optimisation**
- Optimising Glue job performance (DPU allocation, parallelization, partitioning) according to best practices.
- Collaborating with platform teams to ensure tooling and optimization alignment.
- **Migration & Automation**
- Aggressively migrating source tables to Bronze layer, initially using manual approaches with standards/templates, later leveraging AI-enabled acceleration.
- Ensuring jobs are version-controlled and production deployment is automated via Git and Terraform.
- **Governance & Monitoring**
- Implementing source system connectivity into CDP in collaboration with source system owners.
- Ensuring jobs comply with data contracts and are properly monitored.
- Preparing documentation and handover to operational support teams.
- **Collaboration**
- Working closely with Data Architect for ingestion patterns and standards.
- Coordinating with Data Assurance Lead to apply quality checks across all jobs.
- Partnering with platform engineers for tooling and optimisation.
Requirements
- ***You will have experience in:***
- AWS Glue, PySpark, and ETL pipeline development;
- substantial knowledge of Lakehouse architecture and Medallion design principles;
- familiarity with CDC, delta loads, and historical data ingestion strategies; and;
- 5+ years experience in data engineering roles, with hands-on experience in AWS Glue.
- ***You should also have knowledge of:***
- AWS services: Glue, S3, Athena, Lambda;
- Git, Terraform for CI/CD automation;
- data quality frameworks (e.g., Soda Core);
- identifying ways to automate their work / repetitive tasks;
- working in a fast-paced environment and deliver aggressive migration targets;
- collaborating and communication with different stakeholder levels; and;
- working with Jira and agile way of working.
Benefits
- **Benefits & Growth Opportunities:**
- · Competitive salary and performance bonuses
- · Comprehensive health insurance
- · Professional development and certification support
- · Opportunity to work on cutting-edge AI projects
- · Flexible working arrangements
- · Career advancement opportunities in a rapidly growing AI company
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Senior Lead Data Engineer designing and building scalable data solutions utilizing AI technology for a globally recognized financial institution. Serving sophisticated clients across the globe.
Data Engineer Consultant building and maintaining enriched data infrastructure for analytical thinking at Northwest Permanente. Involves data collection, cleansing, and transformation for business intelligence.
Vice President - Business, Data Architect role at TD Securities focusing on business data architecture and analytics capabilities. Collaborate with stakeholders to define and govern data models and ensure alignment with strategy.
Staff Data Engineer at Headspace building privacy - first data platforms for mental health support. Leading data engineering strategies and mentoring team members to enhance data - driven decision making.
Senior Data Engineer building and implementing data pipelines at Headspace. Collaborating with analytics and data science teams to enhance personalized mental health support.
Data Engineering Intern working on data pipelines and infrastructure in fast - growing fintech. Collaborating with data engineers, learning best practices and developing data solutions.
Senior Software Engineer building and maintaining data infrastructure for Gusto. Collaborating with Data Science and Business Intelligence teams to achieve their goals.
Data Engineer building and maintaining scalable data pipelines for AI Search Infrastructure at You.com. Collaborating across teams to ensure data quality and enable AI capabilities.
Data Engineer developing and managing technology - based data solutions for clients in different industries in Greece. Participating in software development lifecycle within Agile team setting.