Senior Data Engineer developing and maintaining data pipelines for clients in an Agile setting. Collaborating with teams to enhance data quality and mentoring junior engineers.
About the role
- Data Engineer at Logcomex, developing robust data pipelines and ensuring data quality for trade solutions. Collaborating with various teams to optimize data governance and performance.
Responsibilities
- Develop and maintain robust, scalable, and testable data pipelines using PySpark and Airflow.
- Design and implement data ingestion and transformation processes to populate the data lake using a layered architecture (Bronze, Silver, Gold).
- Work on data quality control, documentation, and lineage management using OpenMetadata.
- Collaborate with product and capture squads to ensure data consistency and coverage.
- Ensure data governance, versioning, and auditing of production pipelines.
- Optimize ETL/ELT routines and query performance in relational databases, data warehouses, and engines such as Elasticsearch and Athena.
Requirements
- We are looking for people with knowledge in...
- PySpark
- Apache Airflow
- AWS S3, Glue, Athena, EC2
- SQL (Athena, PostgreSQL)
- Elasticsearch/OpenSearch
- Docker
- Pandas
- Jupyter
- Unix (Linux), Bash
- DBT
- Nice to have:
- Glue
- Delta Lake
- Kubernetes
- NoSQL
- Elasticsearch
- Airbyte
Benefits
- Swile benefits card with a fixed monthly value of R$2,540.00 (food, mobility, multi-balance, and home office allowance);
- National health plan (Unimed or Amil);
- Dental plan (Odontoprev);
- Life insurance (MetLife);
- TotalPass;
- Starbem (health services platform for your physical, mental, and emotional well-being);
- Pharmacy discount program with Panvel;
- Extended maternity and paternity leave through the Empresa Cidadã program;
- Subsidy for professional development in partnership with Unico Skill, offering various options for undergraduate, postgraduate, language courses, mentorships, etc.;
- Private English lessons for leadership and specialists level II and above;
- School/Education assistance;
- Fresh fruit, cookies, coffee, tea, and energy drinks available at any time;
- Celebrations, integration events, and team building activities;
- Partnership with KÜK Station to provide the best for our Loggers on office days;
- In-company massage;
- Birthday day-off;
- Birthday gift;
- Service anniversary gifts;
- Recruta Loggers (employee referral program with bonuses).
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Lead Data Engineer overseeing the migration from Redshift to AWS RDS and mentoring data analysts in Python and DataOps practices.
Senior Data Engineer designing and maintaining scalable data pipelines using modern technologies. Collaborating with cross - functional teams and providing mentorship in a dynamic environment.
Data Architect leading design and implementation of cloud data platforms for digital transformation. Collaborating with stakeholders to define data strategies and governance models.
Data Engineer Consultant designing and optimizing data infrastructure for clients' business needs. Working with SQL and data visualization tools in a mainly remote role with some onsite responsibilities in Denver.
Data Engineer creating Real - Time Data Processing applications for a leading iGaming operator. Work involves stream data manipulation and collaboration in an Agile environment.
Data Engineer at Voodoo optimizing real - time data pipelines for gaming and consumer apps to support growth. Joining a top - tier data team dedicated to monetizing via advertising partners in a competitive landscape.
Cloud Data Engineer designing data architectures for cloud platforms at fifty - five. Collaborating with local and global teams to optimize marketing ROI and customer experience.
SAP Specialist responsible for designing, developing, and executing data migration objects in Hydro’s SAPEX program. Ensuring successful ETL processes and maintaining data quality.
Senior Data Engineer building scalable data pipelines and data models within retail at Avaron. Collaborating closely with business and technical teams to ensure reliable data solutions.