Senior Data Engineer designing and maintaining data processing pipelines for analytics and machine learning in a fast-paced startup. Collaborating with cross-functional teams to ensure data accuracy and security.
Responsibilities
Design, develop, and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark, PySpark, and Apache Beam.
Build and maintain microservices in Python that serve data-driven features in production.
Develop internal tools to support CI/CD pipelines, experiment tracking, and data versioning.
Collect, process, and integrate large datasets from multiple sources, including databases, file systems, and APIs.
Ensure data integrity, consistency, and quality through robust validation and monitoring processes.
Optimize data systems for performance, scalability, and high availability.
Implement best practices for data security, access control, and privacy.
Collaborate with data scientists, analysts, and engineers to support analytics and ML workflows.
Requirements
5+ years of professional experience in software engineering or data engineering.
Strong software engineering skills with Python in large-scale, high-performance production environments.
Hands-on experience with Spark/PySpark and other big data frameworks.
Expertise in data modeling and working with both structured and unstructured data.
Hands-on experience with streaming data platforms, particularly Apache Kafka.
Strong understanding of distributed systems and modern data architectures.
Experience working with cloud platforms, preferably GCP (BigQuery, Dataflow, Pub/Sub, Dataproc).
Excellent problem-solving and communication skills.
Benefits
Office Snacks and Activities: Fuel your work with various snacks and enjoy fun activities that keep our team spirit high. Whether it's a darts match, board games, or yoga, we believe a happy team is productive.
Senior Manager - Data Architect leading enterprise - level data architectures and cloud data platforms. Working in a hybrid consulting environment focused on AI - driven decision - making in Switzerland.
Data Engineer developing tools and analytical capabilities for tracking commodity flows within the Dry Bulk market. Lead data interventions and engage with cross - functional teams for efficient cargo data management.
BI Data Engineer supporting analytics and decision - making for Kpler's products. Responsible for building scalable pipelines and robust data models in a dynamic market landscape.
Senior Data Engineer at Clorox designing and maintaining data pipelines and solutions on cloud platforms. Collaborating with cross - functional teams to support data - driven business decisions.
Data Engineering & Warehousing Manager leading the design and development of enterprise data pipelines. Collaborating on data governance standards and ensuring scalable data solutions for Hastings Insurance.
Senior Data Engineer at Air Methods leading data - driven solutions and mentoring team members. Responsible for designing and improving data architecture and analytics to create impactful business insights.
Data Engineer III developing high - performance data solutions for Walmart Global Tech. Collaborating with teams to build scalable data pipelines and ensure data governance.
Data Engineer optimizing and maintaining data architecture for fintech solutions in Latin America. Involved in data governance, pipeline development, and cross - team collaboration for tech innovation.
DataOps Engineer at Eeze focusing on data pipeline stability across multiple products. Collaborating with IT teams to maintain quality, observability, and operational efficiency.
Data Engineer developing and enhancing data pipelines and models at ERNI Schweiz. Required skills include SQL and Python with opportunities for remote work in Europe.