Data Engineer at Studyportals responsible for data pipelines and infrastructure. Join a team ensuring accurate and trustworthy data for analytics and business decisions.
About the role
- Senior Software Engineer developing scalable data processing solutions for data engineering team. Implementing data models and building ETL pipelines with Python, Spark, and SQL.
Responsibilities
- Design and develop scalable data processing solutions using Spark and PySpark
- Implement robust data models and optimize SQL queries
- Collaborate with data analysts and stakeholders
- Build and maintain ETL pipelines leveraging Python, Hive, and SQL
- Conduct thorough code reviews, performance tuning, and debugging
- Monitor, troubleshoot, and resolve issues in production data workflows
- Document technical processes, data models, and workflow architectures
- Stay updated with industry trends in big data technologies
Requirements
- 2 - 4 years of experience in software development
- Advanced proficiency in Spark and PySpark
- Strong knowledge of SQL (basic and advanced)
- Expertise in Python programming for data processing
- Experience with Hive for data warehousing solutions
- Solid understanding of data modelling fundamentals
- Ability to design and optimize ETL pipelines
- Hands-on experience with large-scale data processing
- Proficient in performance tuning of SQL and Spark jobs
- Familiarity with distributed computing concepts
- Competence in debugging and troubleshooting data workflows
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field
- Relevant certifications in big data technologies or data engineering are advantageous
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
AI/ML Engineer designing and refining prompts and workflows using large language models. Responsible for developing data pipelines and delivering scalable AI solutions in a hybrid work environment.
AWS Data Architect at Fractal designing and operationalizing AWS data solutions at enterprise scale. Collaborating with clients and mentoring engineers in best practices.
Senior Data Engineer driving data - driven success at Pacific Life. Collaborating with a team to build scalable and secure data solutions in Newport Beach, CA or Charlotte, NC.
Data Architect managing Commercial Data architecture initiatives for Valmet's sales and service team. Leading AI - driven data integrity and quality efforts in a global context.
Data Solutions Architect leading business intelligence solutions and analytics at Crowe. Overseeing data pipelines and analytics frameworks to drive decision - making and compliance.
Senior Lead Data Engineer designing and building scalable data solutions utilizing AI technology for a globally recognized financial institution. Serving sophisticated clients across the globe.
Data Engineer Consultant building and maintaining enriched data infrastructure for analytical thinking at Northwest Permanente. Involves data collection, cleansing, and transformation for business intelligence.
Vice President - Business, Data Architect role at TD Securities focusing on business data architecture and analytics capabilities. Collaborate with stakeholders to define and govern data models and ensure alignment with strategy.
Senior Data Engineer building and implementing data pipelines at Headspace. Collaborating with analytics and data science teams to enhance personalized mental health support.