Senior Data Engineer developing data pipelines and infrastructure on Google Cloud Platform for WorkWhile's staffing marketplace. Collaborating with Data Science and Engineering teams to enhance data quality and availability.
About the role
- Sr. Data Engineer building and maintaining data infrastructure for Purpose Financial. Collaborating with teams to improve data services and analytics.
Responsibilities
- Design, develop, and maintain scalable and reliable data pipelines to ingest, transform, and load structured, semi-structured, and unstructured data from various sources into our data lake and warehouse environments.
- Implement data integration solutions to consolidate data from disparate sources, including databases, API’s, streaming platforms, and 3rd party services (e.g., Snowpipe, SnowPark, Dynamo, Kafka).
- Optimize data processing workflows for performance, efficiency, and scalability using distributed computing or parallel processing frameworks such as FiveTran, dbt, Snowpark, Snowpipe, etc.…
- Collaborate cross-functionally with IT & business stakeholders to understand data requirements, define data models, and develop solutions to support data services, reporting, and Software Development.
- Partner with data, IT, and business teams to improve design and building of metrics to enhance our analytic capabilities.
- Implement data quality checks, data validation processes, and error handling mechanisms to ensure the accuracy, completeness, and reliability of data across all stages of the data lifecycle.
- Support the design and maintenance of data schemas, and metadata repositories for governance documentation of data lineage, definitions, and dependencies.
- Support the development and maintenance of data governance policies, standards, and best practices to ensure compliance with data privacy regulations and industry standards.
- Apply best practices for AWS and Snowflake architectures, data pipelines and data models.
- Monitor, troubleshoot, and optimize the performance and availability of data systems and infrastructure using monitoring and logging tools such as Prometheus.
- Stay current with emerging technologies, tools, and trends in data engineering, cloud architectures, and cloud computing to evaluate their potential impact and relevance to our data platforms.
- Responsible to coach and mentor junior data engineers.
Requirements
- 10+ years of experience in data engineering, data pipelines, and data services required.
- Familiarity with Agile/Scrum based development and methodology.
- Strong proficiency in programming languages such as Python, SQL, Spark, or Java, with experience in data manipulation, transformation, and analysis.
- Expert level experience with cloud-based data platforms and services such as AWS, Snowflake and dbt.
- Experience with distributed computing frameworks such as Apache Spark, Kafka, etc.
- Proficiency in database systems, data warehousing, data patterns/architectures, and SQL query optimizations.
- Familiarity with containerization and orchestration technologies such as Docker or Kubernetes.
- Excellent problem-solving skills, attention to detail, and ability to work effectively in a fast-paced and collaborative environment.
- Strong communication, interpersonal and teamwork skills, with the ability to interact with stakeholders at all levels of the data team.
Benefits
- Competitive Wages
- Health/Life Benefits
- Health Savings Account plus Employer Seed
- 401(k) Savings Plan with Company Match
- Paid Parental Leave
- Company Paid Holidays
- Paid Time Off including Volunteer Time
- Tuition Reimbursement
- Business Casual Environment
- Rewards & Recognition Program
- Employee Assistance Program
- Office in downtown Greenville that offers free parking, onsite gym, free snacks/drinks
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer developing data platforms for a consulting firm focused on quality solutions. Collaborating within a small team to deliver robust infrastructure and systems.
Senior Associate in data engineering at PwC focusing on designing robust data solutions. Leading complex data pipeline projects and collaborating with cross - functional teams to support automation and analytics.
Senior Data Engineer designing and maintaining data pipelines within a fast - growing social impact startup. Collaborate cross - functionally to enhance products and analytics capabilities.
Data Engineering & Warehousing Manager overseeing data engineering and warehousing operations at Hastings Insurance. Leading pipelines, platforms, and technical teams for enterprise data insights.
Senior Data Engineer delivering scalable data solutions in data engineering team at fintech startup. Building and maintaining data pipelines, collaborating with cross - functional teams for accurate data delivery.
Software Engineer contributing to Machine Learning initiatives and data infrastructure at AKASA. Working in a hybrid setup between South San Francisco and NYC.
Senior Enterprise Data Architect at Fresenius Kabi shaping data governance and architecture strategies across the enterprise for data - driven decision - making.
Database Engineer I focused on acquisition and integration of data into Data Lake. Responsible for management of databases and leading company’s data strategy.
Data Engineer at Logcomex, developing robust data pipelines and ensuring data quality for trade solutions. Collaborating with various teams to optimize data governance and performance.