Hybrid Senior Principal Software Engineer, AI Inference

at Red Hat

• Job title: Senior Principal Software Engineer, AI Inference
• Job type: Full Time
• Experience level: Senior
• Salary: $189,600 - $312,730 per year
• Degree requirement: Bachelor's Degree
• Tech skills: Ansible, Cloud, Kubernetes, OpenShift, Python, PyTorch, Terraform
• Location requirements: Hybrid, Boston, United States

About the role

• Senior Principal Software Engineer at Red Hat building and releasing AI Inference Server. Develops and manages multi-cloud GPU infrastructure, CI/CD pipelines, and model validation processes.

Responsibilities

• Build and release vLLM wheels across multiple hardware backends and CPU architectures, managing complex native dependency chains including PyTorch, Triton, and other accelerator-specific libraries
• Design and maintain CI/CD pipelines spanning multiple platforms including GitHub Actions, GitLab CI, and Buildkite for build, test, and release workflows
• Manage and scale multi-cloud GPU infrastructure using Terraform and Ansible, including both bare-metal and Kubernetes-based compute runners
• Own the model validation pipeline, orchestrating accuracy evaluation, performance benchmarking, tool-calling validation, and smoke testing across dozens of LLMs on both bare metal and OpenShift
• Develop and maintain the Python tooling and automation that powers the build, packaging, validation, and release processes
• Drive adoption of agentic AI and intelligent automation to streamline engineering workflows, accelerate debugging, and reduce toil across the team

Requirements

• 8+ years of software engineering experience with significant depth in build systems, release engineering, or infrastructure
• Strong Python development skills with experience building well-tested, maintainable tooling and automation
• Hands-on experience building and packaging Python projects with native compiled extensions, including familiarity with C++ and CUDA build toolchains, wheel packaging, and multi-architecture builds
• Deep familiarity with container ecosystems, including Dockerfiles and Containerfiles, image registries, and container build pipelines
• Understanding of LLM evaluation methodology, including accuracy benchmarks such as MMLU, GSM8K, and HellaSwag, as well as inference performance metrics like throughput and latency
• Experience with CI/CD platforms such as GitHub Actions, GitLab CI, Tekton, or Buildkite
• Solid understanding of release engineering practices including reproducible builds, artifact management, dependency pinning, and security scanning
• Experience with infrastructure-as-code tools such as Terraform and Ansible, and managing cloud resources at scale
• Working knowledge of Kubernetes and/or OpenShift for deploying and testing workloads
• Enthusiasm for applying LLM-based agents and AI-assisted tools to automate engineering workflows, with a track record of identifying repetitive processes and replacing them with intelligent automation
• Excellent communication skills, capable of interacting effectively with both technical and non-technical team members.
• A Bachelor's or Master's degree in computer science, computer engineering, or a related field. A Ph.D. in an ML-related domain is a significant advantage.

Benefits

• Comprehensive medical, dental, and vision coverage
• Flexible Spending Account - healthcare and dependent care
• Health Savings Account - high deductible medical plan
• Retirement 401(k) with employer match
• Paid time off and holidays
• Paid parental leave plans for all new parents
• Leave benefits including disability, paid family medical leave, and paid military leave
• Additional benefits including employee stock purchase plan, family planning reimbursement, tuition reimbursement, transportation expense account, employee assistance program, and more!