Hybrid Senior QA Analyst, AI – Threat Analytics

at Keeper Security, Inc.

• Job title: Senior QA Analyst, AI – Threat Analytics
• Job type: Full Time
• Experience level: Senior
• Salary: Not specified
• Degree requirement: Bachelor's Degree
• Tech skills: AWS, Cloud, Python
• Location requirements: Hybrid, United States

About the role

• Senior QA Analyst focused on testing AI and ML-driven features ensuring quality and performance standards. Collaborates closely with engineering teams for cybersecurity software.

Responsibilities

• Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
• Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
• Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
• Validate data pipelines, input/output transformations and model integration points across Keeper’s platform
• Conduct regression testing on ML models and AI workflows to ensure reliability across releases
• Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
• Validate security, privacy and zero-trust requirements for AI model interactions and data handling
• Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
• Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures

Requirements

• 3+ years of experience in QA, test automation or evaluation of complex systems
• Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
• Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
• Experience testing LLM-integrated features or AI-driven workflows
• Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
• Experience with API testing, JSON validation and test data generation
• Ability to test nondeterministic systems and evaluate model quality with structured methodologies
• Strong analytical, troubleshooting and communication skills
• Experience testing cloud-integrated applications or systems deployed on AWS
• Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience

Benefits

• Medical, Dental & Vision (inclusive of domestic partnerships)
• Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
• Voluntary Short/Long Term Disability Insurance
• 401K (Roth/Traditional)
• A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
• Above market annual bonuses**