Hybrid Senior Research Scientist, Model Evaluation

at Cohere

• Job title: Senior Research Scientist, Model Evaluation
• Job type: Full Time
• Experience level: Senior
• Salary: Not specified
• Degree requirement: Bachelor's Degree
• Location requirements: Hybrid, Toronto, Canada

About the role

• Senior Research Scientist creating evaluation methods and benchmarks for LLMs at Cohere. Working with cross-functional teams to advance AI capabilities and model performance evaluation.

Responsibilities

• Create ambitious new evaluation benchmarks that push the limits of what our models can accomplish.
• Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations.
• Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges; refining LLM-based data synthesis pipelines; and improving evaluation efficiency.
• Build scalable and reusable tools for digging into model performance.

Requirements

• You enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of, and you have developed resources to measure those capabilities.
• You have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality.
• You are obsessive about rigorously measuring AI capabilities, and also about making sure your measurements actually align with the capabilities you care about.
• You have strong software engineering skills.

Benefits

• An open and inclusive culture and work environment
• Work closely with a team on the cutting edge of AI research
• Weekly lunch stipend, in-office lunches & snacks
• Full health and dental benefits, including a separate budget to take care of your mental health
• 100% Parental Leave top-up for 6 months for employees based in Canada, the US, and the UK
• Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
• Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
• 6 weeks of vacation