Member of Technical Staff, Pretraining evaluations

Member of Technical Staff, Pretraining Evaluations

Join to apply for the Member of Technical Staff, Pretraining Evaluations role at Cohere
.

Become part of a team building AI systems to power magical experiences across content generation, semantic search, RAG, and agents. Cohere trains and deploys frontier models for developers and enterprises, aiming to scale intelligence to serve humanity.

As a Member of Technical Staff in the pretraining evals team, you will play a key role in helping us make modelling decisions based on experimental outcomes for our large language models (LLMs). Your primary focus will be on developing better ways to measure base model progress.

This role combines expertise in statistics, data science, model evaluation and experience with base model capabilities and how to measure them. If you are interested in measuring model performance accurately as a crucial part of advancing artificial intelligence, we encourage you to apply.

Note:

We have offices in London, Paris, Toronto, San Francisco, and New York, but we embrace remote-friendly policies with no restrictions on location.

• Deeply understand each evaluation task in our base model evaluation suite, knowing what each task measures, its strengths and limitations.
• Suggest and implement improvements to the evaluation suite, adding new tasks to measure unmeasured capabilities or removing redundant or low-signal tasks.
• Improve statistical understanding of evaluation benchmarks and increase signal-to-noise ratio of the suite.

• Familiarity with base model evaluations and their differences from post-trained models.
• Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance.
• Ability to convey statistical information effectively to broad audiences using visualizations and easy-to-understand numbers.
• Extremely strong software engineering skills.
• Proficiency in programming languages such as Python and ML frameworks (PyTorch, Tensor Flow, JAX).
• Excellent communication skills to collaborate effectively with cross-functional teams and present findings.
• One or more papers at top-tier venues (NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, EMNLP).

If some of the above doesn’t line up perfectly with your experience, we still encourage you to apply!

🤝 Open and inclusive culture.

🧑💻 Work closely with a team on the cutting edge of AI research.

🍽 Weekly lunch stipend, in-office lunches & snacks.

🦷 Full health and dental benefits, including a separate budget for mental health.

🐣 100% parental leave top-up for up to 6 months.

🎨 Personal enrichment benefits toward arts and culture, fitness, well-being, quality time, and workspace improvement.

🏙 Remote-flexible; offices in Toronto, New York, San Francisco, London, Paris, and co-working stipend.

✈️ 6 weeks of vacation (30 working days!).

Mid-Senior level

Full-time

Engineering and Information Technology

Referrals increase your chances of interviewing at Cohere by 2x.