Senior Principal Machine Learning Engineer

Senior Principal Machine Learning Engineer

Atlassians have flexibility in where they work – whether in an office, from home, or a combination of the two. We can hire people in any country where we have a legal entity. Interviews and onboarding are conducted virtually, a part of being a distributed-first company.

Atlassian is seeking a Senior Principal Machine Learning Engineer to join our GenAI Platform organization, focusing on the quality and reliability of Rovo Chat
.

Rovo is Atlassian’s AI teammate, embedded across our products to help teams search, understand, and act on their work. In this role, you will be the technical driver behind making Rovo Chat exceptionally accurate, trustworthy, observable, and reliable at scale
. You will define what “great” looks like for GenAI chat quality, build the platforms and evaluation systems to measure it, and lead cross‑org efforts that materially improve customer outcomes and reduce incidents.

This work sits at the intersection of LLMs, retrieval-augmented generation (RAG), evaluation and quality frameworks, observability, and large‑scale production systems
.

You will join the GenAI Platform pillar within Central AI / Engineering‑AI
, working closely with the Rovo Chat product and engineering teams.

Our mission is to:

• Provide a central GenAI platform (models, infra, evaluation, safety, and tooling) that powers AI experiences across Atlassian.
• Ensure Rovo Chat is a highly reliable, high‑quality assistant across Jira, Confluence, and the rest of our product suite.
• Drive quality, observability, and debuggability for GenAI experiences, so we can quickly detect, root‑cause, and fix issues that impact customers, incidents, Disturbed tickets, and DoS escalations.

You’ll collaborate with:

• Rovo Chat and Search & Conversation teams on chat UX and retrieval quality,
• AI Fundamentals / AI Modeling / ML Platform on modeling, evaluation, training, and serving,
• SRE / Tech Ops / Support (Disturbed / DoS) on reliability, incident response, and root‑cause tooling.

As a Senior Principal Machine Learning Engineer, you will:

Set the bar for Rovo Chat quality & reliability

Define and evolve a north‑star quality and reliability framework for Rovo Chat, spanning:

• Answer correctness, faithfulness, and grounding,
• Safety and policy adherence,
• Latency, robustness, and uptime,
• Incident, Disturbed, and DoS impact.

Translate these into measurable metrics, SLAs/SLOs, and dashboards that are adopted across product and platform teams.

Build the evaluation & observability stack for GenAI chat

Design and lead implementation of end‑to‑end evaluation pipelines for Rovo Chat, including:

• LLM‑as‑a‑judge and other automated evaluation techniques.

Drive observability and debuggability improvements (e.g., tracing, attribution, feature logging, and model behavior introspection) so engineers can quickly root‑cause regressions and incidents
.

Partner with SRE/Tech Ops to connect evaluation and observability signals into incident management
, improving:

• % of incidents successfully root‑caused,
• Disturbed ticket and DoS resolution efficiency.

Lead technical strategy for GenAI platform quality

Define and own technical roadmaps for GenAI platform features that directly impact Rovo Chat quality and reliability (e.g., retrieval quality, RAG orchestration, guardrails, safety filters, fallback strategies, model selection/routing).

• LLM and RAG architectures,
• Knowledge ingestion and retrieval,
• Evaluation & monitoring infra,
• Trust & Safety layers.

Identify and prioritize cross‑pillar investments (e.g., shared eval frameworks, reusable prompt libraries, safety and policy enforcement) that raise the bar across Atlassian AI.

Deliver high‑impact improvements to customer outcomes

Use data from incidents, Disturbed tickets, DoS escalations, and product telemetry to identify systemic quality and reliability gaps.

• Reduce production incidents and regressions,
• Improve “first‑try success” rate of answers,
• Decrease hallucinations and unsafe outputs,
• Improve CSAT/NPS and key adoption/retention metrics for Rovo Chat.

Work closely with PMs and designers to ensure quality and reliability are visible, explainable, and…