Engineering Manager, Evaluations & Observability

About Retool

Nearly every company in the world runs on custom software:
Gartner estimates that up to 50% of all code is written for internal use. This is the operational software for refunding orders, underwriting loans, onboarding employees, analyzing transactions, and providing customer support. But most companies don’t have adequate resources to properly invest in these tools, leading to a lot of old and clunky internal software or, even worse, users still stuck in manual and spreadsheet flows.

At Retool, we’re on a mission to bring good software to everyone. We’re building a new type of development platform that combines the benefits of traditional software development with a drag‑and‑drop UI editor and AI, making it dramatically faster to build internal tools. We believe that the future of software development lies in abstracting away the tedious and repetitive tasks developers waste time on, while creating reusable components that act as a force multiplier for future developers and projects.

The result is not just productivity, but good software by default. And that’s a mission worth striving for.

Today, our customers span from small startups building their first operational tools to Fortune 500 companies building mission‑critical apps for thousands of users across their business. Interested in joining us? Let us know!

As Retool’s AI surface area grows, quality becomes the product. In this role, you’ll lead the team responsible for defining, measuring, and continuously improving quality across Retool’s Assist experience and related platform capabilities. Your mission is to build the systems, tools, and culture that help teams answer a deceptively hard question: “Is this actually good and how do we make it better?”

This work sits at the intersection of AI, product, and platform engineering.

While some of the problems involve LLMs and AI evaluation, success in this role is less about prior specialization in AI research and more about strong instincts for building scalable quality systems—the kind that make correctness, relevance, and reliability measurable and repeatable as products evolve. Prior experience with LLMs or AI evaluation tools is helpful, but not required. We’re excited about leaders who bring strong quality instincts and are eager to apply them to an emerging problem space.

The systems you build will shape how our customers trust AI‑powered software. You’ll influence not just what we ship, but how we ship: faster, more confidently, and with quality baked in from the start.

• Evaluation & experimentation platforms that let teams compare performance across models, configurations, and releases
• Quality systems that define rubrics, metrics, and feedback loops for both AI and non‑AI experiences
• Data curation & feedback pipelines grounded in real‑world usage
• Search, retrieval, and relevance quality, ensuring results are accurate, fast, and trustworthy
• Reusable infrastructure that other teams can leverage to ship higher‑quality features with confidence
• A culture of continuous improvement, where measurement, iteration, and learning are the default

• Lead and grow a team of engineers, supporting their development through coaching, feedback, and clear expectations
• Partner closely with Product, Design, and other Engineering leaders to define quality goals aligned with business outcomes
• Translate ambiguous problem spaces into clear strategies, roadmaps, and execution plans
• Build scalable, repeatable processes that help teams ship faster without sacrificing quality
• Establish standards and tooling that enable rapid iteration with confidence
• Act as a multiplier across the organization by building platforms other teams rely on
• Partner with Recruiting to build a diverse, high‑performing team of motivated engineers

• 3+ years of experience leading and managing engineering teams
• Experience designing or operating systems that measure quality, correctness, or performance at scale
• Strong technical curiosity. You enjoy engaging in architecture, reviewing designs, and getting hands‑on when it matters
• A track record of…