Lead AI QA

Our client is seeking a Senior-level AI Quality Assurance Leader specializing in Generative AI, LLM systems, and AI agents, responsible for defining and driving end-to-end quality strategy for scalable and responsible AI deployments. (Remote, USA Only)

Must Have Skills

• Generative AI (GenAI) system testing – 7+ years overall QA experience with hands-on GenAI validation
• Large Language Models (LLMs) – 5+ years experience validating LLM outputs for accuracy, safety, and bias
• Retrieval-Augmented Generation (RAG) systems – 5+ years experience testing pipeline performance and retrieval quality
• Azure AI Foundry – 3+ years experience testing Azure-based AI solutions
• Lang Graph – 3+ years experience validating orchestration and multi-agent workflows
• Test Automation Frameworks for AI Systems – 5+ years experience building automated validation and evaluation pipelines
• Lang Smith evaluation workflows – 3+ years experience in LLM evaluation and monitoring
• Multi-agent AI architectures – 3+ years experience testing agent coordination and decision logic
• Responsible AI practices – 5+ years experience in bias detection, safety validation, and compliance testing
• Performance and load testing for AI systems – 5+ years experience validating scalability of AI services
• CI/CD integration for AI pipelines – 5+ years experience embedding QA into deployment workflows

Responsibilities

• Define and lead QA strategy for Generative AI pipelines, RAG systems, and multi-agent workflows
• Validate LLM outputs for accuracy, safety, bias, and performance across environments
• Oversee quality assurance of Azure AI Foundry-based AI solutions
• Ensure quality across Lang Graph orchestration and Lang Smith evaluation workflows
• Establish automated testing frameworks and AI-specific evaluation metrics
• Lead QA teams to ensure scalable, reliable, and responsible AI deployments