Artificial Intelligence-QA, Mid

Company Overview

Acentra Health exists to empower better health outcomes through technology, services, and clinical expertise. Our mission is to innovate health solutions that deliver maximum value and impact.

Lead the Way is our rallying cry at Acentra Health. Think of it as an open invitation to embrace the mission of the company; to actively engage in problem-solving; and to take ownership of your work every day. Acentra Health offers you unparalleled opportunities. In fact, you have all you need to take charge of your career and accelerate better outcomes – making this a great time to join our team of passionate individuals dedicated to being a vital partner for health solutions in the public sector.

Acentra Health seeks an Artificial Intelligence-QA, Midto join our growing team.

As an AI QA Specialist within Acentra Health’s Product team, you will ensure the accuracy, reliability, and validity of our AI, ML, and Generative AI (GenAI) products and services. Leveraging your expertise in quality assurance methodologies, you will perform Functional, UI, and API testing to effectively validate products such as AI chatbots, ML/GenAI-generated clinical documents, microservices, and other software solutions.

Your role will tackle unique challenges in GenAI environments, including output quality assurance and hallucination detection, ensuring stringent standards are consistently met. Aligned with this, you will support and contribute to our maturing AI Inter-Rater Reliability (AI-IRR) process and services.

Collaborating closely with data scientists, software developers, engineers, and stakeholders, you will design and execute comprehensive test plans and strategies specific to our AI and GenAI platforms. Responsibilities include creating and refining detailed test cases, verifying prompt-response accuracy, and ensuring the ethical and responsible use of AI outputs, ultimately contributing to improved patient outcomes.

• Test Planning & Execution:
  
  Develop and implement test strategies, detailed test plans, and test cases tailored to AI, ML, and GenAI solutions.
• Quality Assurance & UAT:
  
  Conduct thorough Functional, UI, and API testing, including GenAI output quality, hallucination detection, and user acceptance testing to ensure optimal application performance.
• Collaboration:
  
  Work closely with data scientists, software developers, engineers, and project managers to validate algorithms and integrate QA best practices into development processes.
• Automation & Optimization:
  
  Implement automation testing frameworks to optimize testing efforts, including automated GenAI evaluation workflows.
• Data Integrity & Bias Detection:
  
  Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training.
• AI Inter-Rater Reliability:
  
  Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines.
• Documentation & Reporting:
  
  Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement.
• Compliance & Best Practices:
  
  Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage.
• Continuous Improvement:
  
  Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies.
• Read, understand, and adhere to all corporate policies including policies related to HIPAA and its Privacy and Security Rules.

The list of accountabilities is not intended to be all-inclusive and may be expanded to include other education- and experience-related duties that management may deem necessary from time to time.

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent work experience will be considered.
• 3–5 years of quality assurance experience, with a strong background in traditional QA and UAT methodologies.
• Proven experience in testing AI, ML, and Generative AI…