AI Business Analyst

Overview

The AI Quality Assurance Analyst is the quality champion for Allied’s emerging AI capabilities, ensuring that every model, integration, and automation works reliably and safely within our business processes. Working alongside developers, product owners, and business users, the Analyst will design tests that push the limits of our applications and help deliver trustworthy AI solutions.

• Test AI features integrated into core systems (e.g., Dynamics CRM, Qic Link claims platform, Docuvantage) to ensure they function correctly in end-to-end workflows.
• Write and automate test cases for AI outputs and traditional software features; define clear pass/fail criteria (using statistical sampling when appropriate) to validate AI results.
• Own regression testing for AI capabilities: continuously re-run critical test scenarios as models are updated or data changes, to catch issues early.
• Contribute to the overall test strategy and perform adversarial or edge-case testing (e.g., stress tests, prompt injections) to evaluate the robustness and security of AI components.
• Document and recommend solutions for issues found; coordinate with the AI Product Owner and SMEs to clarify requirements for fixes and improvements.
• Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and improve requirements. Emphasize shared accountability for quality across the team.

• Bachelor’s degree in related field or equivalent work experience required.

• 7+ years in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations required.
• Proven expertise in test automation (UI and API) and building regression test suites in CI/CD environments.
• Familiarity with AI/ML system testing principles. Comfortable validating outputs that may vary statistically and determining acceptable accuracy thresholds.
• Strong analytical skills to identify edge cases and pinpoint defects across interconnected systems (frontend, backend, data, model).
• Excellent documentation and communication skills for writing test plans, reporting issues, and guiding teams on quality best practices.
• Experience in industries like insurance or healthcare is a plus (provides domain context for Allied’s processes).
• Test automation tools such as Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing.
• Testing frameworks like PyTest or JUnit, integrated with CI/CD pipelines (Azure Dev Ops, Jenkins) for continuous testing.
• Version control (Git) for maintaining test scripts, and monitoring/logging tools (Azure Monitor, Splunk) to investigate issues.
• Performance testing or security testing tools for evaluating AI systems under load or threat scenarios.

• Job Knowledge
• Time Management
• Accountability
• Communication
• Initiative
• Customer Focus

This is a standard desk role – long periods of sitting and working on a computer are required.

Remote

Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home, while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100

Mbps download/25

Mbps upload. Reliable internet service is essential for staying connected and productive.

The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills, and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate.

Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life & Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend.

Allied reserves the right to amend, change, alter, and…