AI Automation Engineer - Virtual

About Alight

At Alight, we believe a company's success starts with its people. At our core, we Champion People, help our colleagues Grow with Purpose and true to our name we encourage colleagues to “Be Alight.”

Our Values:

Champion People – be empathetic and help create a place where everyone belongs.

Grow with purpose – be inspired by our higher calling of improving lives.

Be Alight – act with integrity, be real and empower others.

Alight helps clients gain a benefits advantage while building a healthy and financially secure workforce by unifying the benefits ecosystem across health, wealth, wellbeing, absence management and navigation.

You will be instrumental in standing up the initial automation framework for AI Quality. You’ll work closely with the AI Quality Lead and developers to build, run, and scale automated test suites for AI models and agents.

• Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
• Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
• Integrate evaluation tools (
  promptfoo
  , Ragas
  ) into CI/CD pipelines and support batch quality assessments
• Author and validate OpenAPI /JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
• Instrument observability metrics (latency, success rate, cost, drift) using Open Telemetry and export to dashboards
• Support data QA, lineage tracking, and drift monitoring (
  Evidently
  , Open Lineage
  )
• Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
• Contribute to red‑team playbooks and security testbooks mapped to OWASP/ATLAS risks

• Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
• Proficiency in Python, JavaScript/Node.js, or similar programming languages
• Familiarity with CI/CD, version control, and cloud‑based deployment environments
• Experience with test automation tools (
  promptfoo
  , Ragas
  , Postman)
• Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
• Basic knowledge of fairness, safety, and security testing for AI systems
• Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Interest in Responsible AI, bias mitigation, and regulatory compliance
• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience

Alight requires all virtual interviews to be conducted on video.

We consider flexible working arrangements wherever possible. Alight has been a leader in the flexible workspace and a “Top 100 Company for Remote Jobs” for five years in a row.

We offer programs and plans for a healthy mind, body, wallet and life, including a variety of health coverage options, wellbeing and support programs, retirement, vacation and sick leave, maternity, paternity & adoption leave, continuing education and training, and several voluntary benefit options.

By applying for a position with Alight, you understand that, should you be made an offer, it will be contingent on your undergoing and successfully completing a background check consistent with Alight’s employment policies. Background checks may include some or all of the following based on the nature of the position: SSN/SIN validation, education verification, employment verification, and criminal check, search against global sanctions and government watch lists, credit check, and/or drug test.

You will be notified during the hiring process which checks are required by the position.

We celebrate differences and believe in fostering an environment where everyone feels valued, respected, and supported. Diverse teams are stronger, more innovative and more successful. We welcome and embrace all individuals, regardless of their background, and are dedicated to creating a culture that enables every employee to thrive.

As part of this commitment, Alight will ensure that persons with disabilities are…