Artificial Intelligence​/Machine Learning Scientist

Location: Hybrid — (In-Person: Tuesday — Thursday)

Department: Innovation and Competency-Based Education

Reports to: Director, Assessment Innovation Research

Exemption Status: Exempt

The American Board of Pediatrics (ABP) is seeking a highly skilled and innovative Artificial Intelligence (AI) /Machine Learning (ML) Scientist to join our team. This role is crucial in leveraging AI and ML to enhance the ABP’s assessment programs and services, including processes for initial certification and supporting the continuous professional development of pediatricians as part of continuing certification. The AI/ML Scientist will lead cross-functional efforts to integrate advanced AI techniques into ABP’s assessment design and implementation, enhancing ABP’s operations and innovation research.

• Develop and maintain production-quality Python code leveraging libraries such as PyTorch, Tensor Flow/JAX, Hugging Face Transformers, scikit-learn, spaCy, Num Py, pandas, and related ML/NLP toolkits to analyze large-scale text and response data, extract actionable insights from statistical analysis, and inform assessment design and decision-making.
• Build, evaluate, and deploy AI agents and model-based systems (e.g., LLM-powered agents, retrieval-augmented generation pipelines, automated scoring and feedback agents) to support assessment development, review, and longitudinal learning processes.
• Lead and collaborate with cross-functional teams (e.g. psychometricians, content experts, IT experts, vendors, etc.) to conduct research projects and integrate AI solutions into operational assessment systems; serve as a subject matter expert in ML and AI system design for ABP leadership and stakeholders.
• Identify and prototype opportunities for innovation across ABP’s assessment portfolio, proposing and implementing AI-driven, Python-based solutions that are scalable, interpretable, and aligned with governance and validity requirements.
• Communicate and disseminate effectively, including presenting and publishing work on AI/ML methodologies and agent-based systems to internal and external audiences such as the ABP Senior Leadership Team, Board of Directors, and the broader pediatric and ABMS communities.

• Supervise additional staff as needed.
• Attend relevant ABP meetings as needed.
• Perform other duties as assigned.

There are no current direct supervisory responsibilities.

• Bachelor’s degree in statistics/machine learning, NLP/computational linguistics, computer science, bioinformatics, clinical informatics, artificial intelligence, or a related field.
• For those without an advanced degree (master’s and/or PhD), minimum 5 years of experience in AI/machine learning, with a strong background in developing and implementing AI solutions.
• Proficiency in programming languages such as Python, R, Java, and cloud platforms such as AWS, Azure, or Google Cloud. Expertise in machine learning, deep learning, and NLP techniques.
• Innovative thinker, collaborative team player, and passionate about leveraging AI to improve measurement science, assessment of clinical skills of physicians, and pediatric health care.
• Experience conducting research studies and publishing findings at conferences and journals in AI/ML or related fields.
• Experience in working on cross-departmental teams and working with external stakeholders.

• Hands-on experience in building ML pipelines or AI agentic design.
• An advanced degree (master’s and/or PhD) is highly preferred.

Travel is expected to be approximately 5%, primarily for national conferences.