Catalog and Metadata Evaluator

Overview

If you are an avid fan and a serious consumer of visual entertainment, the Member Experience team at the client is seeking a unique candidate who can capture the essence of films, series, and specials, all while keeping the experience of our users top of mind. As a Catalog & Metadata Evaluator, your work will directly contribute to the training, evaluation, and improvement of GenAI applications that power member-facing experiences.

In this role, you will apply expert judgment to rate, annotate, and review large-scale content datasets used to train and refine AI systems. By bringing nuance, consistency, and clarity to content evaluation, you will help ensure that AI-driven solutions better understand films and series, respond accurately to member needs, and deliver high-quality, trusted experiences at scale.

• Has a deep and broad knowledge of visual media and is passionate about global entertainment, with a strong understanding of films and series across genres and markets. They can succinctly and objectively communicate what is important about a piece of content—and why—distilling complex creative works into clear, structured insights.
• The ideal candidate is experienced as an expert annotator, rater, or dataset reviewer, particularly in structured or rubric-based environments that support AI or machine-learning use cases. They are comfortable applying expert judgment within defined frameworks, navigating nuance and ambiguity while maintaining consistency and quality at scale.
• They thrive in a high-volume, high-quality, deadline- and data-driven environment, and are highly detail-oriented, reliable, and disciplined in their work. They are flexible, adaptive, and curious, with the ability to evolve alongside changing tools, processes, and organizational needs, and are motivated by work that directly improves member-facing experiences through high-quality data.

• Rate, annotate, and review content recommendations and generated text to support training, evaluation, and quality improvement of large language models (LLMs)
• Apply established guidelines and expert judgment to capture nuance, intent, and context within creative content
• Review and validate datasets used in AI and machine-learning workflows for accuracy, consistency, and completeness
• Identify gaps, edge cases, or patterns in annotations that may impact model performance or member-facing outcomes
• Document evaluations and rationale in a clear, objective, and concise manner
• Collaborate with internal partners across product, data, and AI teams to continuously improve dataset quality and relevance

• Prior experience in media as a writer, editor, content designer, or metadata analyst OR experience as a domain-expert content labeller and AI trainer.
• Passionate about films and series with a deep knowledge of the U.S. entertainment industry and the global streaming landscape
• Ability to distinguish nuances within different content genres and distill the essence of a film/series, including sharing findings in an objective, concise manner
• Ability to master internal tools and systems successfully
• Fluency, and strong conversational writing skills in English required, other languages a plus.
• Experience working with large data catalogs is a plus

• $70-75/hr.