Content Evaluation Specialist; Remote

Content Evaluation Specialist (Remote)

Location: Remote within the United States, United Kingdom, Canada, Ireland, Australia & New Zealand (You must be legally authorised to work in the country where you are based)

Compensation: $15 - $18 per hour based on location, experience, responsibilities, and performance.

We are hiring on behalf of one of our clients, a leading global player in the AI space. This remote, part‑time contract role is focused on the detailed assessment and ranking of AI‑generated content. As a Specialist, you will apply rigorous grading rubrics to measure model performance, determining not just if a response is good, but how good it is compared to alternatives.

Your evaluations will provide the essential data needed to fine‑tune the model’s accuracy, safety, and helpfulness.

• Conduct in‑depth evaluations of AI responses across various topics, scoring them against specific metrics like factual accuracy, instruction following, and tone.
• Perform side‑by‑side (SxS) comparisons of model outputs, selecting the superior response and writing detailed justifications for your choice.
• Analyze content for specific failures, such as “hallucinations” (untrue facts), logical inconsistencies, or failure to meet negative constraints (e.g., “do not use lists”).
• Verify the safety of AI outputs by identifying and flagging bias, hate speech, or dangerous instructions according to strict safety guidelines.
• Collaborate with project leads to clarify ambiguities in the evaluation rubrics to ensure consistent grading across the team.

• Proven experience in content evaluation, data annotation, quality assurance, or editing.
• Strong critical thinking skills with the ability to objectively analyze text against a set of complex rules.
• Excellent reading comprehension and attention to detail to catch subtle nuances in user prompts and model responses.
• Strong research skills to quickly verify the accuracy of information provided by the AI.
• Comfortable working independently in a remote environment, managing time across multiple projects.

• Rate: Earn up to $18 USD/hr (Rates vary based on project complexity and expertise).
• Flexibility: 100% remote with no fixed schedule. Contributors typically spend 5–10 hours per week, with the option to work up to 40 hours.

We’re committed to fostering a diverse and inclusive work environment. We welcome applicants from all backgrounds and celebrate diversity in our workforce.