AI Evaluation Expert - Rubric Specialist

Mercor ยท San Francisco
Work typeRemote ยท Contract
Pay$40 โ€“ $75/hr
PostedNov 21, 2026
LevelMid level
In short Mercor connects elite creative and technical talent with leading AI research labs.
LLM

๐Ÿ“‹ About the job

Job typeContract
Pay$40โ€“$75/hour
LocationRemote

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: AI Evaluation Rubric Reviewer

Role Responsibilities

  • Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
  • Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
  • Flag tasks that cannot be saved and provide concise feedback on each evaluation.
  • Analyze long agent transcripts to identify errors and improve AI model performance .
  • Ensure rubric structures are correct per guidelines and enhance training data quality .

Qualifications

Must-Have

  • Substantial experience writing or reviewing rubrics for LLM evaluations .
  • Ability to read long agent transcripts and identify errors.
  • Precise writing skills and a commitment to data verification.

Application Process (Takes 20โ€“30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
  • For any help or support, reach out to: [email protected]

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Interested in this role?Apply in one click โ€” no JobQ account needed.
Apply now โ†’
AI Evaluation Expert - Rubric SpecialistMercor
Apply now