← Back to all jobs
MercorFinance & AccountingVerified 13 hours ago

AI Evaluation Rubric Reviewer

Location: RemoteEligible locations: WorldwideContract type: ContractSalary: $40 – $75/hPosted: November 17, 2026
Apply now →
Check my CV for this role

Free. See how your CV fits this role before you apply.

Pay

$40 – $75/h

Who can apply

No country restriction listed by the platform · checked November 17, 2026

Checked

We checked this role on the platform 13 hours ago and re-check every day.

About the Role

Flash, hiring through Mercor, is testing AI agents on realistic shopping and delivery tasks. As a rubric reviewer you check whether each recorded failure is actually fair to the model, and repair the grading where it is not.

What You'll Do

  • Review AI-drafted tasks (prompt plus grading rubric) together with model attempts
  • Check whether the needed information was actually available to the agent
  • Confirm the rubric grades what the prompt really asked for
  • Spot cases where a valid answer would have been marked wrong
  • Verify the rubric structure follows the project guidelines
  • Fix unfair criteria, flag tasks that cannot be saved and write a short note explaining each decision

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application)
  • Ability to read long agent transcripts and pinpoint where things went wrong
  • Precise writing and the habit of checking the data rather than trusting the reference answer

Compensation & Logistics

  • $40 to $75 per hour
  • Fully remote; no location restriction is stated

Get similar roles by email

One weekly email with new roles like this one, filtered by field and pay. Free, no spam.

Related roles

Mercor
Mercor

AI Evaluation Rubric Auditor

Audit reviewers' work on AI agent shopping and delivery tasks, checking that rubric failures are fair and giving feedback to reviewers. Remote, $60-$85/hr.

Remote · Worldwide·Contract·$60 – $85/h