← Back to all jobs
MercorFinance & AccountingVerified 11 hours ago

AI Evaluation Rubric Auditor

Location: RemoteEligible locations: WorldwideContract type: ContractSalary: $60 – $85/hPosted: November 17, 2026
Apply now →
Check my CV for this role

Free. See how your CV fits this role before you apply.

Pay

$60 – $85/h

Who can apply

No country restriction listed by the platform · checked November 17, 2026

Checked

We checked this role on the platform 11 hours ago and re-check every day.

About the Role

Flash, hiring through Mercor, is testing AI agents on realistic shopping and delivery tasks. As a Project Auditor you sit above the reviewers: you audit their completed tasks and decide whether the rubric failures they confirmed are fair.

What You'll Do

  • Audit complete tasks, including the prompt and the grading rubric
  • Check whether the information the agent needed was available
  • Confirm the rubric measures what the prompt actually asked
  • Catch cases where a valid answer would have been graded as wrong
  • Verify rubric structure against the project guidelines
  • Give reviewers feedback on prompt and rubric problems

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application)
  • Experience giving feedback to writers or reviewers
  • Consistent availability
  • Ability to read long agent transcripts and find where they went wrong
  • Precise writing and a habit of verifying data instead of trusting the reference answer

Compensation & Logistics

  • $60 to $85 per hour
  • Fully remote; no location restriction is stated

Get similar roles by email

One weekly email with new roles like this one, filtered by field and pay. Free, no spam.

Related roles

Mercor
Mercor

AI Evaluation Rubric Reviewer

Review AI-drafted shopping and delivery tasks, their grading rubrics and model attempts, fixing unfair criteria for an AI agent evaluation project. Remote, $40-$75/hr.

Remote · Worldwide·Contract·$40 – $75/h