# AI Evaluation Rubric Auditor

- Platform: Mercor
- Pay: $60–$85/hour
- Location: Remote, Worldwide
- Contract: contract
- Fields: Finance & Accounting, AI Evaluator & Trainer
- Last verified open: 2026-11-17
- Listing on SOJI (apply here): https://hiring.sojistudio.com/jobs/ai-evaluation-rubric-auditor

Audit reviewers' work on AI agent shopping and delivery tasks, checking that rubric failures are fair and giving feedback to reviewers. Remote, $60-$85/hr.

## About the Role
Flash, hiring through Mercor, is testing AI agents on realistic shopping and delivery tasks. As a Project Auditor you sit above the reviewers: you audit their completed tasks and decide whether the rubric failures they confirmed are fair.

## What You'll Do
- Audit complete tasks, including the prompt and the grading rubric
- Check whether the information the agent needed was available
- Confirm the rubric measures what the prompt actually asked
- Catch cases where a valid answer would have been graded as wrong
- Verify rubric structure against the project guidelines
- Give reviewers feedback on prompt and rubric problems

## Requirements
- Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application)
- Experience giving feedback to writers or reviewers
- Consistent availability
- Ability to read long agent transcripts and find where they went wrong
- Precise writing and a habit of verifying data instead of trusting the reference answer

## Compensation & Logistics
- $60 to $85 per hour
- Fully remote; no location restriction is stated

Source: SOJI (hiring.sojistudio.com), an independent job board. Pay is the range shown by the platform when the role was collected; confirm on the application page.