# AI Evaluation Rubric Reviewer

- Platform: Mercor
- Pay: $40–$75/hour
- Location: Remote, Worldwide
- Contract: contract
- Fields: Finance & Accounting, AI Evaluator & Trainer
- Last verified open: 2026-11-17
- Listing on SOJI (apply here): https://hiring.sojistudio.com/jobs/ai-evaluation-rubric-reviewer

Review AI-drafted shopping and delivery tasks, their grading rubrics and model attempts, fixing unfair criteria for an AI agent evaluation project. Remote, $40-$75/hr.

## About the Role
Flash, hiring through Mercor, is testing AI agents on realistic shopping and delivery tasks. As a rubric reviewer you check whether each recorded failure is actually fair to the model, and repair the grading where it is not.

## What You'll Do
- Review AI-drafted tasks (prompt plus grading rubric) together with model attempts
- Check whether the needed information was actually available to the agent
- Confirm the rubric grades what the prompt really asked for
- Spot cases where a valid answer would have been marked wrong
- Verify the rubric structure follows the project guidelines
- Fix unfair criteria, flag tasks that cannot be saved and write a short note explaining each decision

## Requirements
- Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application)
- Ability to read long agent transcripts and pinpoint where things went wrong
- Precise writing and the habit of checking the data rather than trusting the reference answer

## Compensation & Logistics
- $40 to $75 per hour
- Fully remote; no location restriction is stated

Source: SOJI (hiring.sojistudio.com), an independent job board. Pay is the range shown by the platform when the role was collected; confirm on the application page.