Job Description
Type: Hourly contract
Compensation: $20–$30/hour
Location: Remote
Commitment: 10–40 hours/week
Role Responsibilities
- Evaluate outputs from large language models and autonomous agent systems using defined rubrics and quality standards.
- Review multi-step agent workflows, including screenshots and reasoning traces, to assess accuracy and completeness.
- Apply benchmarking criteria consistently while identifying edge cases and recurring failure patterns.
- Provide structured, actionable feedback to support model refinement and product improvements.
- Participate in calibration sessions to ensure consistent evaluation alignment across reviewers.
- Adapt to evolving guidelines and ambiguous scenarios with sound judgment.
- Document findings clearly and communicate insights to relevant stakeholders.