QA Engineer

About The Job
Mercor
connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include
Benchmark
,
General Catalyst
,
Peter Thiel
,
Adam D'Angelo
,
Larry Summers
, and
Jack Dorsey
.

Position:
AI Coding Assistant Evaluator

Type:
Contract
Compensation:
$40–$85/hour
Location:
Remote
Commitment:
Flexible, task-based
Role Responsibilities

• Review long-form transcripts between users and AI coding assistants.
• Analyze the AI's logic, execution, and stated actions in detail.
• Score each transcript using a 10-point rubric across multiple criteria.
• Optionally write brief justifications citing examples from the dialogue.
• Detect mismatches between claims and actions, such as saying "I'll run tests" but not doing so.
• Work independently and asynchronously to complete tasks on your own schedule.

Qualifications
Must-Have

• Senior or Staff Engineers with deep code review experience and execution insight.
• QA Engineers with strong verification and consistency-checking habits.
• Technical Writers or Documentation Specialists skilled at comparing instructions vs. implementation.

Preferred

• Backend or Full-Stack Developers comfortable with function calls, APIs, and test workflows.
• DevOps or SRE professionals familiar with tool orchestration and system behavior analysis.
• Proficiency in Python is helpful.
• Familiarity with other languages like JavaScript, TypeScript, Java, C++, Go, Ruby, Rust, or Bash is a plus.
• Comfort with Git workflows, testing frameworks, and debugging tools is valuable.

Compensation & Legal

• Independent service providers classification.
• Payments issued weekly via Stripe Connect.

Application Process (Takes 20–30 mins to complete)

• Submit your resume to begin.
• If selected, you'll receive rubric documentation and access to the evaluation platform.
• Most applicants hear back within a few business days.

Resources & Support

• For details about the interview process and platform information, please check:
• For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
,