RLHF pipeline for a frontier LLM lab

A frontier AI lab shipping a new flagship model needed to scale their reinforcement-learning-from-human-feedback (RLHF) pipeline from a small in-house team to production-grade throughput ahead of a launch window. Existing crowd-sourced vendors were producing preference data with inter-annotator agreement below 70% — high enough to train a model, low enough to leave real capability gains on the table.

Recruit, vet, and onboard 150+ subject-matter annotators across code, math, safety, and reasoning within 60 days, while holding annotator agreement above 85% and maintaining a full audit trail per response for every preference pair delivered.

1. Step 1

   Sourced 150 vetted specialists from our 2,000+ PhD / Masters network — 40% with published research in ML, reasoning, or applied math

2. Step 2

   Built a domain-tiered annotation pipeline: 4 skill tiers with escalation paths, calibration rounds, and blind gold-set injection at 5% sample rate

3. Step 3

   Deployed dedicated program managers, senior QC leads, and a full governance layer with per-annotator quality dashboards

4. Step 4

   Ran weekly calibration syncs with the client's eval team; re-graded the bottom-quartile annotators every 14 days