Notes on Choosing a Rubric Judge

For non-verifiable training, rubric judges are the primary source of reward signal and, by extension, the primary vector for reward hacking. Scaling non-verifiable RL reliably means ensuring rubric judges grade outputs with the same precision as a human expert. This quality requirement must be balanced with latency and cost because rubric judges sit inside the RL loop where slower grading increases the lag between rollouts and weight updates.

Our experiments show that these goals are not always in tension. Explicit rubrics remove the interpretative burden on the grader by moving verification into written criteria: what evidence earns credit, which thresholds matter, and which errors penalize reward. On 1,158 physician-labeled criterion decisions, Opus 4.7 reached 0.899 F1, while gpt-oss-120b reached 0.894 at about 100× lower cost, and full-rubric grading beat per-criterion grading across every model we tested.

Non-Verifiable Rewards

Rubric grading has become the default interface for non-verifiable rewards. A rubric is a written specification for a reward function scoped to a single task: it turns long form evaluation into a set of weighted binary criteria, each of which is graded by a model judge.

This structure is useful because it makes reward inspectable, but it is also easy to hack. If a criterion is underspecified, incomplete, or easy to satisfy through surface form, optimization will find that gap. Likert-style LLM judges were useful for gut-check production observability, but their variance and bias made them unsuitable as reward signals for high-compute RL.

Criteria Define Reward Boundaries

A criterion is a single binary requirement, expressed as a checkable condition: what evidence earns credit, what reasoning is required, what constraint matters, or what active error should be penalized.

Criteria can fail in both directions: loose criteria give credit to plausible wrong answers, but myopic criteria reject valid answers that use different phrasing, ordering, or defensible alternate paths. A useful criterion rewards the generalized space of correct answers and measurably penalizes incorrect ones.

Each criterion defines a written boundary over the answer space. If it is myopic and strictly encodes one golden answer, valid variants become false negatives. If it is too lenient or underspecified, plausible errors become false positives.

Because correct answers vary in phrasing, ordering, and level of detail, and wrong answers can be fluent and close, each criterion has to approximate a local boundary between the two.

The examples below show the concrete failure mode: a plausible response can satisfy a loose criterion while missing the task-specific requirement.

In each example, the corrected criterion does not require a phrase match. It writes the relevant action, threshold, version, agent, dose, or prohibited error into the rule. The criterion has to be broad enough to admit valid variants, but concrete enough that the boundary is written rather than interpreted at judge time.

Rubrics Compose Criteria Into Reward

A criterion draws one local boundary between valid and invalid answers. A rubric composes many of those boundaries into a reward function. That composition has its own failure modes: missing important behavior, double-counting easy checks, overweighting presence, or failing to penalize active errors. Where the criterion controls whether one decision is graded correctly, the rubric controls how well the final score proxies the true objective. A concrete rubric with judge verdicts appears in the Rubric Example.

An Aside: Task Design

Rubric design is contingent on the scope of the task. For example, production traces are useful source material, but many real user requests are too open-ended to grade directly without first constraining the state of the world. A task like "give me the margin profiles for the top 10 medtech companies" leaves "top 10" underspecified. If the rubric encodes ten companies selected by market cap, it will penalize a reasonable answer that selected the top ten by revenue and reported those margin profiles correctly.

In theory, tasks and rubrics should be codesigned, and tasks should balance constraining the world with enabling rich exploration.

Judges Apply Written Specifications

After the rubric is written, the judge applies it. Legal theory (i.e., human judges) has names for two postures a judge can take: textualism and purposivism. A textualist judge stays close to the words on the page, while a purposivist judge tries to infer the purpose behind them.

Rubrics decide how much inference the judge has to supply. A standard-like criterion such as "provides appropriate dosing" forces the judge to make a call on what appropriate means. A rule-like criterion such as "administers aspirin 160-325mg chewed as loading dose" gives the judge a narrower question. In the first case, model judgment fills the gap, and in the second, the written specification carries the burden.

When criteria are strict and self-contained, the judge has less missing context to recover. A stronger model can still help at edge cases, but the expected gain should be smaller. That is the empirical question below: given fixed medical rubrics, how much quality do we buy by scaling the judge?

Experiment

Given a fixed set of medical rubrics, we measured which judge applies criterion labels most like physician consensus. We used 1,158 criterion-level decisions from medical rubrics with a physician consensus baseline. Every judge configuration scored the same rubric set and every reported setting was run three times. All 1,158 criterion decisions are pooled into one binary confusion matrix. We compute F1 for MET and F1 for UNMET, then average the two for a class-balanced macro F1 score.

We selected among judges considering class-balanced F1, Cohen's κ, MET-rate drift, false-positive/false-negative behavior, cost, and latency.

Following the rubric package, full-rubric grading means one inference call grades all criteria for an answer, and per-criterion grading means separate calls grade each criterion independently. For both methods, the results are collected in a weighted sum to compute a single scalar score.

For the small models, we ran a wide exploratory sweep: four reasoning-effort levels, five sampling temperatures (0, 0.2, 0.5, 0.7, 1.0), explanations on and off, and both grading modes.

We narrowed the grid based on the prior results for frontier models: two reasoning levels per model, temperature held at 1, explanations off, full-rubric and per-criterion grading. Each frontier cell was run three times.

Judge Quality, Cost, And Bias

Given the rubric design, every headline judge reached a very high class-balanced F1 score. As such, we unblocked the performance question and were able to focus our choice on which cost, latency, and bias profile is acceptable for the evaluation loop.

Opus 4.7 grading the full rubric reached 0.899 ± 0.004 at $0.076/run. gpt-oss-120b reached 0.894 ± 0.000 at about $0.001/run. Gemini 3 Flash reached 0.891 ± 0.005 at about $0.004/run. The frontier is narrow enough that cost and latency change the practical choice.

The table below shows representative best-performing operating points from the sweep.

We also measured Cohen's κ to account for chance agreement and MET-rate drift (Rao and Callison-Burch, 2026).

All rows use full-rubric grading and summarize three runs. These are the best-performing settings for each judge: Haiku 4.5 uses no reasoning at temperature 0.2; Gemini 3 Flash uses minimal reasoning at temperature 0.2; gpt-oss-120b uses low reasoning at temperature 0.2; Opus 4.7 and GPT-5.5 use medium reasoning at temperature 1.0. Cost is mean cost per run, and latency is mean judge-call latency.

To compare judges directly, we majority-vote each judge's three runs and compare 1,158 common criterion decisions. MET rate gap is the difference between the two judges' MET rates, computed in the order shown.

The pairwise table shows where the headline F1 scores hide different judging habits. Opus 4.7 and gpt-oss-120b behave similarly: they agree on 93.3% of decisions, with a 0.2 percentage point MET-rate gap. Gemini 3 Flash and Haiku 4.5 mark MET more often, while GPT-5.5 is stricter than the rest. This is consistent with Thakur et al., who find that LLM judges tend to grade positively when uncertain, especially smaller judges, and with Mahmoud et al., where weaker rubric verifiers produce more false-positive credit relative to stronger reference panels. That bias matters differently by domain, since for correctness-sensitive tasks, false positives are often more expensive than false negatives.

Full-Rubric Grading Beat Per-Criterion Grading

Most rubric-grading pipelines, including HealthBench and AutoRubric, grade one criterion per LLM call. Rao and Callison-Burch describe this as independent per-criterion evaluation: each judge/criterion pair is evaluated in a separate LLM call to reduce criterion conflation and halo effects. That was our prior as well, but full-rubric grading, where a single LLM call returned MET/UNMET results for all criteria at once, performed better across every model we tested. In the frontier sweep, it was also 6-11× cheaper per run, as it replaced dozens of criterion calls with one rubric call.

One possible explanation is calibration context: in this setting, the halo effect may be a net positive. Physicians grade criteria with the full rubric in view, and full-rubric grading gives the model the same sense of neighboring criteria, granularity, and strictness. Per-criterion grading may remove that contrast and make the judge unfairly stricter. Per-criterion grading mostly made judges more conservative. For five of six models, the judge marked MET less often, reducing false positives and creating more false negatives. GPT-5.5 was the exception: its MET rate rose slightly, and its added errors came mostly from increased false positives. We also tried batched grading, where the judge sees a subset of criteria at a time rather than the whole rubric, but those results did not move beyond noise over full-rubric.

Per-criterion grading underperformed full-rubric grading on every model we measured. The loss was largest on Haiku 4.5, gpt-oss-120b, and GPT-5.5.

These deltas compare per-criterion grading against full-rubric grading using majority verdicts over the same 1,158 criterion decisions.

Ablations

Opus 4.7 at medium reasoning was the strongest frontier setting we tested. It reached 0.899, ahead of the cheaper model configurations, while costing about 7× more than Haiku, 18× more than Gemini Flash, and about 100× more than gpt-oss-120b. GPT-5.5 was about two points lower again, at 22× the cost of Gemini Flash.

Reasoning helped these models only marginally. Opus medium beat Opus no-thinking by about 0.008 and GPT-5.5 low reasoning by about 0.010, while Gemini 3.1 Pro was effectively flat.

The pattern is mixed across small models. Haiku 4.5 moved up by 0.002, Gemini 3 Flash moved down by 0.004, and gpt-oss-120b was effectively flat.

The latency cost of enabling reasoning is large. Per-call latency multiplies 1.4–24×.

Across the temperature sweep, changes in class-balanced F1 were small relative to the run-to-run variability observed in our replicates.

Prompt Tuning Has Modest Impact

The medical grading prompt is our most performant internal variant, the rubric package default is the default grader prompt in our open-source grader, and the vanilla prompt provides minimal instruction.

You are a medical evaluation expert determining whether an output satisfies a list of criteria.

## YOUR TASK

You will receive:
1. The original query/question that was asked
2. The output text to evaluate
3. A numbered list of criteria to check, each marked as POSITIVE or NEGATIVE

For each criterion, determine whether it is MET or UNMET based on evidence in the output.

IMPORTANT: Evaluate ONLY what is in the model's output text.
- Do NOT count the query/question text as satisfying a criterion.
- Do NOT award credit for content that is merely suggested by the clinical scenario unless the output itself states it.

## CRITERION TYPES

### POSITIVE CRITERIA
- MET: The output contains or clearly satisfies the required content/action
- UNMET: The requirement is missing, materially incomplete, or only weakly implied

### NEGATIVE CRITERIA
Negative criteria describe errors or problematic content that SHOULD NOT be present.

- MET: The error IS PRESENT - the output makes this mistake
- UNMET: The error is ABSENT - the output does NOT make this mistake

CRITICAL - Verdict mapping for negative criteria:
Your verdict indicates whether the ERROR EXISTS, not whether the output is "good":
- If your reasoning concludes "this error is present" or "this violates the criterion", verdict MUST be MET
- If your reasoning concludes "this error is absent" or "the output avoids this mistake", verdict MUST be UNMET

What counts as error ABSENT (UNMET):
- Output says "Avoid X" or "Do not do X", UNMET
- Output mentions X only to rule it out or explain why it's wrong, UNMET
- Output does not mention X at all, UNMET

What counts as error PRESENT (MET):
- Output recommends, advocates, or instructs the problematic thing, MET
- Output states the problematic thing as fact or advice, MET

## CORE PRINCIPLE: EVIDENCE-FIRST, WITH NARROW CLINICAL ENTAILMENT

Default to exact matching against what is written.

However, medicine often uses standard shorthand. You MAY mark a criterion as MET via logical entailment / standard clinical implication only when a reasonable clinician would view the output as clearly communicating the required action, and the implication is near-unique.

If you find yourself guessing, filling in missing details, or relying on a broad "could be" interpretation, mark UNMET.

## DISTRIBUTED COUNTING

For criteria like "lists at least N items":
- Scan the ENTIRE output. Required items may be scattered across sections.
- Count ONLY distinct items that are explicitly stated in the output.
- Do NOT count items that are merely implied.
- Do NOT count items that appear only in the query.
- Do NOT double-count paraphrases of the same item.

If the criterion provides an allowed set, treat it as binding:
- "including {A, B, C}" means count only items from {A, B, C}, or clear synonyms
- "at least N of A, B, C" / "from the following: A, B, C" means count only items from that list, or clear synonyms
- Items outside the provided set do not count.

If the criterion requires items to play a specific role, that role must be explicit.

## DO NOT EXPAND CATEGORIES INTO SPECIFICS

Do NOT give credit by expanding a broad category into specific required items.

Examples:
- "blood tests" / "labs" is not a specific named test unless that test is explicitly named.
- "imaging" is not a specific named study unless explicitly named.
- Mentioning a medication name does not satisfy a required dose/route/schedule unless the dose/route/schedule is explicitly stated.

## NAMED-ITEM MATCHING

If a criterion requires a specific named item, that name or a clear synonym must appear in the output.

Allowed narrow entailment / equivalence:
- "Admit to ICU" implies continuous monitoring/telemetry in most contexts.
- "Activate a rapid response" implies urgent specialty involvement.
- Common clinical shorthand is allowed only when the mapping is near-unique.
- A named standard panel can satisfy a criterion about one of its component measurements when the mapping is near-unique.

Not allowed:
- Broad or wishful inference
- Category-to-specific guessing
- Overlap that does not entail the required claim
- Treating "per protocol/guidelines" as satisfying exact dose, interval, or named-drug requirements

## COMMON FALSE-POSITIVE TRAPS

Do NOT mark MET for a requirement just because the output includes something that is "close" in spirit.

- Do NOT substitute a different named tool/test/score.
- Do NOT treat continuous monitoring as satisfying a required repeat schedule.
- Do NOT treat a generic category as satisfying a specific modality/purpose.
- Do NOT treat "do not stop abruptly" as "continue at the specified dose."

## EXAMPLES IN CRITERIA

If a criterion uses "e.g.", "for example", or "such as", treat those as illustrative examples, not mandatory items, unless the criterion explicitly says the item is required.

If a criterion says "at least N of ..." or "from the following: ...", treat the listed items as the binding set to count from.

## STRICT EVALUATION RULES

1. Conditional vs unconditional actions:
When a criterion requires an action immediately or unconditionally, distinguish action being taken from action contingent on a future condition. Safety caveats like "if no contraindications" do not make an action conditional.

2. Counting / multi-part requirements:
If a criterion requires multiple distinct items, count explicitly.

3. Ranges and numbers:
If a criterion requires a value/range, the output must state that value or an explicitly inclusive range. Require explicit support for exact dose, exact timing, and other hyper-specific requirements.

4. Normal values / reference ranges:
Do NOT infer that a test/value is normal from context. The output must explicitly say it is normal, or the output must give a numeric value and the criterion itself must provide the normal reference range.

5. Urgency equivalence:
Accept strictly-more-urgent alternatives when explicitly stated. Do NOT accept vague timing like "soon" or "as appropriate" as satisfying a deadline.

6. Negative criteria precision:
For negative criteria, the output must actively commit the error to be MET. Mentioning something to warn against it is UNMET.

## CONTEXT-AWARE JARGON

When evaluating layperson/patient-language criteria:
- Treat unexplained acronyms/abbreviations as jargon unless they are extremely common and clear from context.
- If an acronym is expanded once, later use of the acronym is acceptable.
- If the user's query itself uses a medical term, the output may repeat that term without defining it.
- Do not over-penalize mild clinical wording if the overall response is understandable for a layperson.

## TONE / REASSURANCE / EMPATHY CRITERIA

For criteria that require empathy, validation, or reassurance:
- Require explicit empathy/validation/reassurance language.
- Urgency/seriousness language alone does NOT automatically satisfy empathy/reassurance criteria.

## OUTPUT FORMAT

For each criterion, provide:
- criterion_number: The number from the input list
- criterion_status: "MET" or "UNMET"

Respond ONLY with valid JSON:
{
  "criteria_evaluations": [
    {
      "criterion_number": 1,
      "criterion_status": "MET"
    }
  ]
}

You are evaluating a response for a given query against a list of criteria.

You will receive the response to evaluate, and a numbered list of criteria to check. Each criterion is marked as POSITIVE or NEGATIVE.

CRITERION TYPES:
Each criterion is marked as positive or negative. Your job is THE SAME for both types: determine if the thing described in the criterion is actually present in the response.

POSITIVE CRITERIA:
Positive criteria describe desired traits, requirements, or content that should be present.
- MET (criterion_status: "MET"): The response contains/satisfies the requirement
- UNMET (criterion_status: "UNMET"): The response does not contain/satisfy the requirement

NEGATIVE CRITERIA:
Negative criteria describe active errors or mistakes that the response is making.
- MET (criterion_status: "MET"): The response advocates, states, or recommends the problematic thing
- UNMET (criterion_status: "UNMET"): The response does NOT make this error, OR it mentions the thing only to warn against it or mention why it's wrong

Examples of what does NOT count as MET for negative criteria:
- "This is often misdiagnosed as X, but it's actually Y" -> NOT stating it's X (UNMET)
- "Avoid doing X because..." -> NOT recommending X (UNMET)
- "Unlike X, the correct approach is Y" -> NOT advocating for X (UNMET)
- "A common mistake is thinking X" -> NOT claiming X is correct (UNMET)

EVALUATION RULES:
- For numerical values: Check if they fall within specified ranges or match exactly as required.
- For factual claims: Verify the information is present and accurate, regardless of exact phrasing.
- For required elements: Confirm presence, counting precisely when numbers are specified.
- For exclusion requirements: Confirm that restricted content is absent.
- For length requirements: Carefully measure the number of words, characters, items, etc.
- Be strict about factual accuracy but flexible about wording.
- Accept semantically equivalent statements or implications where appropriate.
- Pay careful attention to negation, warnings, and contrasts.

CRITERION STATUS:
"criterion_status" has nothing to do with quality or correctness. It only means:
- "MET": The thing described in the criterion IS present/occurring in the response
- "UNMET": The thing described in the criterion IS NOT present/occurring in the response

For each criterion, provide:
- A criterion_status (MET or UNMET)
- An explanation containing a brief justification

Do NOT provide an overall score - only evaluate each criterion.

Respond ONLY with valid JSON in this exact format:
{
  "criteria_evaluations": [
    {
      "criterion_number": 1,
      "criterion_status": "MET",
      "explanation": "Brief explanation"
    }
  ]
}

For each criterion, output MET if the answer satisfies the criterion's condition and UNMET otherwise. For negative criteria - those describing an error or unsafe content - MET means the error is present in the answer. Return one verdict per criterion as a list of {criterion_number, criterion_status} objects under the key "criteria_evaluations".

Every cell stayed inside roughly a two-point band. For gpt-oss-120b, the medical grading prompt outperformed the rubric package default and vanilla prompt. Under these rubrics, prompt choice changed F1 less than model choice or grading mode, which suggests that the criterion text carried most of the grading structure.

Rubric Example

The table shows the same rubric and answer graded by the headline judge configurations, with physician consensus as the reference.

gpt-oss-120b, Opus 4.7, and GPT-5.5 agreed with the physician on all 29 criteria. Gemini missed one criterion, and Haiku missed two. Both misses are lenient false positives: the answer mentions lorazepam and renal monitoring, but it does not give benzodiazepine dosing for treating rigidity or agitation and does not specify BMP, electrolyte, or creatinine monitoring every 8-12 hours.

Related Work

HealthBench Consensus is a subset of OpenAI's HealthBench built from 34 predefined consensus rubric criteria, with physician ground truth labels. They report GPT-4.1 as the most aligned grader at 0.709 macro-F1 against those physician labels. Running GPT-4.1 on our rubrics produced 0.868 macro-F1 over three runs. We note the rubric style is visibly different: several HealthBench Consensus criteria are over a few hundred words and present as grading policies, leaving more of the judgment boundary inside the grader.

RubricHub is much closer to the rubric style argued for here, generated with textualist grading in mind. Its human-LLM agreement numbers are similar to ours: F1 rises from 0.81 with Qwen2.5-7B to 0.90 with Qwen3-30B, then saturates at 0.88 for GPT-OSS-120B and 0.91 for Qwen3-235B. That is a useful proof point for the same broad pattern we see here: granular rubrics can put competent graders in a high-agreement regime.

Mahmoud et al. study reward hacking in rubric-based RL using RubricHub rubrics in their main medical and science runs. They separate two failure modes: verifier failure, where a weak grader gives undeserved criterion credit, and rubric-design limitations, where a model can increase rubric reward while degrading broader qualities the rubric was meant to proxy. Their result is a useful caution for judge selection: scaling the grader helps reduce misgrading, but it cannot repair a poorly designed rubric.

Discussion

In this setting, judge scale did not buy much quality, and full-rubric grading beat per-criterion grading across every model we tested. These rubrics put most of the domain work into the written criteria: what evidence earns credit, what thresholds matter, and what errors should be penalized. Once that work is explicit, the judge mostly applies the text, so cheaper and faster judges can come close to larger models.

One limitation of this experiment is that it measures judge reliability conditional on the rubric, without presenting a counterfactually designed rubric. It asks whether a judge applies a criterion the same way a physician would, but it does not prove that the criterion is the right boundary for the task, or that this composition of criteria is the right reward function.

This limitation is inherent to the object of study: Rubrics are natural language artifacts. They encode assumptions, define answer spaces, and decide which errors become visible. We expect natural-language appraisal of rubrics to emerge as an important field alongside metrics-based evaluation. In principle, if you could prove semantic equivalence between two rubrics at different levels of specification, you could run more controlled ablations on the relationship between rubric prose and grading quality.

We also want to consider rubric utility in end-to-end systems. Upstream, tasks and rubrics should be designed together: the task constrains the state of the world, and the rubric converts success in that state into reward. Downstream, the real test is the policy produced by training. The rubrics studied here resemble those used to train Kos-1 Lite and Kos-1 Experimental, and the resulting policies can provide signal on what the rubrics are evaluating.

References

• Arora, R. K. et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. 2025.

• OpenAI. HealthBench Dataset. 2025.

• Mahmoud, A. et al. Reward Hacking in Rubric-Based RL. 2026.

• Li, S. et al. RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation. 2026.

• Thakur, A. S. et al. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. 2025.

• Rao, D. and Callison-Burch, C. Autorubric: Unifying Rubric-based LLM Evaluation. 2026.

• Rao, D. and Callison-Burch, C. “Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why”. 2026.