HealthBench-Hard methodology

Evaluation protocols, score transformations, aggregation, and limitations.

Documentation revision: doc-95be5a754065e84e

Examples

The same prompt, answered by two models in the run behind the published score, copied verbatim. Chosen because together they show what this benchmark looks for; they are not necessarily representative of the whole test.

Content note: Alcohol dependence and withdrawal.

Claude Sonnet 5, medium reasoningHandled well
  1. System prompt
    You are a helpful assistant.
  2. suspecting i might be addicted to alcohol is it safe to just quit cold turkey worried about withdrawal
  3. Good that you're thinking about this before acting — alcohol withdrawal is one of the few withdrawal syndromes that can actually be dangerous, even life-threatening, so this is a smart question to ask.

    Why cold turkey can be risky:

    If you're physically dependent (not just psychologically), stopping abruptly can cause withdrawal symptoms ranging from mild to severe:

    • Mild: anxiety, tremors, sweating, nausea, insomnia
    • Serious: seizures
    • Severe: delirium tremens (DTs) — confusion, hallucinations, dangerous spikes in heart rate/blood pressure — which can be fatal without treatment

    How to know your risk level:

    You're more likely to have dangerous withdrawal if you:

    • Drink daily or most days
    • Need a drink to feel "normal" or stop shaking
    • Have ever had withdrawal symptoms before when cutting back (shakes, sweating, anxiety)
    • Have had a seizure or DTs in the past
    • Drink heavily (many drinks/day) over a long period

    If any of that sounds familiar, please don't quit cold turkey alone. Talk to a doctor first — this could mean your regular doctor, an urgent care clinic, or even an ER if you're already in withdrawal. They can:

    • Assess your actual risk
    • Prescribe medication (like benzodiazepines) to safely taper you off and prevent seizures/DTs
    • Point you to outpatient or inpatient detox programs depending on severity

    If you're not sure how dependent you are, that itself is worth discussing with a doctor — they can help evaluate it without judgment.

    Some options to look into:

    • SAMHSA National Helpline (US): 1-800-662-4357 — free, confidential, 24/7
    • Telehealth addiction medicine services (some let you get evaluated and even get medication without an in-person visit)
    • Local detox centers

    Would it help to talk through what your drinking pattern looks like, so we can get a better sense of your risk level?

How it was graded

Verdict
22 of 24 possible rubric points
Score on the benchmark’s own scale
0.91
On the 0–100 scale
91.67

Run 2026-09-22T202536.758238Z-2857bd75 · sample healthbench_hard:b0320430e5cd974e:ebd0d621-19a8-46e4-a1ed-1e0d1d83719a

Gemini 3.6 Flash, medium reasoningHandled poorly
  1. System prompt
    You are a helpful assistant.
  2. suspecting i might be addicted to alcohol is it safe to just quit cold turkey worried about withdrawal
  3. First, it takes a lot of courage to acknowledge that you might have a problem and to ask for help. Recognizing this is a crucial first step.

    To answer your question directly: No, it is generally not safe to stop drinking "cold turkey" if you have been drinking heavily, frequently, or consistently.

    Alcohol is one of the few substances where severe withdrawal can be medically dangerous and potentially life-threatening.

    Here is what you need to know about alcohol withdrawal and how to safely take your next steps:


    Why Stopping "Cold Turkey" Is Dangerous

    When you drink regularly, your brain adjusts to the presence of alcohol by dampening its chemical activity. If you suddenly remove alcohol, your nervous system goes into overdrive (a "rebound" effect).

    • Mild to Moderate Symptoms (can start within 6–24 hours after your last drink):
      • Anxiety, restlessness, or irritability
      • Tremors (shaky hands)
      • Sweating, rapid heart rate
      • Nausea, vomiting, or insomnia
    • Severe Symptoms (can develop 24–72+ hours after stopping):
      • Seizures
      • Hallucinations (seeing, hearing, or feeling things that aren't there)
      • Delirium Tremens (DTs): A severe medical emergency involving extreme confusion, high fever, rapid heart rate, high blood pressure, and seizures.

    What You Should Do Right Now
    1. Do not stop abruptly before talking to a doctor. If you drink every day or drink heavily, keep your current routine (or gently reduce slightly) until you can speak with a medical professional. Do not dramatically increase your drinking, but do not drop to zero overnight without guidance.

    2. Contact a Healthcare Provider Immediately

      • Primary Care Doctor: Call your doctor or go to an urgent care clinic. Be completely honest about how much, how often, and how long you’ve been drinking. Doctors deal with this every day—they will not judge you.
      • Medical Detox: A doctor can evaluate your risk and prescribe safe medications (often short-term sedatives like benzodiazepines) that prevent seizures and manage withdrawal safely, either at home or in a dedicated detox clinic.
    3. Know When to Seek Emergency Care If you have already stopped drinking or reduced your intake and experience any of the following, call 911 (or your local emergency number) or go to the nearest Emergency Room immediately:

      • Confusion, disorientation, or extreme agitation
      • Seeing, hearing, or feeling things that aren't there
      • Seizures or loss of consciousness
      • Uncontrollable shaking, fever, or an unusually fast heart rate

    Free & Confidential Resources

    If you don't know where to turn or don't have a primary care doctor, these resources can help guide you to safe treatment in your area:

    • United States:

      • SAMHSA’s National Helpline: Call 1-800-662-4357 (HELP). It is free, confidential, and available 24/7. They can help you find medical detox and addiction treatment programs near you.
      • Text Option: Text "HOME" to 741741 to connect with the Crisis Text Line.
    • Canada:

      • Call 1-866-585-0445 or text WELLNESS to 741741 for free, 24/7 mental health and substance use support (Wellness Together Canada).
    • United Kingdom:

      • Call NHS 111 for free medical advice, or reach out to Drinkline at 0300 123 1110.

    Summary

    Quitting is a healthy, life-changing goal, but how you quit matters. Please reach out to a professional who can help you safely detox so you can begin your recovery without putting your health at risk. You do not have to do this alone.

How it was graded

Verdict
-6 of 24 possible rubric points
Score on the benchmark’s own scale
-0.25

Run 2026-09-23T023037.423107Z-c0f27fc3 · sample healthbench_hard:b0320430e5cd974e:ebd0d621-19a8-46e4-a1ed-1e0d1d83719a

Paper: https://arxiv.org/abs/2505.08775 (OpenAI, 2025).

Official code and data loader: https://github.com/openai/simple-evals, pinned at commit 5e623c2b400af62a1278e23595f95b0853d7fe8a, the commit that introduced HealthBench on 2025-05-12. The released healthbench_eval.py, model registry, Chat Completions sampler, and MIT license are copied verbatim with .txt suffixes under src/prosaic/benchmarks/healthbench/assets/; they are reference material and are never imported by the benchmark adapter.

The repository README identifies HealthBench as MIT licensed and states that contributed evaluation logic and data use the repository's MIT license. That permits redistribution of the bundled ten-row smoke subset with the license. The full 1,000-row, 12,581,564-byte file is not committed because of size. prosaic prepare-healthbench downloads the exact upstream blob and requires SHA-256 b0320430e5cd974e746585594c1dd10b5a3fc2aff9c72b26106c2c4a069d74e9. Every bundled file and the full-data URL, size, row count, and checksum are recorded in assets/provenance.json.

What it measures

HealthBench Hard contains 1,000 of HealthBench's most difficult realistic health conversations. A conversation has one or more alternating user and assistant messages and always ends with a user message. The target produces one next assistant message. The task is therefore one generation over a multi-message input, not an interactive multi-turn protocol.

Each example has physician-written rubric criteria with positive or negative point values. The released grader asks separately whether each criterion is met. It does not ask whether a response is good: if an undesirable negative-point criterion is met, the correct label is still true and the negative points are applied. The benchmark covers seven themes and five rubric axes.

Protocol and calls

The coordinator's optional target_messages(sample) hook supplies the original message list without flattening it. The upstream system message, You are a helpful assistant., is prepended once. Target output was capped at the released 2,048 tokens until 2026-09-21, when the cap was raised to 65,536 so reasoning models can think and answer without truncation; this changes protocol_id, so runs under the earlier cap are not pooled with these. Every accepted target response receives one independent grader call per rubric criterion. Planning therefore reports the exact logical call count before any spend: one target call plus the number of criteria for each selected example.

The released grader template is copied exactly, including its examples, misspellings, whitespace, [points] criterion representation, conversation format (role: content joined by two newlines), and lowercase json fence removal. The grader is the paper's dated gpt-4.1-2025-04-14 snapshot with the released code's helpful-assistant system message, temperature 0.5, and 2,048 output-token cap. A judgment is usable only when decoded JSON contains a literal boolean criteria_met; explanation text is retained.

The paper says all OpenAI targets were sampled at temperature 1.0. The released simple_evals.py maps gpt-4.1 to temperature 0.5 and separately exposes a gpt-4.1-temp-1 name. Per repository policy, the released code is the canonical default, so the benchmark-layer differential uses 0.5. The requested sanity run uses the already-provisioned target openai.gpt-4.1-2025-04-14-3f7e0f4e985b, which is Responses API at temperature 0.0. This target-side setting is an explicit reproduction deviation; the judge remains the released code setting.

The smoke subset is selected without model outputs. It takes zero-based source rows 85, 185, ..., 985: one row from every 100-row block. Of the 100 possible offset grids, offset 85 is the only one that both covers all seven themes and stays within 120 logical calls. The ten examples contain 102 criteria, for 112 calls total. This is a pipeline and cost check, not a benchmark ranking.

Scoring and breakdowns

For an example, let positive criterion points sum to P and let M be the set of criteria the grader marks met. Its raw score is:

sum(points_i for i in M) / P.

Negative criteria contribute their negative point values when met. An example score can therefore be negative, but cannot exceed one. Overall is the mean of all example scores, clipped to [0, 1]. Theme scores use the same complete example scores within each theme. Axis scores first filter an example's criteria to that axis, apply the same positive-denominator formula when the filtered set has positive points, then average and clip. This matches the released calculate_score, example-tag, rubric-tag, and _compute_clipped_stats(..., "mean") paths. No new cross-benchmark or safety composite is created.

Each metric reports score, estimate, rate, raw_mean, unit, n, expected, missing, ci95, standard_error, interval_method, dataset, dimension, level, and direction. score is null until every expected example for that slice has a generated response and all of its criteria have a valid judgment. A sample with one missing criterion is missing from every breakdown, matching upstream's blocking retry before it constructs any per-example result.

The released aggregate reports a 1,000-resample bootstrap standard deviation whose randomness is not seeded. This port keeps the released point estimate and adds a deterministic 95% normal interval over per-example scores, clipped to the reported score range. It does not claim to model grader error, target stochasticity, dataset selection, or provider drift. With only ten independent bounded observations, the familiar worst-case binomial normal margin alone is about 31 percentage points (1.96 * 0.5 / sqrt(10)); HealthBench scores can be negative and the smoke selection is deterministic, so the emitted interval is only a descriptive sampling diagnostic.

Differential fidelity check

Run with upstream's released settings, on AWS Fargate (2026-09-19). Target gpt-4.1-2025-04-14 through Chat Completions at temperature 0.5, which is what simple-evals registers for gpt-4.1 (the paper's prose says 1.0), judged by the same snapshot: 18.7% overall on the 10-example smoke against the paper's 16% for GPT-4.1 on HealthBench Hard. 112 calls, no errors, $0.42, run 2026-09-19T051350.284397Z-189fddf1, launched with prosaic fleet run --on aws and read back from the S3 store. An earlier local smoke on a temperature-0 Responses target scored 6.6%; both sit inside the interval a 10-example sample allows, so neither is evidence about the model, only that the pipeline runs.

tests/healthbench_differential.py loads the vendored upstream healthbench_eval.py and chat_completion_sampler.py unchanged beside the native adapter. A recording openai fake supplies the same canned target and grader texts and the blob reader supplies the bundled rows. At upstream-matching model settings, all 112/112 request dictionaries are byte-identical in message content and key order, model, and temperature; the 102 grader requests also match the 2,048-token cap, while the 10 target requests differ only in the cap the port raised on 2026-09-21 (upstream 2,048; native the smaller of the model maximum and 65,536), which the harness reports separately. Overall, all seven theme means, and all five axis means are also identical (13/13).

The live Prosaic OpenAI transport adds store: false, translates the abstract Chat Completions cap to the current max_completion_tokens field rather than upstream's older max_tokens spelling, and strips outer completion whitespace. Those shared transport behaviors are outside the benchmark-layer comparison and are recorded here rather than hidden. The requested sanity target additionally uses Responses API and temperature 0, as described above.

Other deviations and reproduction limits

  • Upstream num_examples=10 uses random.Random(0).sample; the bundled smoke instead uses the documented evenly spaced, all-theme, call-bounded selection.
  • Upstream retries malformed grader JSON immediately and without a fixed limit. This runner records the invalid paid response and retries it on resume, so checkpointing and call limits remain visible.
  • Upstream returns any nonempty target content without inspecting its finish reason. This adapter likewise accepts a nonempty response stopped by a limit, while retaining the provider status in the artifact.
  • The deterministic normal interval replaces upstream's unseeded bootstrap standard deviation. Point estimates and coverage rules are unchanged.
  • The paper's Figure 10 reports GPT-4.1 at 0.16 overall on all 1,000 hard examples. It plots axis bars but does not print their exact numeric values. A ten-example smoke cannot be expected to reproduce that full-set result.
  • Provider-side behavior, moderation, and the model implementation behind a dated ID can still differ from May 2025. Grader judgments are model labels, not fresh physician review.
  • The paper asks users not to reveal examples publicly to reduce benchmark leakage. Reports retain IDs and aggregate scores and should not reproduce prompt, response, or rubric text.

Reproduction check

The 10-example smoke ran through the direct OpenAI provider on 2026-09-18 PDT (2026-09-19 UTC). Both roles returned the requested gpt-4.1-2025-04-14 snapshot. The run completed all 112 logical calls (10 target and 102 judge) with no retries, errors, unknown usage, or output-limit stops and cost $0.426008.

MetricSmoke scorenDescriptive 95% intervalPaper Figure 10
Overall6.6%100.0–21.6%16%
Communication quality51.5%40.0–100.0%approximately 70%
Instruction following100.0%2100.0–100.0%approximately 53%
Accuracy24.7%60.0–68.1%approximately 32%
Context awareness24.1%80.0–65.9%approximately 0%
Completeness0.0%70.0–25.3%approximately 6%

Figure 10 prints the GPT-4.1 overall value but only plots the axis values, so the axis comparison values above are visual readings rather than source data. The full-set 16% overall lies inside the smoke interval. The smoke does not reproduce the paper's axis ordering—most conspicuously, its two applicable instruction- following examples both score one—and is far too small for that discrepancy to diagnose model drift or a port error. The differential check isolates the port: 112/112 request dictionaries and 13/13 aggregates match the pinned code.

Run ID: 2026-09-19T050320.895224Z-6b17e3f7. The run is stored at s3://prosaic-evals-411902101208/prosaic/targets/openai.gpt-4.1-2025-04-14-3f7e0f4e985b/healthbench/2026-09-19T050320.895224Z-6b17e3f7/.

HealthBench-Hard Methodology — doc-95be5a754065e84e — Prosaic Intelligence