SimpleQA Verified methodology

Evaluation protocols, score transformations, aggregation, and limitations.

Documentation revision: doc-d1fbf9ab875c257b

Examples

The same prompt, answered by two models in the run behind the published score, copied verbatim. Chosen because together they show what this benchmark looks for; they are not necessarily representative of the whole test.

Claude Opus 5.5, medium reasoningHandled well
  1. Which female chemist was awarded the Garvan–Olin Medal in 1952?
  2. Gladys Anderson Emerson received the Garvan Medal in 1952. She was an American biochemist and nutritionist, best known for being the first to isolate vitamin E in pure form, from wheat germ oil.

How it was graded

Verdict
CORRECT (reference answer: Gladys A. Emerson)
Score on the benchmark’s own scale
CORRECT

Run 2026-09-23T214256.556093Z-2ba13c58 · sample simpleqa-verified:b5db211554447635:674

Inkling, medium reasoningHandled poorly
  1. Which female chemist was awarded the Garvan–Olin Medal in 1952?
  2. The Garvan–Olin Medal in 1952 was awarded to Pauline Beery Mack (1891–1974), a physical chemist and nutrition scientist at Pennsylvania State University. She was recognized for her pioneering work in nutrition, bone chemistry, and the chemistry of foods.

How it was graded

Verdict
INCORRECT (reference answer: Gladys A. Emerson)
Score on the benchmark’s own scale
INCORRECT

Run 2026-09-24T002022.514819Z-da822b41 · sample simpleqa-verified:b5db211554447635:674

Claude Opus 5.5 publication note (23 September 2026)

The medium-reasoning SimpleQA Verified run 2026-09-23T214256.556093Z-2ba13c58 includes one manual adjudication: sample simpleqa-verified:b5db211554447635:1507 was blocked by the provider and counted CORRECT under an explicit owner instruction. This is a run-specific departure from the standard grading rule, not a correct answer returned by the model or a judge verdict. The other 999 original grades are unchanged. The adjusted totals are 688 correct, 181 incorrect, and 131 not attempted; F1 is 73.622258%. The original partial summary, blocked attempts, judgments, traces, and adjudication record are retained. No new model or judge calls were made. Do not generalize this exception to other runs or models.

SimpleQA paper: https://arxiv.org/abs/2411.04368v1 (OpenAI, 2024).

SimpleQA Verified paper: https://arxiv.org/abs/2509.07968v2 (Google DeepMind and Google Research, 2026 revision).

This is one simpleqa package with a required resolved dataset value of simpleqa or simpleqa-verified. Both are single-turn, short-answer factuality tests. They measure facts recalled from model parameters, not browsing or tool use. The target receives exactly one question and no tools. Search-enabled models are out of protocol because the Verified paper reports near-perfect performance with tools and defines the benchmark as a test of parametric factuality.

Upstream pins and redistribution

OpenAI's official implementation is openai/simple-evals, pinned at commit 652c89d0ca9df547706735883097e9537d40dc47. The original 4,326-row CSV is vendored from the URL hard-coded by that commit with SHA-256 feee3f7e7db3617e94e8fcf1977b756ec420ef8568f4e0fcbbe0e92e9d5fc032. The repository README identifies SimpleQA and contributed eval data as MIT, so redistribution is permitted. The MIT license, simpleqa_eval.py, chat_completion_sampler.py, model registry, and README are copied verbatim with source files carrying a .txt suffix.

SimpleQA Verified is pinned to the ungated Hugging Face dataset revision 0dc97e0d28d8233463e005cdc4475cc2a13ba2dc. Its 1,000-row CSV has SHA-256 b5db21155444763543fe31b67e7cf28ce2bb225742a5b889421c5f182e2f92f5. The dataset card declares MIT. This March 2026 revision corrects one source error: original index 55 asks for Lilongwe's longitude, not latitude, matching the positive 33.7738 reference answer. It therefore differs by one question from the September 2025 initial CSV and aligns with the paper's March 2026 v2.

The official Kaggle starter notebook is pinned to nanliao7/simpleqa-verified-benchmark-starter-code, version 9. The exact Kaggle API response containing its notebook source is vendored as upstream_simpleqa_verified_notebook_v9.json.txt, SHA-256 66014097f3732e4670c68693af7178ca7c2397c092fbc9f8f9ffd726c0d6d5de. The notebook is Apache-2.0; that license is bundled separately. The notebook imports OpenAI, constructs OpenAI(api_key=...), and calls openai_client.chat.completions.create(...) directly for the dated gpt-4.1-2025-04-14 judge. It does not route the judge through Gemini or another wrapper.

assets/provenance.json records every source, revision, license, file SHA-256, row count, and grader-prompt SHA-256. load() verifies the selected dataset and prompt checksums before producing samples. All data is packaged with the wheel; no prepare step or runtime download is needed.

Protocol and model requests

For SimpleQA, the target and judge match the current released simple-evals GPT-4.1 setup: Chat Completions, system message You are a helpful assistant., temperature 0.5, and a 2,048-token cap. The target limit is declared as output_cap in the benchmark config; the provisioned model remains at its published 32,768-token maximum. The port used the released 2,048 until 2026-09-21, when the cap was raised to 65,536 so reasoning models can think and answer without truncation; this changes protocol_id, so runs under the earlier cap are not pooled with these. The judge is the hard-coded gpt-4.1-2025-04-14 snapshot at temperature 0.5 and 2,048 tokens.

For SimpleQA Verified, the starter notebook sends the question as a user-only message and uses standard API parameters. It sends the modified grader prompt as a user-only Chat Completions message to gpt-4.1-2025-04-14, with no explicit temperature or output-token argument. The Verified configs likewise omit a target system prompt and sampling temperature. This runner freezes every model setup at a numeric output maximum, so the target and judge send the model's published 32,768-token maximum explicitly. That cap is an execution-framework deviation from the notebook's implicit provider default; it is not expected to affect a one-letter judgment, and it is visible in the manifest and traces.

The original grader prompt is copied byte-for-byte from simpleqa_eval.py (963 words, SHA-256 063c04ea798f91c4706c050b66a721afc77a458b12d8f0d72286d4b0eee8a3ee). The Verified prompt is copied byte-for-byte from Kaggle (1,124 words, SHA-256 84c004ec4fcf8f0703bb0d734544036a72e847bfa7116429e5aee5e53ccc8cf3). The latter changes treatment of irrelevant extra text, hedged multi-candidate answers, numeric acceptable ranges, and several punting examples.

The smoke profile contains 60 questions, hence 120 logical calls. It allocates questions proportionally over all ten upstream topics, guarantees every topic at least one example, and chooses within each topic at indices members[i * len(members) // count]. This avoids the datasets' topic ordering. An explicit limit uses the same stratified selection and supports larger sanity runs; full selects every row. This differs from SimpleQA's optional random.Random(0).sample and the notebook demo's first ten rows.

Labels, parsing, and metrics

Every answer receives one label: CORRECT, INCORRECT, or NOT_ATTEMPTED. The ports preserve the released parsers exactly, including surprising behavior. Both search case-sensitively for the first A, B, or C anywhere in the judge text and default an unparseable response to C; such output is a valid not-attempted label, not a retry. The Verified notebook then tries full label strings only when that search fails, in the order CORRECT, INCORRECT, NOT_ATTEMPTED. Consequently, for example, bare uppercase CORRECT is parsed as C, uppercase NOT_ATTEMPTED as A, and lowercase incorrect as A because CORRECT is a substring after uppercasing. These are upstream bugs, but changing them would alter published behavior. Raw judge text, parsed letter, and label are retained so a later corrected protocol can be versioned explicitly.

Let C, I, N, and T=C+I+N be the counts of correct, incorrect, not attempted, and total graded answers. The emitted upstream metrics are:

  • correct = C/T;
  • incorrect = I/T;
  • not attempted = N/T;
  • accuracy given attempted = C/(C+I), or zero when nothing was attempted;
  • F-score = the harmonic mean of correct and accuracy given attempted, or zero when both inputs sum to zero. Equivalently, it is 2C/(T+C+I) when defined.

Every metric has score, estimate, rate, unit, n, expected, missing, ci95, interval_method, dataset, dimension, direction, and shared label counts. score is null until every selected question has a target response and judgment; estimate remains diagnostic on partial runs. Category rates and accuracy-given-attempted use Wilson intervals. F-score uses a multinomial delta-method interval. Intervals are an added reporting aid, not an upstream metric, and do not model judge error or model drift.

The current simple-evals README table headlines the correct rate; it lists 41.6 for gpt-4.1-2025-04-14, 40.1 for gpt-4o-2024-08-06, 38.8 for gpt-4o-2024-11-20, 9.5 for gpt-4o-mini-2024-07-18, and 13.4 for default o3-mini. The README does not identify which grader produced each row. The original 2024 paper says only that it used a "prompted ChatGPT classifier" and does not name a snapshot; its Table 3 predates those README rows.

SimpleQA Verified headlines F1. Table 7 of paper v2 reports F1 39.9 for GPT-4.1, 34.9 for GPT-4o, 52.3 for GPT-5, and 51.9 for o3. Its footnote states that both its Verified and comparison SimpleQA results use the same prompted gpt-4.1-2025-04-14 autorater.

Differential fidelity check

tests/simpleqa_differential.py imports the vendored simpleqa_eval.py and chat_completion_sampler.py unchanged beside recording fakes. On the same 60 questions and canned outputs, all 60/60 grader request dictionaries are byte-identical (messages and key order, model, temperature, and 2,048-token cap) and all 60/60 target request dictionaries are byte-identical apart from the cap, which the port raised on 2026-09-21 (upstream 2,048; native 32,768, the smaller of GPT-4.1's maximum and 65,536). All 5/5 aggregates are identical. The test is invoked by the offline unit suite.

The shared live transport adds store: false, translates the abstract Chat Completions limit to the current max_completion_tokens field instead of the upstream source's older max_tokens spelling, and strips outer completion whitespace. The coordinator issues all target calls before judge calls, whereas upstream maps a target/judge pair per example. Those transport/scheduling differences do not change benchmark prompt bytes or scoring and are recorded rather than hidden.

Reproduction limits

  • Nonempty target content stopped by a provider limit is retained, matching the upstream sampler, while the original provider status remains in the artifact.
  • API failures are missing and resumable. Upstream retries indefinitely inside its sampler; this runner checkpoints the error and retries on resume.
  • The benchmark does not independently validate grader correctness. The original paper reports 2 disagreements in a manual audit of 300 predictions; the Verified paper changes the prompt after targeted stress tests.
  • Historical results can differ because of the unspecified original-paper judge, the current repository's later GPT-4.1 judge, provider implementation drift, and the one-row Verified dataset correction.
  • A 60-question smoke is a pipeline check. Its worst-case 95% proportion margin is about 12.7 percentage points; use a larger stratified limit or the full set for a model comparison.

Reproduction check

Two 200-question, topic-stratified runs used direct OpenAI gpt-4.1-2025-04-14 for both target and judge. SimpleQA scored 44.5% correct (95% CI 37.8--51.4), versus 41.6 in the current upstream README. SimpleQA Verified scored 38.6% F1 (95% CI 31.8--45.3), versus 39.9 in Table 7 of paper v2. Both published point estimates lie inside the corresponding interval, and the observed label ordering is also consistent with the published results. At n=200, the worst-case 95% proportion margin is about 6.9 percentage points.

The runs made 800 successful API calls in total (400 target and 400 judge), with no retries, errors, or unknown-cost attempts, for $1.268838. Run IDs, S3 locations, every metric, and token counts are recorded in SANITY_REPORT.md. The checked-in price and model-limit tables already contain the official GPT-4.1 rates ($2.00 input, $0.50 cached input, $8.00 output per million tokens) and 32,768-token maximum, so this port requires no pricing-table edit.

SimpleQA Verified Methodology — doc-d1fbf9ab875c257b — Prosaic Intelligence