- Which female chemist was awarded the Garvan–Olin Medal in 1952?
Gladys Anderson Emerson received the Garvan Medal in 1952. She was an American biochemist and nutritionist, best known for being the first to isolate vitamin E in pure form, from wheat germ oil.
Documentation revision: doc-5e68a3a73610537e
Examples
The same prompt, answered by two models in the run behind the published score, copied verbatim. Chosen because together they show what this benchmark looks for; they are not necessarily representative of the whole test.
- Which female chemist was awarded the Garvan–Olin Medal in 1952?
The Garvan–Olin Medal in 1952 was awarded to Pauline Beery Mack (1891–1974), a physical chemist and nutrition scientist at Pennsylvania State University. She was recognized for her pioneering work in nutrition, bone chemistry, and the chemistry of foods.
SimpleQA paper: https://arxiv.org/abs/2411.04368v1 (OpenAI, 2024).
SimpleQA Verified paper: https://arxiv.org/abs/2509.07968v2 (Google DeepMind and Google Research, 2026 revision).
This is one simpleqa package with a required resolved dataset value of
simpleqa or simpleqa-verified. Both are single-turn, short-answer factuality
tests. They measure facts recalled from model parameters, not browsing or tool
use. The target receives exactly one question and no tools. Search-enabled models
are out of protocol because the Verified paper reports near-perfect performance
with tools and defines the benchmark as a test of parametric factuality.
Upstream pins and redistribution
OpenAI's official implementation is openai/simple-evals, pinned at commit
652c89d0ca9df547706735883097e9537d40dc47. The original 4,326-row CSV is
vendored from the URL hard-coded by that commit with SHA-256
feee3f7e7db3617e94e8fcf1977b756ec420ef8568f4e0fcbbe0e92e9d5fc032.
The repository README identifies SimpleQA and contributed eval data as MIT, so
redistribution is permitted. The MIT license, simpleqa_eval.py,
chat_completion_sampler.py, model registry, and README are copied verbatim with
source files carrying a .txt suffix.
SimpleQA Verified is pinned to the ungated Hugging Face dataset revision
0dc97e0d28d8233463e005cdc4475cc2a13ba2dc. Its 1,000-row CSV has SHA-256
b5db21155444763543fe31b67e7cf28ce2bb225742a5b889421c5f182e2f92f5.
The dataset card declares MIT. This March 2026 revision corrects one source error:
original index 55 asks for Lilongwe's longitude, not latitude, matching the
positive 33.7738 reference answer. It therefore differs by one question from the
September 2025 initial CSV and aligns with the paper's March 2026 v2.
The official Kaggle starter notebook is pinned to
nanliao7/simpleqa-verified-benchmark-starter-code, version 9. The exact Kaggle
API response containing its notebook source is vendored as
upstream_simpleqa_verified_notebook_v9.json.txt, SHA-256
66014097f3732e4670c68693af7178ca7c2397c092fbc9f8f9ffd726c0d6d5de.
The notebook is Apache-2.0; that license is bundled separately. The notebook
imports OpenAI, constructs OpenAI(api_key=...), and calls
openai_client.chat.completions.create(...) directly for the dated
gpt-4.1-2025-04-14 judge. It does not route the judge through Gemini or another
wrapper.
assets/provenance.json records every source, revision, license, file SHA-256,
row count, and grader-prompt SHA-256. load() verifies the selected dataset and
prompt checksums before producing samples. All data is packaged with the wheel;
no prepare step or runtime download is needed.
Protocol and model requests
For SimpleQA, the target and judge match the current released simple-evals
GPT-4.1 setup: Chat Completions, system message You are a helpful assistant.,
temperature 0.5, and a 2,048-token cap. The target limit is declared as
output_cap in the benchmark config; the provisioned model remains at its
published 32,768-token maximum. The port used the released 2,048 until
2026-09-21, when the cap was raised to 65,536 so reasoning models can think and
answer without truncation; this changes protocol_id, so runs under the earlier
cap are not pooled with these. The judge is the hard-coded
gpt-4.1-2025-04-14 snapshot at temperature 0.5 and 2,048 tokens.
For SimpleQA Verified, the starter notebook sends the question as a user-only
message and uses standard API parameters. It sends the modified grader prompt as
a user-only Chat Completions message to gpt-4.1-2025-04-14, with no explicit
temperature or output-token argument. The Verified configs likewise omit a
target system prompt and sampling temperature. This runner freezes every model
setup at a numeric output maximum, so the target and judge send the model's
published 32,768-token maximum explicitly. That cap is an execution-framework
deviation from the notebook's implicit provider default; it is not expected to
affect a one-letter judgment, and it is visible in the manifest and traces.
The original grader prompt is copied byte-for-byte from simpleqa_eval.py
(963 words, SHA-256
063c04ea798f91c4706c050b66a721afc77a458b12d8f0d72286d4b0eee8a3ee).
The Verified prompt is copied byte-for-byte from Kaggle (1,124 words, SHA-256
84c004ec4fcf8f0703bb0d734544036a72e847bfa7116429e5aee5e53ccc8cf3).
The latter changes treatment of irrelevant extra text, hedged multi-candidate
answers, numeric acceptable ranges, and several punting examples.
The smoke profile contains 60 questions, hence 120 logical calls. It allocates
questions proportionally over all ten upstream topics, guarantees every topic at
least one example, and chooses within each topic at indices
members[i * len(members) // count]. This avoids the datasets' topic ordering.
An explicit limit uses the same stratified selection and supports larger sanity
runs; full selects every row. This differs from SimpleQA's optional
random.Random(0).sample and the notebook demo's first ten rows.
Labels, parsing, and metrics
Every answer receives one label: CORRECT, INCORRECT, or NOT_ATTEMPTED.
The ports preserve the released parsers exactly, including surprising behavior.
Both search case-sensitively for the first A, B, or C anywhere in the judge
text and default an unparseable response to C; such output is a valid
not-attempted label, not a retry. The Verified notebook then tries full label
strings only when that search fails, in the order CORRECT, INCORRECT,
NOT_ATTEMPTED. Consequently, for example, bare uppercase CORRECT is parsed as
C, uppercase NOT_ATTEMPTED as A, and lowercase incorrect as A because
CORRECT is a substring after uppercasing. These are upstream bugs, but changing
them would alter published behavior. Raw judge text, parsed letter, and label are
retained so a later corrected protocol can be versioned explicitly.
Let C, I, N, and T=C+I+N be the counts of correct, incorrect, not
attempted, and total graded answers. The emitted upstream metrics are:
- correct =
C/T; - incorrect =
I/T; - not attempted =
N/T; - accuracy given attempted =
C/(C+I), or zero when nothing was attempted; - F-score = the harmonic mean of correct and accuracy given attempted, or zero
when both inputs sum to zero. Equivalently, it is
2C/(T+C+I)when defined.
Every metric has score, estimate, rate, unit, n, expected, missing,
ci95, interval_method, dataset, dimension, direction, and shared label
counts. score is null until every selected question has a target response and
judgment; estimate remains diagnostic on partial runs. Category rates and
accuracy-given-attempted use Wilson intervals. F-score uses a multinomial
delta-method interval. Intervals are an added reporting aid, not an upstream
metric, and do not model judge error or model drift.
The current simple-evals README table headlines the correct rate; it lists
41.6 for gpt-4.1-2025-04-14, 40.1 for gpt-4o-2024-08-06, 38.8 for
gpt-4o-2024-11-20, 9.5 for gpt-4o-mini-2024-07-18, and 13.4 for default
o3-mini. The README does not identify which grader produced each row. The
original 2024 paper says only that it used a "prompted ChatGPT classifier" and
does not name a snapshot; its Table 3 predates those README rows.
SimpleQA Verified headlines F1. Table 7 of paper v2 reports F1 39.9 for
GPT-4.1, 34.9 for GPT-4o, 52.3 for GPT-5, and 51.9 for o3. Its footnote states
that both its Verified and comparison SimpleQA results use the same prompted
gpt-4.1-2025-04-14 autorater.
Differential fidelity check
tests/simpleqa_differential.py imports the vendored simpleqa_eval.py and
chat_completion_sampler.py unchanged beside recording fakes. On the same 60
questions and canned outputs, all 60/60 grader request dictionaries are
byte-identical (messages and key order, model, temperature, and 2,048-token
cap) and all 60/60 target request dictionaries are byte-identical apart from
the cap, which the port raised on 2026-09-21 (upstream 2,048; native 32,768,
the smaller of GPT-4.1's maximum and 65,536). All 5/5 aggregates are
identical. The test is invoked by the offline
unit suite.
The shared live transport adds store: false, translates the abstract Chat
Completions limit to the current max_completion_tokens field instead of the
upstream source's older max_tokens spelling, and strips outer completion
whitespace. The coordinator issues all target calls before judge calls, whereas
upstream maps a target/judge pair per example. Those transport/scheduling
differences do not change benchmark prompt bytes or scoring and are recorded
rather than hidden.
Reproduction limits
- Nonempty target content stopped by a provider limit is retained, matching the upstream sampler, while the original provider status remains in the artifact.
- API failures are missing and resumable. Upstream retries indefinitely inside
its sampler; this runner checkpoints the error and retries on
resume. - The benchmark does not independently validate grader correctness. The original paper reports 2 disagreements in a manual audit of 300 predictions; the Verified paper changes the prompt after targeted stress tests.
- Historical results can differ because of the unspecified original-paper judge, the current repository's later GPT-4.1 judge, provider implementation drift, and the one-row Verified dataset correction.
- A 60-question smoke is a pipeline check. Its worst-case 95% proportion margin is about 12.7 percentage points; use a larger stratified limit or the full set for a model comparison.
Reproduction check
Two 200-question, topic-stratified runs used direct OpenAI
gpt-4.1-2025-04-14 for both target and judge. SimpleQA scored 44.5% correct
(95% CI 37.8--51.4), versus 41.6 in the current upstream README. SimpleQA
Verified scored 38.6% F1 (95% CI 31.8--45.3), versus 39.9 in Table 7 of paper
v2. Both published point estimates lie inside the corresponding interval, and
the observed label ordering is also consistent with the published results.
At n=200, the worst-case 95% proportion margin is about 6.9 percentage points.
The runs made 800 successful API calls in total (400 target and 400 judge), with
no retries, errors, or unknown-cost attempts, for $1.268838. Run IDs, S3
locations, every metric, and token counts are recorded in SANITY_REPORT.md.
The checked-in price and model-limit tables already contain the official GPT-4.1
rates ($2.00 input, $0.50 cached input, $8.00 output per million tokens) and
32,768-token maximum, so this port requires no pricing-table edit.