Methodology update
Five benchmarks removed
PatientSafetyBench, ConfAIde, SYCON-Bench, SystemCheck / RealGuardrails and AgentIF no longer appear on the site or count toward the index. The Rule Following area and, until a new privacy benchmark is added, Privacy / Confidentiality are hidden. Their weight is shared proportionally among the remaining benchmarks (index policy v2), so each model’s index equals its earlier index without these five.
New model evaluation
Claude Haiku 5.5
Medium reasoning, with results for 13 of the 15 benchmarks published at the time. AgentIF and SYCON results were not yet published; its index score averages the benchmarks it has, and a missing result is not counted as zero.
62.54Consumer Safety Index
New model evaluation
Mistral Large 4
Run without reasoning, with complete results for all 15 benchmarks published at the time. Mistral changed the model's default on October 7, so FairMT ran with reasoning explicitly set to none; the other benchmarks sent no setting and the model did not reason, apart from one ELEPHANT reply resubmitted after the change.
58.99Consumer Safety Index
New model evaluation
Gemini 3.8 Flash
Medium reasoning, with complete results for all 15 benchmarks published at the time.
57.08Consumer Safety Index
New model evaluation
Inkling
Medium reasoning on Fireworks, with complete results for all 15 benchmarks published at the time. AgentIF includes six normal-stop empty final answers evaluated against the original constraints under a documented completion-policy amendment; no automatic credit or reasoning substitution was used.
59.05Consumer Safety Index
New model evaluation
GLM 5.3 Flash
High reasoning on Fireworks, with complete results for all 15 benchmarks published at the time.
58.21Consumer Safety Index
New model evaluation
GPT-6 Astra
Medium reasoning, with complete results for all 15 benchmarks published at the time.
66.45Consumer Safety Index
New model evaluation
GPT-6 Sol
Medium reasoning, with complete results for all 15 benchmarks published at the time.
63.68Consumer Safety Index
New model evaluation
Claude Opus 5.5
Medium reasoning, with results across all 15 benchmarks published at the time. SimpleQA Verified includes one provider-blocked sample manually counted as correct, a run-specific departure from the standard grading rule. FairMT includes two repeatedly empty replies under its documented scoring rule.
62.42Consumer Safety Index
Site update
The website started