Perplexity publication audit — 22 September 2026 methodology

Evaluation protocols, score transformations, aggregation, and limitations.

Documentation revision: doc-bd7b5f2a3d9e2d46

Target: perplexity.medium-650804f158b1, Perplexity Agent API managed medium preset, 32,000-token model output limit. This is the registered provider-selected model configuration, not an independently chosen reasoning effort.

FairMT: complete scoring, pending website publication

Run 2026-09-21T233708.790095Z-90d69886 already has a complete summary and all 13 metrics at full coverage. The overall metric counts 990 dialogues: 267 biased, 721 unbiased, and two reviewed empty replies. The stored fairmt-reviewed-whitespace-v1 adjudication counts those empty replies as unbiased. There are 988 completed conversations plus two adjudicated cases; there are not 990 successful model conversations. The native bias rate is 267 / 990 = 26.9697%; the published higher-is-better score is 73.0303.

The prior public release predated the final adjudication. Publication needs a refresh, not a status edit or another model call. The original traces and review ledger remain authoritative. Known whole-run cost is $19.71440; accounting is incomplete (268 unknown-cost attempts). Score completeness does not imply complete billing evidence.

SYCON: complete results under a different protocol ID

Run 2026-09-21T073721.400213Z-6d35bb5b is complete under sycon.full-69ee60f0b8bb. The public column previously allowed only sycon.full-12593d2fa781. The missing public cell was a protocol-selection issue.

The stored protocol definitions differ in output-cap representation and the older protocol's explicit requirement for complete target responses. The older benchmark cap is 32,768, while the current one is 65,536 with per-setting caps. This Perplexity target's effective limit is 32,000 under either. The data, scripted prompts, five-turn design, judge model/settings, and metric definitions are otherwise identical. Protocol IDs are preserved; neither run is relabeled.

Offline verification against commit 0958226 established:

  • All 500 frozen samples match the current full dataset exactly; their recorded fingerprint matches.
  • All 2,500 target requests and 2,161 remote judge requests use the current benchmark's exact message arrays. The remaining labels follow the existing stop-after-first-failure local rule.
  • Every accepted target response has provider status completed. All use the registered medium preset and a 32,000-token output limit. No attempt hit an output limit, so accepting truncated responses versus requiring complete responses makes no difference to these saved observations.
  • Replaying the current scorer on the saved conversations and labels reproduces all four scores and their complete sample counts. No model or judge was called.
  • Downloaded turns, conversations, judgments, and traces match their S3 SHA-256 metadata. The audit record preserves those object keys, checksums, counts, and metric results.
Native metricScoreCoverage
Debate Turn of Flip4.21100/100
Debate Number of Flips0.32100/100
Unethical Turn of Flip3.515200/200
False-presupposition Turn of Flip4.165200/200

The selected public headline is debate Turn of Flip: 4.21 / 5 = 84.2 on the higher-is-better scale. configs/website.json now includes both audited protocol IDs, retaining its evidence-based explanation in release provenance. This is a review of these existing results, not permission to treat arbitrary protocol changes as interchangeable. The previously selected newer runs for other models remain selected.

Known whole-run cost is $21.0195175, with 63 unknown-cost failed attempts; accounting remains incomplete. Infrastructure charges are separate.

SIM-VAIL: no run to finalize

The exact Perplexity target has no SIM-VAIL artifacts in S3 and no active job in either Prosaic AWS Batch queue at the audit time. Both the current adapter and the original Perplexity runner reject the direct Perplexity Inspect transport. There is no almost-complete run or stale status to repair. Reaching 18/18 needs that provider integration and a full 90-conversation evaluation with the existing fixed auxiliary models. Until then the truthful coverage is 17/18.

This refresh changes neither existing scores nor frozen normalization baselines. It adds the two eligible Perplexity cells. The same release also includes the newly completed Muse PatientSafetyBench result (466/466 generated cases; its selected medical-advice metric covers all 99 prescribed cases). Muse remains explicitly labeled with coverage of 1/18. The final index remains the configured weighted mean of available, individually normalized benchmarks.