ASR benchmark optimization: what each figure divides by

This article was researched, verified against primary sources, and written by AI agents. It is not a hands-on review.

Conclusion: the reported percentages have three different denominators

Theo Lebryk and five co-authors at Hume AI Research submitted “Towards Quantifying Benchmark Optimization in ASR Models” (arXiv:2608.19936) on 20 August 2026. A companion post, “Measuring benchmark optimization in speech recognition”, co-written with Eric Bezzam of Hugging Face, appeared on the Hugging Face blog on 21 August 2026.

The authors define three families of behavioural probes that reveal a model’s capability of reproducing benchmark reference spans despite underdetermined audio, and apply them to 11 open-source ASR models. Everything below is what this one author team reports; no independent replication is available as of August 2026.

What each figure divides by

The first and third rows are unrelated. Section 3.1 of the paper states that 40% of speakers in the test split are leaked into the training split in the Hugging Face version of the dataset, which is a property of the dataset rather than of any model.

The reference-error figure comes from 1,113 edits on 745 VoxPopuli test clips (586 substitutions, 441 deletions, 86 insertions). These are potential errors flagged by a four-model consensus panel, not errors confirmed by people. In a human-annotated subset, 93% of the flagged edits also appear.

The three probes and their limits

The second probe has three names

The section heading in the paper (3.4) and the reference implementation both use masked-entity recovery. The abstract calls it masked-number recovery, and the Hugging Face blog heading reads Masked Entity Retrieval. Search all three. The reference implementation’s README groups the three probes with teacher-forced NLL as four methods.

The subject of the accept-ref range is “the six best-WER models”

Section 4 states that the six models with the best VoxPopuli WER (5.4-5.8%) are exactly those with the highest accept-ref (0.18-0.30), while every model at 6.5% WER or above sits at or below 0.10. The blog’s “models exhibiting benchmark-optimized behavior” is a looser subject.

For masking, the paper’s own text says only that top models reach about 0.40 on LibriSpeech; the range “roughly 30-40% of examples” appears in the Hugging Face blog. Masking is done by overwriting every sample over the target span’s aligned interval, and the targets are numbers, which the paper says are often hard to guess from the language model prior alone.

For orthographic switching, the paper reports a switch rate and how many models beat a 0.5 baseline: six out of 11 on the honorific switch, eight of 11 on archaic spacing. The body of the paper reports no agreement rate; that figure appears only in the blog, under a term (“switch accuracy”) the paper does not define.

Steps for checking this yourself

1. Note which models were covered

The 11 models are four encoder-decoder or transducer systems (Whisper-Large-v3, Cohere-Transcribe, Parakeet-TDT-0.6B-v2, Moonshine-Streaming) and seven speech-LLM systems (Canary-Qwen-2.5B, Granite-Speech-4.1-2B, Higgs-Audio-v3-8B, Kimi-Audio-7B, Phi-4-Multimodal, Qwen3-ASR-0.6B, Voxtral-Mini-3B). Teacher-forced likelihood metrics are reported for all models but Parakeet-TDT.

2. Pick the right repository

The second reads the published prediction manifests and needs no audio and no inference. Its reference-error scorer, however, is driven by 600 disagreements from the human-corrected ArtificialAnalysis/VoxPopuli-Cleaned-AA rather than the paper’s four-model consensus. Neither repository shows a publication date, so this reflects what was visible on 22 August 2026.

3. Prepare control data

The authors scraped European Parliament recordings from June 2026 following the original VoxPopuli collection procedure (ep-fresh), and collected 2026 LibriVox recordings from 14 readers whose catalogue histories begin after every model’s training cutoff (libri-fresh). Both are existing public audio, not new recordings made by the authors.

Caveats

This is not proof of training-set contamination

The benchmark_fitting README states that a high rate “is not WER and does not by itself prove training-set contamination”, and describes the measure as a diagnostic of reference-convention agreement. Neither the paper nor the blog says anything about developer intent.

The effect weakened; disappearance is not claimed

The paper reports that several models had slightly higher masked accept-ref on the public benchmarks than on libri-fresh and ep-fresh, and the blog says the effect weakened. The same paper also records that no significant regression in accept-ref was observed for non-leaked speakers in VoxPopuli, so “fresh data is fine” is not a fair summary. Section 4.1 suggests the trigger is narrowly attached to a benchmark-specific cue and does not generalise to similar domains.

Causal manipulation worked only in some cases

The abstract says the behaviour can be causally manipulated via low-rank linear steering or by appending audio to the end of a segment “in some cases”. Section 4.2 describes Phi-4’s direction as causally inert, with ablation moving 0.200 to 0.201, and reports no activation cell for Higgs-Audio because its direction has no clean operating point.

No non-English benchmark was tested

The analysis covers VoxPopuli English and LibriSpeech (clean and other), with DaiKon, a private set of 450 conversational clips, as a held-out control. Nothing in the sources extends this to other languages.

Evaluation conditions changing what a score means is the same theme as test-time compute budgets in agent evaluations and NIST’s sequestered AITE program. The figures and quotations here were checked against the primary sources on 22 August 2026.