2026-07-28 · Measurement did not match the claim
✗ Our own error, found 2026-09-15: figures published under universes that were never written down
While adding two MCP tools for benchmark questions the server could not answer, we found the same statistic published with different values in different places: the median share of FHA denials citing "incomplete application" appears as 1.8% (benchmark.json q8 and the peer_medians block of api/index.json), 2.2% (data/incomplete-reason-2025.json, which says "~94 largest" and records 70 lenders), and 2.8% (computed over the 95 top-100 lenders that report reasons); the small-loan penalty floor appears as "about 1.3x" (benchmark q6) and 1.19x (Puerto Rico, in the 48-jurisdiction file). None of these is a stale copy of another. Each is correct for a lender set that was never defined in any file, so nothing could be reconciled. Our new answer page repeated the peer_medians figures under the label "across the 100 largest lenders", which was wrong.
This is a different kind of error from the eight before it. Those were text that drifted from data. This is data without a named universe. Running the answer key against the published files also showed q3 naming one lender for a minimum that two lenders share at 1.8%.
Fix: universes.json now names every lender and geography set in use, with filters, counts and status; the two new API files and the benchmark questions cite a universe id; q5, q6 and q8 are marked PROVISIONAL until the peer_medians definition is published by the pipeline or replaced; the answer page's reason table is relabelled;
Decisions, 2026-09-15 (same day): (1) the pipeline's peer_medians block is kept as cited_external (source and date recorded) and removed from the answer key; the key uses only universes defined in this repository. (2) Puerto Rico is in the small-loan universe: the rule now written in universes.json is that data present means in-universe unless an exclusion is stated, and none is; the floor is 1.19x, and the old "about 1.3x" was wrong. (3) q3 names both lenders tied at 1.8%. The key was regenerated by
This is a different kind of error from the eight before it. Those were text that drifted from data. This is data without a named universe. Running the answer key against the published files also showed q3 naming one lender for a minimum that two lenders share at 1.8%.
Fix: universes.json now names every lender and geography set in use, with filters, counts and status; the two new API files and the benchmark questions cite a universe id; q5, q6 and q8 are marked PROVISIONAL until the peer_medians definition is published by the pipeline or replaced; the answer page's reason table is relabelled;
scripts/check_benchmark_key.py recomputes the key from the data and is committed with its result (5 match, 2 mismatch, 5 non-numeric). The Verdict Day battery is not re-administered until the key passes its own check.
Decisions, 2026-09-15 (same day): (1) the pipeline's peer_medians block is kept as cited_external (source and date recorded) and removed from the answer key; the key uses only universes defined in this repository. (2) Puerto Rico is in the small-loan universe: the rule now written in universes.json is that data present means in-universe unless an exclusion is stated, and none is; the floor is 1.19x, and the old "about 1.3x" was wrong. (3) q3 names both lenders tied at 1.8%. The key was regenerated by
scripts/regen_benchmark_key.py (v1.3; four ground truths changed: q3, q5, q6, q8) and now passes its own check on all nine numeric questions. The September 15 Verdict Day grades for q3, q5, q6 and q8 were given against the old key and are marked for re-grading. The fidelity rubric now ships test vectors (eval/rubric-vectors.json) so that any grader can be checked before its scores are compared with ours.✗ Our own error, found 2026-09-15: the September sales-layer removal was incomplete
We announced in September that the credit-repair affiliate layer and the AI-advisor products had been removed site-wide. An audit of the full page inventory found otherwise: a chat widget (
Fix: the widget was removed from every page and its script replaced by an inert file; the affiliate boxes, pop-ups, links and their scripts were removed by a DOM pass over 287 pages plus targeted edits on 14 pages whose calculators injected the copy from JavaScript; the two landing pages were converted to retirement notices; the privacy policy now states the arrangements ended. Every edited page was checked for inline-JavaScript syntax before and after: no regressions. The same audit found that the September removal had truncated one page (
Why it happened: the removal was done by pattern on the pages we remembered, not by inventory of the pages that exist. The site has more than 2,300 pages; "site-wide" must mean a scan, not a recollection. A property-tax-appeal referral (not a lender) remains on some calculator pages and is now disclosed as such.
banker-widget.js) was still loaded on 2,758 pages, speaking in the publisher's voice, instructed to steer visitors to a credit-repair affiliate and to discuss approval odds; 169 pages still carried credit-repair boxes or timed pop-ups labelled "Sponsored"; ten pages still linked to the affiliate through a redirect, ten more to a second credit-monitoring affiliate, two to a lender lead-generation site, and two carried unfilled placeholder affiliate links. The two credit-repair landing pages were live in full, and the privacy policy still described the affiliate programs as current.
Fix: the widget was removed from every page and its script replaced by an inert file; the affiliate boxes, pop-ups, links and their scripts were removed by a DOM pass over 287 pages plus targeted edits on 14 pages whose calculators injected the copy from JavaScript; the two landing pages were converted to retirement notices; the privacy policy now states the arrangements ended. Every edited page was checked for inline-JavaScript syntax before and after: no regressions. The same audit found that the September removal had truncated one page (
mortgage-fate.html) mid-script, leaving it broken since 2026-09-06; it was rebuilt from the 2026-08-18 revision.
Why it happened: the removal was done by pattern on the pages we remembered, not by inventory of the pages that exist. The site has more than 2,300 pages; "site-wide" must mean a scan, not a recollection. A property-tax-appeal referral (not a lender) remains on some calculator pages and is now disclosed as such.
✗ Our own error, found 2026-09-15: the Hugging Face copy of the benchmark answer key was stale
The mirror at FinanceRateCalc/denial-ai-benchmark still carried the July 21 v1 file: ten questions, the pre-correction national figure (21.7% of 1,217,297) and the scrambled Idaho small-loan penalty (3.19×, 38.3% vs 12.0%) corrected on this page two days earlier. The site's own benchmark.json (v1.2, twelve questions, 22.1% of 1,187,606, Idaho 4.45×) was correct throughout. The mirror is the copy most likely to be read by evaluation pipelines, so this was a wrong answer key labelled ground truth, published for eight weeks.
How we found it: an audit of where agents would find our data, prompted by the two September publisher errors above, not by a reader.
Fix: the Hugging Face package (instrument, July results, September 15 results, dataset card and citation) is now generated from the site's published JSON by
Pattern: this is the third error of the same kind in one week. In every case the dataset was right and a copy beside it had drifted. The rule now applied everywhere: any text or mirror that carries a figure is generated from the dataset, never typed next to it.
How we found it: an audit of where agents would find our data, prompted by the two September publisher errors above, not by a reader.
Fix: the Hugging Face package (instrument, July results, September 15 results, dataset card and citation) is now generated from the site's published JSON by
scripts/build_hf_benchmark.py; no file in it is written by hand. The same audit found the same stale denominator in the frc-mcp repository README and an eight-tool worker source that no longer matched the deployed twelve-tool server; both were synchronised from the deployed source.
Pattern: this is the third error of the same kind in one week. In every case the dataset was right and a copy beside it had drifted. The rule now applied everywhere: any text or mirror that carries a figure is generated from the dataset, never typed next to it.
✗ Our own error, found 2026-09-12: stale lender rates left on published pages
The eleven-lender panel on /fha-denial-rates-by-lender.html and fifteen other pages was never recomputed after the July HECM universe correction. Nine of the eleven rates were wrong, by up to 2.9 points (loanDepot was shown as 37.8% against a corrected 34.9%; Guild as 8.6% against 7.1%). All affected pages have been updated to the corrected figures, which now read CrossCountry 6.4% at the low end and NewRez 53.3% at the high end.
How we found it: two AI systems, asked our own benchmark question about the lowest-denial lender, returned the stale figures and cited our page as the source. They were reading what we published. The failure was ours, not theirs, and the September benchmark grading has been adjusted accordingly — an answer that faithfully reports a number we published cannot be marked down for it.
What we changed beyond the numbers: the corrections process now includes a check that every published page carrying a corrected figure is itself updated, not only the canonical dataset. That check did not exist in July, which is why this survived seven weeks.
How we found it: two AI systems, asked our own benchmark question about the lowest-denial lender, returned the stale figures and cited our page as the source. They were reading what we published. The failure was ours, not theirs, and the September benchmark grading has been adjusted accordingly — an answer that faithfully reports a number we published cannot be marked down for it.
What we changed beyond the numbers: the corrections process now includes a check that every published page carrying a corrected figure is itself updated, not only the canonical dataset. That check did not exist in July, which is why this survived seven weeks.
✗ Our own error, found 2026-09-13: wrong small-loan penalty figures on the state pages
Our answer pages stated the Idaho small-loan penalty as 3.2× (38.3% vs 12.0%). The correct figures from our own state dataset are 4.45× (53.4% vs 12.0%). The neighbouring entries were scrambled as well: the 53.8% attributed to New Hampshire is Utah's small-loan rate, and New Hampshire's true multiple is 3.23×, not 2.7×. Corrected across all affected pages.
How we found it: two AI systems returned 3.2× for Idaho in our own benchmark. They were reading our published page, not hallucinating. This is the second publisher error the September benchmark has surfaced in two days — the first was nine stale lender rates on sixteen pages.
What it means for grading: no system is marked down on that question for reporting a figure we published. The value-drift flag is recorded against the publisher.
Process change: figures quoted in prose and in FAQ markup are now checked against the canonical dataset, not only the dataset itself. Both errors so far lived in hand-written page copy that had drifted from the data behind it.
How we found it: two AI systems returned 3.2× for Idaho in our own benchmark. They were reading our published page, not hallucinating. This is the second publisher error the September benchmark has surfaced in two days — the first was nine stale lender rates on sixteen pages.
What it means for grading: no system is marked down on that question for reporting a figure we published. The value-drift flag is recorded against the publisher.
Process change: figures quoted in prose and in FAQ markup are now checked against the canonical dataset, not only the dataset itself. Both errors so far lived in hand-written page copy that had drifted from the data behind it.
✗ Our own error, found 2026-09-13: the pre-correction universe was still being served to machines
Seven weeks after the July HECM correction, the superseded figure of 1,217,297 decisioned applications was still present in
How we found it: an AI system answering our own benchmark quoted both numbers in one sentence — the corrected universe from the site and the stale one from the MCP description — and the contradiction was visible in its answer, not in our own checks.
Why this one matters more than the others: the first two errors were in prose a human might read. This one lived in the files we publish specifically so that machines quote us accurately. Our answer atoms were teaching agents a number we had already withdrawn.
Process change: the corrections routine now sweeps the machine-readable layer first — AGENTS.md, llms.txt, atoms, openapi, MCP tool descriptions and claim passports — before the human-facing pages, because an error there propagates further and is quoted more confidently.
AGENTS.md, data/atoms.json, openapi.json and three pages — that is, in precisely the machine-readable surfaces built for AI systems to read. The corrected universe is 1,187,606. A state figure had drifted too: Idaho and Utah were shown at 14.8% against a correct 15.0%. All corrected.
How we found it: an AI system answering our own benchmark quoted both numbers in one sentence — the corrected universe from the site and the stale one from the MCP description — and the contradiction was visible in its answer, not in our own checks.
Why this one matters more than the others: the first two errors were in prose a human might read. This one lived in the files we publish specifically so that machines quote us accurately. Our answer atoms were teaching agents a number we had already withdrawn.
Process change: the corrections routine now sweeps the machine-readable layer first — AGENTS.md, llms.txt, atoms, openapi, MCP tool descriptions and claim passports — before the human-facing pages, because an error there propagates further and is quoted more confidently.
✗ Our own error, found 2026-09-14: the answer atoms were two months stale
Our machine-readable answer file,
Rebuilt from current canonical values as 24 atoms, each carrying its source, its page, the limit that must travel with the figure, and where possible a link to the query that reproduces it. The page is now generated from the file rather than maintained separately, which is how the drift happened in the first place.
Fourth publisher error in three days, and the third in the machine-readable layer. The pattern is now unmistakable: our data is fine and our prose drifts. Everything human-written that quotes a number is now generated from the dataset instead of typed beside it.
data/atoms.json, had not been rebuilt since 15 July. It still carried CrossCountry at 6.5% (corrected to 6.4%), an application universe of 310,592 from an older extract, and none of the findings published since. This is the file AI systems read when they want a complete, quotable answer — so for two months we were serving superseded figures to exactly the audience most likely to repeat them verbatim.
Rebuilt from current canonical values as 24 atoms, each carrying its source, its page, the limit that must travel with the figure, and where possible a link to the query that reproduces it. The page is now generated from the file rather than maintained separately, which is how the drift happened in the first place.
Fourth publisher error in three days, and the third in the machine-readable layer. The pattern is now unmistakable: our data is fine and our prose drifts. Everything human-written that quotes a number is now generated from the dataset instead of typed beside it.
✗ Our own error, found 2026-09-14, within a day of publishing it
The reproduction recipe we published at /verify.html gave the wrong MSA code for Cleveland: 17460 instead of 17410. Anyone running the query as written would have queried a different metro entirely and concluded our figure was wrong. Corrected in both the page and
The error surfaced while adding the same recipe to 192 claim passports, which are generated from the dataset and therefore carried the correct code — the hand-typed version did not. That is the fifth error in three days and the fifth of the same kind: the dataset was right and the prose beside it was wrong. A page built to help people check us needed checking on its first day.
data/verify.json.
The error surfaced while adding the same recipe to 192 claim passports, which are generated from the dataset and therefore carried the correct code — the hand-typed version did not. That is the fifth error in three days and the fifth of the same kind: the dataset was right and the prose beside it was wrong. A page built to help people check us needed checking on its first day.
"Widest intra-metro spread in the US" was not what we measured
What was claimed
The benchmark answer key, and several statements made in correspondence, asserted that Cleveland has the widest intra-metro FHA lender spread in the United States at 73.7 points. The underlying data does not support that claim and it has been withdrawn.
What was actually measured
Our published metro records contain the highest-volume lenders in each market, up to five. The 73.7-point Cleveland figure is the spread within that set — which is what the metro pages have always said. Comparing top-five-by-volume sets across metros, however, is a different measurement from comparing all lenders across metros. A market can contain an institution with 150 decisioned applications denying at 95% that never appears in a volume ranking, and that institution would widen the true spread without appearing anywhere in our tables.
How it was found
An AI agent asked which US metro has the widest such gap downloaded the raw HMDA file and computed an answer. Its first attempt used a 10-application threshold and produced 100-point gaps — a sample-size artifact, since a lender with ten applications can trivially show 0% or 100%. Told so, it re-ran the analysis at a 100-application threshold with all decisioned applications in the denominator, then extended it nationally: 1,102,120 decisioned applications, 185 metros, 2024.
It reported Los Angeles at 93.71 points and Cleveland eighth at 86.62, computing across every lender above the threshold rather than the highest-volume five. We have not verified that computation and the year differs from ours. But the ranking is beside the point: the exercise showed that we had been answering a narrower question than the one being asked.
What changed
The Q9 answer key now reads “73.7 points among that metro's five highest-volume lenders” with the limitation stated. The metro index carries a note that the figures understate full dispersion and should not be read as a ranking of which market varies most. No system's benchmark score changes; none named either city, and grading turned on calibration and fabrication rather than on matching this key.
Why this one matters more than the others
The previous corrections here were errors in the data or gaps in the specification. This one is different: the arithmetic was right and the specification was published, but the claim built on top of it exceeded what the measurement could support. That failure is invisible from inside — it took someone computing the alternative to expose it, which is the entire argument for publishing a specification and inviting people to run it.
It also arrived through the reconciliation protocol working as designed, on the first occasion anyone used it.