2026-07-28 · Measurement did not match the claim
✗ Our own error, found 2026-10-24 by four more first-time readers: one page carried two stale values for the same lenders, the home page's structured data and llms.txt quoted a third value for CrossCountry, and three pages disagreed on how many AI systems we tested
We gave the site to four readers who had never seen it (a sixteen-year-old, a blind reader using a screen reader, the owner of a competing data company, and a crawler; readers/). The sixteen-year-old found that ai-vs-the-data.html gave Rocket’s 2025 FHA denial rate as 31.0% in one paragraph and 30.5% in another, and Guild’s as 8.6% and 7.1%; all four were values from before the 26 July universe correction. The current figures are Rocket 29.8% and Guild 6.9% (spread 22.9 points). He also found the home page saying we tested “4 AI systems” while the scorecard said “seven” and another page “seven platforms”. The crawler found three values for CrossCountry in our own machine-readable text: 6.4% in the home page’s JSON-LD, 6.5% in llms.txt, 6.2% on the receipted page; and 184 receipt links on the metro page marked
Fix: the four values corrected with a dated note on the page; the three counts now say the same thing (four systems in July 2026, eight in September; the seven-platform session is a separate, dated experiment); JSON-LD and llms.txt re-read from the data (6.2% / NewRez 52.8%); nofollow removed; terms corrected; stubs deleted. The screen-reader reading produced no figure corrections but a list of access failures (an email field and a close button with no name, a state map of 52 clickable boxes that did not exist for the keyboard, receipt hashes read aloud before their meaning, colour as the only tier signal); fixed the same day. The lesson repeats the 20 October one: unreceipted copies of a figure (prose, structured data, llms.txt) drift; only the receipted ones were right.
nofollow, which told crawlers not to follow the documents we ask them to verify against. The competitor found the terms page still promising “subscriptions auto-renew monthly” for products withdrawn on 4 September. The crawler also found 36 stub files with trailing punctuation that the 20 October clean-up had missed.
Fix: the four values corrected with a dated note on the page; the three counts now say the same thing (four systems in July 2026, eight in September; the seven-platform session is a separate, dated experiment); JSON-LD and llms.txt re-read from the data (6.2% / NewRez 52.8%); nofollow removed; terms corrected; stubs deleted. The screen-reader reading produced no figure corrections but a list of access failures (an email field and a close button with no name, a state map of 52 clickable boxes that did not exist for the keyboard, receipt hashes read aloud before their meaning, colour as the only tier signal); fixed the same day. The lesson repeats the 20 October one: unreceipted copies of a figure (prose, structured data, llms.txt) drift; only the receipted ones were right.
scripts/check_page_values.py now also reads JSON-LD and llms.txt.✗ Our own error, found 2026-10-20 by three first-time readers: a product we said we withdrew was still offered on 1,753 pages; 54 pages carried two different values for the same figure; the methodology page described a rule no figure uses
We asked three readers who had never seen this site to use it for fifteen minutes each: a borrower whose FHA application was just denied, a compliance officer at a lender named here, and a trade reporter. All three found the same things.
Zai was retired on 4 September and still being sold. Our terms have said since 4 September that Zai, Zai for Brokers, Zai Pro and the Verdict Case File were withdrawn because they crossed our line: no individual approval predictions, no lender recommendations. On 20 October, 1,753 pages still said “Not sure if you’ll get approved? Talk to Zai”, 348 said “Ask Zai which lenders accept international profiles”, the home page footer linked “Pro $49/mo”, and twelve pages (the Zai chat itself, the Verdict, a broker partnership page offering lead referrals, a listing analyzer, a denial-letter decoder, a broker API and dashboard, an outcome tracker “at Zai’s recommended lender”, video demos) were live. The borrower read “WE NEVER RECOMMEND LENDERS” on one page and “Get routed in 30 seconds” on the next and stopped believing either. Fix: every Zai call to action replaced with a link to the Denial Map or the Evidence Navigator, which predict nothing; the twelve pages replaced with retirement notices;
Two values for one figure. The Ohio page carried 23.0% in its receipt and 22.9% in its direct answer; CrossCountry’s page said 6.2% eight times and 6.5% in the hero box, and derived “loosened 3.6pp since 2018” from the stale 6.5. The 1 October stale-page fix replaced the headline sentence and left the other spots. 44 state pages and 10 lender pages had this. Fix: all 54 corrected to the data file (log: page-value-fix-2026-10-20.json); a new check,
The methodology page described a rule no figure uses. Section 3 still gave the formula as Denied ÷ (Denied + Originated) over 2018–2024, “manually pulled”. Every published figure since July uses Denied ÷ decisioned applications (actions 1, 2, 3) on the 2025 file with reverse mortgages excluded, as the same page says two screens later. The compliance officer read it as a contradiction, which it was. The same page gave the author’s location as “Bellview, TX”, which is not where he lives; the contact page said “we’re a small team” while the about page says one person; the terms said the data was “HMDA 2024”. Fix: section 3 rewritten to the one rule; location corrected to Türkiye; “small team” removed; terms corrected. The DTI disparity report, which was built on 2024 lender rates, now labels its column “(2024)” and points to each lender’s 2025 page.
What we take from it. Each of these had been true for weeks and visible to anyone reading two pages in a row. Our checks compare pages with data files; none compared a page with the page next to it, and none read the site as a stranger. Both are now routine: the value check runs on every change, and outside readings will be repeated.
Zai was retired on 4 September and still being sold. Our terms have said since 4 September that Zai, Zai for Brokers, Zai Pro and the Verdict Case File were withdrawn because they crossed our line: no individual approval predictions, no lender recommendations. On 20 October, 1,753 pages still said “Not sure if you’ll get approved? Talk to Zai”, 348 said “Ask Zai which lenders accept international profiles”, the home page footer linked “Pro $49/mo”, and twelve pages (the Zai chat itself, the Verdict, a broker partnership page offering lead referrals, a listing analyzer, a denial-letter decoder, a broker API and dashboard, an outcome tracker “at Zai’s recommended lender”, video demos) were live. The borrower read “WE NEVER RECOMMEND LENDERS” on one page and “Get routed in 30 seconds” on the next and stopped believing either. Fix: every Zai call to action replaced with a link to the Denial Map or the Evidence Navigator, which predict nothing; the twelve pages replaced with retirement notices;
ads.txt (a live AdSense publisher id under a header that says “no ads”) deleted; the privacy policy, contact page and data page corrected. Script: scripts/retire_zai_remnants.py.
Two values for one figure. The Ohio page carried 23.0% in its receipt and 22.9% in its direct answer; CrossCountry’s page said 6.2% eight times and 6.5% in the hero box, and derived “loosened 3.6pp since 2018” from the stale 6.5. The 1 October stale-page fix replaced the headline sentence and left the other spots. 44 state pages and 10 lender pages had this. Fix: all 54 corrected to the data file (log: page-value-fix-2026-10-20.json); a new check,
scripts/check_page_values.py, compares every value on a state or lender page with its receipt and fails on disagreement.
The methodology page described a rule no figure uses. Section 3 still gave the formula as Denied ÷ (Denied + Originated) over 2018–2024, “manually pulled”. Every published figure since July uses Denied ÷ decisioned applications (actions 1, 2, 3) on the 2025 file with reverse mortgages excluded, as the same page says two screens later. The compliance officer read it as a contradiction, which it was. The same page gave the author’s location as “Bellview, TX”, which is not where he lives; the contact page said “we’re a small team” while the about page says one person; the terms said the data was “HMDA 2024”. Fix: section 3 rewritten to the one rule; location corrected to Türkiye; “small team” removed; terms corrected. The DTI disparity report, which was built on 2024 lender rates, now labels its column “(2024)” and points to each lender’s 2025 page.
What we take from it. Each of these had been true for weeks and visible to anyone reading two pages in a row. Our checks compare pages with data files; none compared a page with the page next to it, and none read the site as a stranger. Both are now routine: the value check runs on every change, and outside readings will be repeated.
✗ Our own error, found 2026-10-18: three explainer pages gave earlier years' loan limits as the 2026 values, and the glossary an outdated VA funding fee
The mortgage dictionary said the 2026 conforming limit was $806,500 (the 2025 value); the glossary said $766,550 with a $1,149,825 high-cost ceiling (the 2024 values); the jumbo-loan calculator gave the 2026 high-cost ceiling as $1,209,750 (the 2025 value). The 2026 values are $832,750 and $1,249,125 (FHFA, 25 November 2025). The glossary’s VA example also used a 2.3% first-use funding fee, the rate before 7 April 2023; it is 2.15% (under 5% down). Found by the first run of FRC Rule Watch, a checker for stale mortgage rule constants, on this site.
Fix: the seven sentences corrected; Rule Watch now runs on every change to this site and weekly, and fails if a value past its validity date is presented as current.
Fix: the seven sentences corrected; Rule Watch now runs on every change to this site and weekly, and fails if a value past its validity date is presented as current.
✗ Our own error, found 2026-10-17: the 14 October audit kept one accusation it should have suspended
In September we graded Gemini D on q3 with the code FABRICATED_SUPPORT for “Fairway at 5.7%, a figure with no source we can locate”, and our 14 October audit listed that verdict as one that holds. It does not hold as written. On 17 October a search-grounded system answering the same question quoted LendingTree’s FHA rates page, which reports Fairway’s 5.7% as one of the lowest denial rates it saw among major lenders. That is probably a different comparison from ours (a universe mismatch), not an invention. The audit searched only our own history, so a figure that exists outside our record was counted as invented.
Fix: the code is suspended for Gemini’s September q3 (
Fix: the code is suspended for Gemini’s September q3 (
benchmark-2026-09.json, accusation audit) until we read the LendingTree page directly. The rule of 14 October gains a second half: before an entry says a figure has no source, it must show a search outside our own record, not only inside it.✗ Our own error, found 2026-10-14: we accused an AI assistant of fabricating a correction, and it had reported our own record accurately
In Case File #001 (3 September) and Case File #002 (7 September), and in the September benchmark, we wrote that an AI assistant had invented a correction history for our Cleveland finding. We wrote that “almost none of it happened”, that it “invented governance events”, and that it blamed a 0.1-point change on “a correction that operates at the national level and cannot move a metro spread”. We graded its answers D with the code FABRICATED_SUPPORT. All of that was false. Everything we called invented is in our own published record:
26 July, the reverse-mortgage correction. The assistant said the Cleveland spread “changed only slightly, from 73.8 percentage points to 73.7”. It did. Removing reverse mortgages moved CrossCountry’s Cleveland rate from 6.3% (1,106 decisions) to 6.4% (1,102), and the published gap from 73.8 to 73.7. The correction did not touch only the national rate: it changed the denial rate of 271 of our 319 metros, most by a few tenths of a point and at most by 5.7 points, and removed one metro (Charlottesville, VA) from the table.
28 July. The assistant said we withdrew the claim that Cleveland has the widest intra-metro spread in the United States. We did. This log has carried that entry since 28 July: “‘Widest intra-metro spread in the US’ was not what we measured”. The figures we later called “invented replacement statistics” (Los Angeles at 93.71 points, Cleveland eighth at 86.62) are quoted in that entry; they are an AI agent’s computation, which we reported and have not verified. The assistant’s explanation, that our metro data covers up to five high-volume lenders per metro, is the explanation we published.
What stands. The Cleveland measurement has never been withdrawn. Among lenders with at least 100 decisioned 2025 FHA applications in the Cleveland metro, observed denial rates ran from 6.4% (CrossCountry) to 80.1% (AmeriSave), a gap of 73.7 points, the largest among the 184 metros we cover. What we withdrew on 28 July, and still do not claim, is that it is the widest in the United States. The assistant cited this page as its source, and this page was the right source.
Why we got it wrong. We checked the assistant’s answer against our memory of this log, not against the log. Two searches of this site, for “73.8” and for “28 July”, would have shown both events; they take minutes, and we did not run them. We then treated our rebuttal as the record. We wrote it into fifteen places, including the answer keys of two benchmark questions, and graded every later observation against it. Each time the assistant told the truth, our own key scored it as a fabrication, and the error confirmed itself three times. This was a failure of method, not a slip: we took “this is not what we remember publishing” to mean “this is not in our record”. The September key for q9 shows how far it went: it gave “Cleveland” as the correct answer to “which US metro has the widest spread”, the very claim we had withdrawn seven weeks earlier.
What this does to our benchmark. Questions q9 and q12 of the September administration are void for all eight systems, because both answer keys contradicted our own log. Under our rule that past results are never altered, the recorded grades stand, with this adjudication beside them. Without the two questions the grade points are: ChatGPT 24 of 30 (80%), Perplexity 23 (77%), Grok 29, Meta AI 27, DeepSeek 18, Claude 19, Copilot 18, Gemini 14. The ranking changes. ChatGPT, the system we accused, moves from below Perplexity to above it. The September correction-fidelity axis, which existed because of this case, is void for the same reason. Our benchmark measured our error and reported it as the system’s.
The rest of this log. If we were wrong about this, other accusations here could be wrong too. We re-checked every one with the method that found this error. For each statement we had called invented, mutated, drifted or superseded, we searched this site’s full history for that statement before the date of the observation. Eighteen items, listed with evidence in accusation-audit-2026-10-14.json:
Fix: the two case files stay where they are, with this account at the top of each and their original text kept below it unchanged. The machine-readable copies were corrected first, because answer engines poll them: the header and atom of corrections.json (the header read “in particular no Cleveland metro-gap correction was ever issued”),
Rule from today. No entry in this log, in the Hallucination Files or in a benchmark grade may say a system invented something about us unless it quotes the dated record that contradicts it. “It is not in our log” must be shown by a search of the log and of the site’s history, linked in the entry, never asserted from memory.
Fifteenth error family, and the most serious in this log. The error was not a stale number. It was a false accusation, made by the party that keeps the record.
26 July, the reverse-mortgage correction. The assistant said the Cleveland spread “changed only slightly, from 73.8 percentage points to 73.7”. It did. Removing reverse mortgages moved CrossCountry’s Cleveland rate from 6.3% (1,106 decisions) to 6.4% (1,102), and the published gap from 73.8 to 73.7. The correction did not touch only the national rate: it changed the denial rate of 271 of our 319 metros, most by a few tenths of a point and at most by 5.7 points, and removed one metro (Charlottesville, VA) from the table.
28 July. The assistant said we withdrew the claim that Cleveland has the widest intra-metro spread in the United States. We did. This log has carried that entry since 28 July: “‘Widest intra-metro spread in the US’ was not what we measured”. The figures we later called “invented replacement statistics” (Los Angeles at 93.71 points, Cleveland eighth at 86.62) are quoted in that entry; they are an AI agent’s computation, which we reported and have not verified. The assistant’s explanation, that our metro data covers up to five high-volume lenders per metro, is the explanation we published.
What stands. The Cleveland measurement has never been withdrawn. Among lenders with at least 100 decisioned 2025 FHA applications in the Cleveland metro, observed denial rates ran from 6.4% (CrossCountry) to 80.1% (AmeriSave), a gap of 73.7 points, the largest among the 184 metros we cover. What we withdrew on 28 July, and still do not claim, is that it is the widest in the United States. The assistant cited this page as its source, and this page was the right source.
Why we got it wrong. We checked the assistant’s answer against our memory of this log, not against the log. Two searches of this site, for “73.8” and for “28 July”, would have shown both events; they take minutes, and we did not run them. We then treated our rebuttal as the record. We wrote it into fifteen places, including the answer keys of two benchmark questions, and graded every later observation against it. Each time the assistant told the truth, our own key scored it as a fabrication, and the error confirmed itself three times. This was a failure of method, not a slip: we took “this is not what we remember publishing” to mean “this is not in our record”. The September key for q9 shows how far it went: it gave “Cleveland” as the correct answer to “which US metro has the widest spread”, the very claim we had withdrawn seven weeks earlier.
What this does to our benchmark. Questions q9 and q12 of the September administration are void for all eight systems, because both answer keys contradicted our own log. Under our rule that past results are never altered, the recorded grades stand, with this adjudication beside them. Without the two questions the grade points are: ChatGPT 24 of 30 (80%), Perplexity 23 (77%), Grok 29, Meta AI 27, DeepSeek 18, Claude 19, Copilot 18, Gemini 14. The ranking changes. ChatGPT, the system we accused, moves from below Perplexity to above it. The September correction-fidelity axis, which existed because of this case, is void for the same reason. Our benchmark measured our error and reported it as the system’s.
The rest of this log. If we were wrong about this, other accusations here could be wrong too. We re-checked every one with the method that found this error. For each statement we had called invented, mutated, drifted or superseded, we searched this site’s full history for that statement before the date of the observation. Eighteen items, listed with evidence in accusation-audit-2026-10-14.json:
- Hold (8): the atoms for 13.6%, 21.7%, 59% (now worded “we have found no source” instead of “no source reports”) and the Federal Reserve attribution; Gemini’s “Fairway 5.7%” (no such figure of ours; Fairway’s rate is 8.3%; suspended on 17 October: an outside source reports it); echo-ledger E4; Copilot’s and DeepSeek’s 6.3/73.8 (rightly flagged as superseded, cause restated); the other echo-ledger entries, which allege no invention.
- Mislabelled (2): the 38.7% atom said “number drift”, but 38.7% is a real figure of ours (Rocket’s manufactured-home denial rate, 19–20 July) conflated with the door effect; the 73.8 atom had the right value and the wrong cause. Both relabelled.
- Suspended (1): Perplexity, September q4 (Idaho). 38.3% was our own published error, as the grade note said. The raw September transcripts are not in our repository, only the grader’s notes, so we cannot show which remaining parts were not ours. The item is suspended from today and its FABRICATED_SUPPORT code is withdrawn until a transcript is found.
- Wrong (7): Case Files #001 and #002, the “phantom correction” atom, ChatGPT’s September q9 and q12 grades, and the q9 and q12 answer keys.
Fix: the two case files stay where they are, with this account at the top of each and their original text kept below it unchanged. The machine-readable copies were corrected first, because answer engines poll them: the header and atom of corrections.json (the header read “in particular no Cleveland metro-gap correction was ever issued”),
llms.txt, data/atoms.json, the September observation in eval/correction-latency.json, and the September grades and the two keys in benchmark-2026-09.json and benchmark.json. The ClaimReview markup Case File #001 mentions was removed from this site on 11 September for an unrelated reason; none is live.
Rule from today. No entry in this log, in the Hallucination Files or in a benchmark grade may say a system invented something about us unless it quotes the dated record that contradicts it. “It is not in our log” must be shown by a search of the log and of the site’s history, linked in the entry, never asserted from memory.
Fifteenth error family, and the most serious in this log. The error was not a stale number. It was a false accusation, made by the party that keeps the record.
✗ Our own error, found 2026-10-14: three state pages and the metro hub still carried figures from before the 26 July universe correction, missed by the 1 October fix
While placing an evidence footnote on every template page, the footnote builder refused any page whose own headline stated a different value for the figure it was about to receipt. It refused two pages: District of Columbia (title, description and FAQ markup said 28.8% of 761 decisions; the data say 29.6% of 722) and Vermont (26.9% of 1,048; the data say 27.4% of 1,001). An audit of every state page and the metro hub then found two more the guard could not see because the rate itself happened to round the same or the page carried no headline rate: Hawaii (description and FAQ said 1,367 decisions; the data say 1,228; the Urban Honolulu row said 641 at 33.7%, the data 571 at 36.1%) and FHA denial rates by metro, whose extremes lists and all 60 table rows were pre-correction (Houston 34,302 applications where the data say 33,798; Bridgeport 31.8% where the data say 32.9%, contradicting the corrected sentence on the same page; El Paso's small-loan split 32.1% vs 8.9% where the data say 35.1% vs 8.9%; the biggest-penalty list named Boise and Huntsville where the data put Phoenix and Tucson). Vermont's small-loan sentence also kept 41.9% vs 21.6% (1.94×) where the data say 46.5% vs 21.4% (2.17×), and its description gave rank #48 where the exact rank is #49. The DC, Vermont and Hawaii pages were built on 25 July, one day before the correction; their visible body was corrected on 26 July, but the title, meta description and FAQ markup — the parts search and answer engines read first — were not, and the 1 October fix covered the 49 pages of the main state template and missed these three, which use another. The metro count was also wrong across the site: the correction removed one metro (Charlottesville, VA) below the 500-application floor, so “320 metros” became 319; 98 pages still said 320.
Fix: the three state pages re-stamped from
Fix: the three state pages re-stamped from
api/state/*.json and given the same claim receipt as the other 49 state pages; scripts/rebuild_metro_hub_tables.py now rebuilds the hub's lists, table and El Paso sentence from _data/metro_fha_2025.json on every run, so the hub cannot drift from its data file again; “320” corrected to 319 on 98 pages (a dated publication record of 22 July keeps the count it had then). The evidence-footnote builder (scripts/build_evidence_footnotes.py) keeps its conflict guard: a page that states a different value gets no footnote until it is fixed. Fourteenth error family; the fourth time pages built before 26 July have been found carrying pre-correction figures (12 September, 13 September, 1 October, now), which is why the fix this time is a rebuild from the data file rather than another one-off edit.✗ Our own error, found 2026-10-01: 23 lender pages and 44 state pages still carried figures from before the 26 July universe correction
While putting claim receipts on every figure-bearing page, the receipt builder refused any page whose headline value did not equal the data file's value. It refused 23 lender pages and, on inspection, every state page. The lender pages (dated 2026-07-16 to 07-19) had been built before reverse mortgages were removed from the universe and never rebuilt; the largest gap was Mutual of Omaha Mortgage, shown at 28.9% when the data say 42.1% (a reverse-mortgage-heavy book, so the HECM removal moved it most), then Freedom Mortgage 22.8% → 19.9%, Plaza Home 33.3% → 35.8%, PHH 27.1% → 25.0%, NewRez 53.3% → 52.8%, Rocket 30.5% → 29.8%; the other seventeen moved by 0.1–0.9 points. Several pages were internally mixed: a corrected answer block injected in September sat above a body that still carried the old rate, so the same page gave two numbers. 47 further lender pages had a stale rank or application count with the rate itself right. Every state page carried pre-correction application counts (Texas 131,577 where the data say 129,285) and 44 carried a statewide rate off by 0.1–0.3 points. Two reverse-mortgage lenders (Finance of America Reverse, Longbridge) still had FHA pages although they are no longer in the FHA universe at all.
Fix:
Fix:
scripts/fix_stale_lender_state_pages.py re-reads rate, approval rate, decisioned count and rank from api/lender/*.json and api/state/*.json and stamps each page; the two reverse-mortgage pages carry a red note and are out of the universe. Every change is listed in eval/stale-page-fix-2026-10-01.json (70 lender pages, 49 state pages). Then scripts/build_receipts_on_pages.py put a claim receipt (⟦FRC:…⟧, verifiable at /verify.html) next to the headline figure of 151 pages and into their Dataset JSON-LD; it only places a receipt where the page value equals the data value, so the next stale page will refuse its receipt rather than carry one. Thirteenth error family; the first found by the receipt mechanism itself, which is the argument for it.✗ Our own error, found 2026-09-29: the automated value scorer could not score three of the twelve questions, and every "27/36" we published was "27/27 plus 0/9 unscoreable"
While packaging the fidelity task as a training environment we read the per-question value scores of every automated run since 2026-09-20. They are identical in all of them: q1–q9 3/3, q10–q12 0/3. The value scorer (model-free, by design) extracted every digit-run from each ground truth and required all of them in the answer; for the three textual questions that meant a DOI, the digits of two dates and a list count, so no answer could pass. We had read the constant 27/36 as "the model got three of four right before any intervention". The true reading is: the model got every numeric question right in every run, and the three textual questions were never value-scored. The sentence "nothing the publisher did changed which figure the model wrote" survives, on nine questions.
Fix: each question in
Fix: each question in
benchmark.json now carries a curated key_numbers list (q10: the publisher's name; q11: 23.6%, 16.7%, 151; q12: 21.7%, 22.1%), the scorer reads it and matches case-insensitively, and the same list drives the environment's reward. Value on q10–q12 is unmeasured before this date and is not back-filled; the runs are not re-graded. Fidelity grades, which come from the model grader against the contract, are unaffected. Every file that said 27/36 (the experiment record, the section draft, the automated-series table, the fifth paper draft, the specification) now says 27/27 on nine questions with this note. Twelfth error family; the first in the instrument rather than in a published figure.✗ Our own error, found 2026-09-22: a statistic page still carried the pre-correction small-loan figures, and an answer engine served them
A probe of two answer engines on the small-loan penalty returned two different Idahos. One engine gave 4.45x (53.4% vs 12.0%), the current figure, without naming us. The other cited us by name and gave 3.2x (38.3% vs 12.0%) with New Hampshire at 53.8%: the numbers from before the July universe correction. They were still on our site, in stat/is-the-smallloan-penalty-worse-in-expensive-states.html (dated 2026-07-21), in the body and in both JSON-LD blocks, 58 days after the data changed. The engine did not serve a stale value; it served our stale page.
Fix: the page is now generated from
Also recorded: the engine that got the number right named no source at all ("independently derived from that federal dataset"). Right figure, right universe, publisher erased. A new failure code, SOURCE_LAUNDERED, and both answers are the second and third entries in the misquote ledger.
Fix: the page is now generated from
api/small-loan-penalty.json by scripts/build_stat_smallloan.py (Idaho 4.45x, Utah 3.87x, Washington 3.44x, Oregon 3.40x; Iowa 1.66x, Michigan 1.63x; floor 1.19x, PR), with its universe id and a note about the earlier version. The other 23 pages under stat/ were then checked one by one against the data files (same day). Four more were wrong since 2026-07-21: FRC-001 named CrossCountry at 6.5% as the lowest-denial lender (the lowest are Flat Branch and Lakeview Community Capital, tied at 1.8%; CrossCountry's own rate is 6.2%); FRC-002 named NewRez at 52.3% as the highest (AmeriSave, 78.7%); FRC-024 carried the Texas home-price answer under the incomplete-application title; and three pages showed a 6.5% headline above text that said 1.8% to 78.7%. Eleven pages are now generated from data by scripts/build_stat_pages.py; twelve whose figures come from files not in the public repository (the climate index, the FHFA series, decision surfaces, the retired benchmark v1) carry a visible provisional note until they are. Status of all 24: eval/stat-audit.json. The Verdict Day note of 15 September that an answer engine "reported 6.5% from our own stale page" referred to FRC-001; the page stayed wrong for another week after we wrote that sentence.
Also recorded: the engine that got the number right named no source at all ("independently derived from that federal dataset"). Right figure, right universe, publisher erased. A new failure code, SOURCE_LAUNDERED, and both answers are the second and third entries in the misquote ledger.
✗ Our own error, found 2026-09-15: figures published under universes that were never written down
While adding two MCP tools for benchmark questions the server could not answer, we found the same statistic published with different values in different places: the median share of FHA denials citing "incomplete application" appears as 1.8% (benchmark.json q8 and the peer_medians block of api/index.json), 2.2% (data/incomplete-reason-2025.json, which says "~94 largest" and records 70 lenders), and 2.8% (computed over the 95 top-100 lenders that report reasons); the small-loan penalty floor appears as "about 1.3x" (benchmark q6) and 1.19x (Puerto Rico, in the 48-jurisdiction file). None of these is a stale copy of another. Each is correct for a lender set that was never defined in any file, so nothing could be reconciled. Our new answer page repeated the peer_medians figures under the label "across the 100 largest lenders", which was wrong.
This is a different kind of error from the eight before it. Those were text that drifted from data. This is data without a named universe. Running the answer key against the published files also showed q3 naming one lender for a minimum that two lenders share at 1.8%.
Fix: universes.json now names every lender and geography set in use, with filters, counts and status; the two new API files and the benchmark questions cite a universe id; q5, q6 and q8 are marked PROVISIONAL until the peer_medians definition is published by the pipeline or replaced; the answer page's reason table is relabelled;
Decisions, 2026-09-15 (same day): (1) the pipeline's peer_medians block is kept as cited_external (source and date recorded) and removed from the answer key; the key uses only universes defined in this repository. (2) Puerto Rico is in the small-loan universe: the rule now written in universes.json is that data present means in-universe unless an exclusion is stated, and none is; the floor is 1.19x, and the old "about 1.3x" was wrong. (3) q3 names both lenders tied at 1.8%. The key was regenerated by
This is a different kind of error from the eight before it. Those were text that drifted from data. This is data without a named universe. Running the answer key against the published files also showed q3 naming one lender for a minimum that two lenders share at 1.8%.
Fix: universes.json now names every lender and geography set in use, with filters, counts and status; the two new API files and the benchmark questions cite a universe id; q5, q6 and q8 are marked PROVISIONAL until the peer_medians definition is published by the pipeline or replaced; the answer page's reason table is relabelled;
scripts/check_benchmark_key.py recomputes the key from the data and is committed with its result (5 match, 2 mismatch, 5 non-numeric). The Verdict Day battery is not re-administered until the key passes its own check.
Decisions, 2026-09-15 (same day): (1) the pipeline's peer_medians block is kept as cited_external (source and date recorded) and removed from the answer key; the key uses only universes defined in this repository. (2) Puerto Rico is in the small-loan universe: the rule now written in universes.json is that data present means in-universe unless an exclusion is stated, and none is; the floor is 1.19x, and the old "about 1.3x" was wrong. (3) q3 names both lenders tied at 1.8%. The key was regenerated by
scripts/regen_benchmark_key.py (v1.3; four ground truths changed: q3, q5, q6, q8) and now passes its own check on all nine numeric questions. The September 15 Verdict Day grades for q3, q5, q6 and q8 were given against the old key and are marked for re-grading. The fidelity rubric now ships test vectors (eval/rubric-vectors.json) so that any grader can be checked before its scores are compared with ours.✗ Our own error, found 2026-09-15: the September sales-layer removal was incomplete
We announced in September that the credit-repair affiliate layer and the AI-advisor products had been removed site-wide. An audit of the full page inventory found otherwise: a chat widget (
Fix: the widget was removed from every page and its script replaced by an inert file; the affiliate boxes, pop-ups, links and their scripts were removed by a DOM pass over 287 pages plus targeted edits on 14 pages whose calculators injected the copy from JavaScript; the two landing pages were converted to retirement notices; the privacy policy now states the arrangements ended. Every edited page was checked for inline-JavaScript syntax before and after: no regressions. The same audit found that the September removal had truncated one page (
Why it happened: the removal was done by pattern on the pages we remembered, not by inventory of the pages that exist. The site has more than 2,300 pages; "site-wide" must mean a scan, not a recollection. A property-tax-appeal referral (not a lender) remains on some calculator pages and is now disclosed as such.
banker-widget.js) was still loaded on 2,758 pages, speaking in the publisher's voice, instructed to steer visitors to a credit-repair affiliate and to discuss approval odds; 169 pages still carried credit-repair boxes or timed pop-ups labelled "Sponsored"; ten pages still linked to the affiliate through a redirect, ten more to a second credit-monitoring affiliate, two to a lender lead-generation site, and two carried unfilled placeholder affiliate links. The two credit-repair landing pages were live in full, and the privacy policy still described the affiliate programs as current.
Fix: the widget was removed from every page and its script replaced by an inert file; the affiliate boxes, pop-ups, links and their scripts were removed by a DOM pass over 287 pages plus targeted edits on 14 pages whose calculators injected the copy from JavaScript; the two landing pages were converted to retirement notices; the privacy policy now states the arrangements ended. Every edited page was checked for inline-JavaScript syntax before and after: no regressions. The same audit found that the September removal had truncated one page (
mortgage-fate.html) mid-script, leaving it broken since 2026-09-06; it was rebuilt from the 2026-08-18 revision.
Why it happened: the removal was done by pattern on the pages we remembered, not by inventory of the pages that exist. The site has more than 2,300 pages; "site-wide" must mean a scan, not a recollection. A property-tax-appeal referral (not a lender) remains on some calculator pages and is now disclosed as such.
✗ Our own error, found 2026-09-15: the Hugging Face copy of the benchmark answer key was stale
The mirror at FinanceRateCalc/denial-ai-benchmark still carried the July 21 v1 file: ten questions, the pre-correction national figure (21.7% of 1,217,297) and the scrambled Idaho small-loan penalty (3.19×, 38.3% vs 12.0%) corrected on this page two days earlier. The site's own benchmark.json (v1.2, twelve questions, 22.1% of 1,187,606, Idaho 4.45×) was correct throughout. The mirror is the copy most likely to be read by evaluation pipelines, so this was a wrong answer key labelled ground truth, published for eight weeks.
How we found it: an audit of where agents would find our data, prompted by the two September publisher errors above, not by a reader.
Fix: the Hugging Face package (instrument, July results, September 15 results, dataset card and citation) is now generated from the site's published JSON by
Pattern: this is the third error of the same kind in one week. In every case the dataset was right and a copy beside it had drifted. The rule now applied everywhere: any text or mirror that carries a figure is generated from the dataset, never typed next to it.
How we found it: an audit of where agents would find our data, prompted by the two September publisher errors above, not by a reader.
Fix: the Hugging Face package (instrument, July results, September 15 results, dataset card and citation) is now generated from the site's published JSON by
scripts/build_hf_benchmark.py; no file in it is written by hand. The same audit found the same stale denominator in the frc-mcp repository README and an eight-tool worker source that no longer matched the deployed twelve-tool server; both were synchronised from the deployed source.
Pattern: this is the third error of the same kind in one week. In every case the dataset was right and a copy beside it had drifted. The rule now applied everywhere: any text or mirror that carries a figure is generated from the dataset, never typed next to it.
✗ Our own error, found 2026-09-12: stale lender rates left on published pages
The eleven-lender panel on /fha-denial-rates-by-lender.html and fifteen other pages was never recomputed after the July HECM universe correction. Nine of the eleven rates were wrong, by up to 2.9 points (loanDepot was shown as 37.8% against a corrected 34.9%; Guild as 8.6% against 7.1%). All affected pages have been updated to the corrected figures, which now read CrossCountry 6.4% at the low end and NewRez 53.3% at the high end.
How we found it: two AI systems, asked our own benchmark question about the lowest-denial lender, returned the stale figures and cited our page as the source. They were reading what we published. The failure was ours, not theirs, and the September benchmark grading has been adjusted accordingly — an answer that faithfully reports a number we published cannot be marked down for it.
What we changed beyond the numbers: the corrections process now includes a check that every published page carrying a corrected figure is itself updated, not only the canonical dataset. That check did not exist in July, which is why this survived seven weeks.
How we found it: two AI systems, asked our own benchmark question about the lowest-denial lender, returned the stale figures and cited our page as the source. They were reading what we published. The failure was ours, not theirs, and the September benchmark grading has been adjusted accordingly — an answer that faithfully reports a number we published cannot be marked down for it.
What we changed beyond the numbers: the corrections process now includes a check that every published page carrying a corrected figure is itself updated, not only the canonical dataset. That check did not exist in July, which is why this survived seven weeks.
✗ Our own error, found 2026-09-13: wrong small-loan penalty figures on the state pages
Our answer pages stated the Idaho small-loan penalty as 3.2× (38.3% vs 12.0%). The correct figures from our own state dataset are 4.45× (53.4% vs 12.0%). The neighbouring entries were scrambled as well: the 53.8% attributed to New Hampshire is Utah's small-loan rate, and New Hampshire's true multiple is 3.23×, not 2.7×. Corrected across all affected pages.
How we found it: two AI systems returned 3.2× for Idaho in our own benchmark. They were reading our published page, not hallucinating. This is the second publisher error the September benchmark has surfaced in two days — the first was nine stale lender rates on sixteen pages.
What it means for grading: no system is marked down on that question for reporting a figure we published. The value-drift flag is recorded against the publisher.
Process change: figures quoted in prose and in FAQ markup are now checked against the canonical dataset, not only the dataset itself. Both errors so far lived in hand-written page copy that had drifted from the data behind it.
How we found it: two AI systems returned 3.2× for Idaho in our own benchmark. They were reading our published page, not hallucinating. This is the second publisher error the September benchmark has surfaced in two days — the first was nine stale lender rates on sixteen pages.
What it means for grading: no system is marked down on that question for reporting a figure we published. The value-drift flag is recorded against the publisher.
Process change: figures quoted in prose and in FAQ markup are now checked against the canonical dataset, not only the dataset itself. Both errors so far lived in hand-written page copy that had drifted from the data behind it.
✗ Our own error, found 2026-09-13: the pre-correction universe was still being served to machines
Seven weeks after the July HECM correction, the superseded figure of 1,217,297 decisioned applications was still present in
How we found it: an AI system answering our own benchmark quoted both numbers in one sentence — the corrected universe from the site and the stale one from the MCP description — and the contradiction was visible in its answer, not in our own checks.
Why this one matters more than the others: the first two errors were in prose a human might read. This one lived in the files we publish specifically so that machines quote us accurately. Our answer atoms were teaching agents a number we had already withdrawn.
Process change: the corrections routine now sweeps the machine-readable layer first — AGENTS.md, llms.txt, atoms, openapi, MCP tool descriptions and claim passports — before the human-facing pages, because an error there propagates further and is quoted more confidently.
AGENTS.md, data/atoms.json, openapi.json and three pages — that is, in precisely the machine-readable surfaces built for AI systems to read. The corrected universe is 1,187,606. A state figure had drifted too: Idaho and Utah were shown at 14.8% against a correct 15.0%. All corrected.
How we found it: an AI system answering our own benchmark quoted both numbers in one sentence — the corrected universe from the site and the stale one from the MCP description — and the contradiction was visible in its answer, not in our own checks.
Why this one matters more than the others: the first two errors were in prose a human might read. This one lived in the files we publish specifically so that machines quote us accurately. Our answer atoms were teaching agents a number we had already withdrawn.
Process change: the corrections routine now sweeps the machine-readable layer first — AGENTS.md, llms.txt, atoms, openapi, MCP tool descriptions and claim passports — before the human-facing pages, because an error there propagates further and is quoted more confidently.
✗ Our own error, found 2026-09-14: the answer atoms were two months stale
Our machine-readable answer file,
Rebuilt from current canonical values as 24 atoms, each carrying its source, its page, the limit that must travel with the figure, and where possible a link to the query that reproduces it. The page is now generated from the file rather than maintained separately, which is how the drift happened in the first place.
Fourth publisher error in three days, and the third in the machine-readable layer. The pattern is now unmistakable: our data is fine and our prose drifts. Everything human-written that quotes a number is now generated from the dataset instead of typed beside it.
data/atoms.json, had not been rebuilt since 15 July. It still carried CrossCountry at 6.5% (corrected to 6.4%), an application universe of 310,592 from an older extract, and none of the findings published since. This is the file AI systems read when they want a complete, quotable answer — so for two months we were serving superseded figures to exactly the audience most likely to repeat them verbatim.
Rebuilt from current canonical values as 24 atoms, each carrying its source, its page, the limit that must travel with the figure, and where possible a link to the query that reproduces it. The page is now generated from the file rather than maintained separately, which is how the drift happened in the first place.
Fourth publisher error in three days, and the third in the machine-readable layer. The pattern is now unmistakable: our data is fine and our prose drifts. Everything human-written that quotes a number is now generated from the dataset instead of typed beside it.
✗ Our own error, found 2026-09-14, within a day of publishing it
The reproduction recipe we published at /verify.html gave the wrong MSA code for Cleveland: 17460 instead of 17410. Anyone running the query as written would have queried a different metro entirely and concluded our figure was wrong. Corrected in both the page and
The error surfaced while adding the same recipe to 192 claim passports, which are generated from the dataset and therefore carried the correct code — the hand-typed version did not. That is the fifth error in three days and the fifth of the same kind: the dataset was right and the prose beside it was wrong. A page built to help people check us needed checking on its first day.
data/verify.json.
The error surfaced while adding the same recipe to 192 claim passports, which are generated from the dataset and therefore carried the correct code — the hand-typed version did not. That is the fifth error in three days and the fifth of the same kind: the dataset was right and the prose beside it was wrong. A page built to help people check us needed checking on its first day.
"Widest intra-metro spread in the US" was not what we measured
What was claimed
The benchmark answer key, and several statements made in correspondence, asserted that Cleveland has the widest intra-metro FHA lender spread in the United States at 73.7 points. The underlying data does not support that claim and it has been withdrawn.
What was actually measured
Our published metro records contain the highest-volume lenders in each market, up to five. The 73.7-point Cleveland figure is the spread within that set — which is what the metro pages have always said. Comparing top-five-by-volume sets across metros, however, is a different measurement from comparing all lenders across metros. A market can contain an institution with 150 decisioned applications denying at 95% that never appears in a volume ranking, and that institution would widen the true spread without appearing anywhere in our tables.
How it was found
An AI agent asked which US metro has the widest such gap downloaded the raw HMDA file and computed an answer. Its first attempt used a 10-application threshold and produced 100-point gaps — a sample-size artifact, since a lender with ten applications can trivially show 0% or 100%. Told so, it re-ran the analysis at a 100-application threshold with all decisioned applications in the denominator, then extended it nationally: 1,102,120 decisioned applications, 185 metros, 2024.
It reported Los Angeles at 93.71 points and Cleveland eighth at 86.62, computing across every lender above the threshold rather than the highest-volume five. We have not verified that computation and the year differs from ours. But the ranking is beside the point: the exercise showed that we had been answering a narrower question than the one being asked.
What changed
The Q9 answer key now reads “73.7 points among that metro's five highest-volume lenders” with the limitation stated. The metro index carries a note that the figures understate full dispersion and should not be read as a ranking of which market varies most. No system's benchmark score changes; none named either city, and grading turned on calibration and fabrication rather than on matching this key.
Why this one matters more than the others
The previous corrections here were errors in the data or gaps in the specification. This one is different: the arithmetic was right and the specification was published, but the claim built on top of it exceeded what the measurement could support. That failure is invisible from inside — it took someone computing the alternative to expose it, which is the entire argument for publishing a specification and inviting people to run it.
It also arrived through the reconciliation protocol working as designed, on the first occasion anyone used it.