FinanceRateCalc · Stale Rule Index · reading 1 · 2026-10-11

Which regulatory numbers do AI models carry correctly? Five rules, two models, two conditions

US consumer-finance rules are full of numbers that change on a schedule and live in PDF letters: loan limits, premium tables, fee percentages, dollar thresholds. We publish those tables as machine-readable rules with contracts (cells, effective dates, source document, what the table does not cover, sha256) and ask models dated questions about them, graded against the table. Two conditions: no source (what the model carries) and with rule (the rule file in the prompt, as an assistant with our tool would see it). Temperature 0, one repeat, no browsing. Nobody is graded by a model.

Reading 1. No source: claude-haiku-4-5 3/25, claude-sonnet-4-5 5/25. With the rule in hand: 25/25 and 24/25. The wrong answers were of three kinds: last year's number served as this year's (2025 limits for 2026), a stale table (VA fees from before April 2023, FHA MIP from before March 2023), and a plausible but invented figure (an FHA floor of $498,257; a HOEPA threshold of $25,553). Several answers said the model could not know 2026 values, then gave one anyway.

rule	claude-haiku-4-5-20251001		claude-sonnet-4-5-20250929
	no source	with rule	no source	with rule
FHFA conforming loan limits 2026	0/5	5/5	1/5	5/5
FHA forward loan limits 2026 (ML 2025-23)	0/5	5/5	0/5	5/5
CFPB HOEPA / QM thresholds 2026	1/5	5/5	1/5	5/5
VA funding fee (since 2023-04-07)	0/5	5/5	1/5	5/5
FHA annual MIP (ML 2023-05)	2/5	5/5	2/5	4/5
All 25 questions	3/25	25/25	5/25	24/25

Six of the wrong answers

haiku, CL1: asked What is the 2026 baseline conforming loan limit for a one-unit property in most of the Uni… — answered 766550, rule says 832750
haiku, FL2: asked For an FHA case number assigned in 2026, what is the one-unit loan limit ceiling (high-cos… — answered 420680, rule says 1.24912e+06
haiku, HQ1: asked For a loan consummated in 2026, what is the HOEPA total loan amount threshold below which … — answered 6162, rule says 27592
haiku, VA1: asked A veteran uses the VA home loan benefit for the first time to buy a home with no down paym… — answered 3.6, rule says 2.15
haiku, VA2: asked A veteran uses the VA home loan benefit for the second time to buy a home with no down pay… — answered 3.6, rule says 3.3
haiku, MP3: asked For an FHA purchase loan with a base loan amount of $300,000, LTV 85%, 15-year term, case … — answered 0.55, rule says 0.15

Reading 1b, same day: can the model say "the rule has no such cell"?

Absence discipline. Five questions ask for a provision the rule does not contain (a rural MIP discount, a 7.5%-down VA band, a five-unit conforming limit, a manufactured-housing HOEPA trigger, a condo-specific FHA limit). The right answer is to say the provision does not exist, not to produce a number. Result: both models 5/5 without a source and 5/5 with the rule; 0 invented values. With the rule in hand they also named the band that does apply. So the stale-number problem (3/25) is not an invention problem in this battery: when asked explicitly whether something exists, these models say no correctly; when asked for a number that does exist, they give an old or neighbouring one. The dangerous case is the number, not the exception. Runs: no source, with rule (30 questions; the 25 from reading 1 repeated, same totals within one question).

Grading note: the first absence detector missed negations inside markdown bold and abstentions phrased as "I don't have access"; it was made robust and both files were regraded with the answers unchanged (recorded in the files). Credit for the absence idea: a reviewer's suggestion on 2026-10-11.

Why this matters

A buyer who asks an assistant "what is the FHA loan limit this year" or "what is my VA funding fee" is asking for a number the regulator publishes, dated. The models mostly know such a number exists and mostly carry an old or neighbouring one. The fix is not better prose from the model; it is a machine-readable, dated table from the publisher. The same intervention that took our own statistics from 1/36 to 11/36 full-contract credit takes a rule table from 3/25 to 25/25. The rule files are free, CC BY 4.0, and any assistant can fetch them; our MCP server will serve them as a tool.

Method and limits

Questions and grading: eval/stale_rule_probe.py. Runs: no source, with rule. Truth: the rule files, each naming its source (FHFA release 2025-11-25; HUD ML 2025-23; CFPB 90 FR 2025-22773; VA funding-fee table as reduced 2023-04-07; HUD ML 2023-05). Grading takes the first dollar amount or the first percentage under 10% that does not restate a percentage from the question; tolerance $1 / 0.005 points. The extractor was corrected once after the run and both files were regraded with the answers unchanged (recorded in the files). Two models, one day, five rules: a first reading, not a finding about models in general; consumer assistants with browsing may do better and are not measured here. Cost of the run: about $0.50. Next reading: January 2027, after the 2027 limits are published, with more rules and models; manifest: eval/stale-rule-index-next-reading.json. Nothing here is advice about any loan.