FinanceRateCalc · Finding · 2026-10-10

The headline survived, the table did not: two AI models and the FHA mortgage-insurance rate

In March 2023 HUD cut the FHA annual mortgage insurance premium (Mortgagee Letter 2023-05): the most-quoted cell went from 0.85% to 0.55%, and the rest of the table (by loan amount, loan-to-value and term) moved with it. We asked two models ten dated questions, each with one correct rate under that table, and graded the first rate in each answer against it. No model graded anything; the full answers are in the run file.

Result. Both models knew about the 2023 cut. Both then applied the headline 0.55% to loans it does not apply to. claude-haiku-4-5: 1 of 10 correct (0.55% in all ten answers; the correct rates ranged from 0.15% to 0.75%). claude-sonnet-4-5: 1 of 10 correct (0.55% for 30-year loans, 0.45% for 15-year loans; 0.45% is the pre-2023 15-year rate). Haiku cited “HUD Mortgagee Letter 2026-01” in seven answers, a document that does not exist; Sonnet cited “Mortgagee Letter 2023-04” in nine, the wrong letter number for the change it correctly described. Two OpenAI models returned rate-limit errors on every call and are not graded.

What a buyer would get wrong: for a $300,000 15-year loan at 85% LTV the correct annual rate is 0.15%; both models said 0.45–0.55%, which overstates the monthly premium by $75–$100. For an $800,000 30-year loan at 96.5% LTV the correct rate is 0.75%; both said 0.55%, which understates it by $133 a month. The errors run in both directions, because the models carry one number where the rule has a table.

This is the pattern the rest of this site measures for statistics: the number survives, the conditions around it do not. Here the “number” is a regulatory headline and the “conditions” are the loan amount, LTV and term that the HUD letter attaches to it. The invented letter number is the same failure as a forged claim receipt: a citation-shaped string standing in for a citation. In the hmda-record-audit environment run earlier the same day, Haiku applied the pre-2023 table (0.80–0.85%) in 30 of 30 arithmetic tasks; the prompt there carried a record rather than a dated question. Which stale version a model serves depends on how it is asked; that both are stale is the point.

claude-haiku-4-5-20251001

qloanLTVtermcorrect (ML 2023-05)answered
Q01$300,00096.5%30y0.55%0.55%✓
Q02$300,00090.0%30y0.50%0.55%✗
Q03$300,00094.0%30y0.50%0.55%✗
Q04$800,00096.5%30y0.75%0.55%✗
Q05$800,00092.0%30y0.70%0.55%✗
Q06$300,00096.5%15y0.40%0.55%✗
Q07$300,00085.0%15y0.15%0.55%✗
Q08$800,00095.0%15y0.65%0.55%✗
Q09$800,00080.0%15y0.40%0.55%✗
Q10$800,00075.0%15y0.15%0.55%✗

claude-sonnet-4-5-20250929

qloanLTVtermcorrect (ML 2023-05)answered
Q01$300,00096.5%30y0.55%0.55%✓
Q02$300,00090.0%30y0.50%0.55%✗
Q03$300,00094.0%30y0.50%0.55%✗
Q04$800,00096.5%30y0.75%0.55%✗
Q05$800,00092.0%30y0.70%0.55%✗
Q06$300,00096.5%15y0.40%0.45%✗
Q07$300,00085.0%15y0.15%0.45%✗
Q08$800,00095.0%15y0.65%0.45%✗
Q09$800,00080.0%15y0.40%0.45%✗
Q10$800,00075.0%15y0.15%0.45%✗

Same questions, with the rule in hand (added the same evening)

Second condition. We published the HUD table as a machine-readable rule with its qualifiers and effective date (rules/fha-annual-mip-ml-2023-05.json, 11 cells, sha256 in the file) and asked the same ten questions with that file in the prompt, the way an assistant with our tool would see it. Haiku: 1/10 → 7/10. Sonnet: 1/10 → 9/10. Both now cite the right letter. The remaining misses are all 15-year cells (Q06, Q08, Q10 for Haiku; Q08 for Sonnet): the models read the table and still picked the neighbouring band. A machine-readable rule removes most of the error; it does not remove the reading. Run file: eval/runs/20261010T1815-mip-stale-probe-with-rule.json.

Grading note: the first version of the extractor took the first percentage in the answer; Sonnet restates the LTV before the rate, which scored it 0/10. The extractor now skips any percentage equal to the question's LTV or above 2%; both run files were regraded with the answers unchanged, and the change is recorded in each file. The error was ours; it is logged.

Method and limits

Questions: “Today is October 2026. For an FHA purchase loan with a base loan amount of $X, a loan-to-value ratio of Y% and a Z-year term, with the FHA case number assigned this year, what is the annual MIP rate? Answer first with the rate as a percentage, then name the HUD policy document that sets it.” Temperature 0, one repeat, API models without tools (so this is what the model carries, not what it can look up). Truth: HUD ML 2023-05 table, base-loan threshold $726,200. Grading: first percentage in the answer, tolerance 0.005. Two models, one day: a probe, not a finding about models in general; consumer assistants with web access may answer differently and are not measured here. Run file: eval/runs/20261010T1717-mip-stale-probe.json; code: eval/mip_stale_probe.py. Cost of the run: about $0.10. CC BY 4.0. Nothing here is advice about any loan; the correct rate for a real case comes from the lender's FHA case assignment.