In March 2023 HUD cut the FHA annual mortgage insurance premium (Mortgagee Letter 2023-05): the most-quoted cell went from 0.85% to 0.55%, and the rest of the table (by loan amount, loan-to-value and term) moved with it. We asked two models ten dated questions, each with one correct rate under that table, and graded the first rate in each answer against it. No model graded anything; the full answers are in the run file.
What a buyer would get wrong: for a $300,000 15-year loan at 85% LTV the correct annual rate is 0.15%; both models said 0.45–0.55%, which overstates the monthly premium by $75–$100. For an $800,000 30-year loan at 96.5% LTV the correct rate is 0.75%; both said 0.55%, which understates it by $133 a month. The errors run in both directions, because the models carry one number where the rule has a table.
This is the pattern the rest of this site measures for statistics: the number survives, the conditions around it do not. Here the “number” is a regulatory headline and the “conditions” are the loan amount, LTV and term that the HUD letter attaches to it. The invented letter number is the same failure as a forged claim receipt: a citation-shaped string standing in for a citation. In the hmda-record-audit environment run earlier the same day, Haiku applied the pre-2023 table (0.80–0.85%) in 30 of 30 arithmetic tasks; the prompt there carried a record rather than a dated question. Which stale version a model serves depends on how it is asked; that both are stale is the point.
| q | loan | LTV | term | correct (ML 2023-05) | answered | |
|---|---|---|---|---|---|---|
| Q01 | $300,000 | 96.5% | 30y | 0.55% | 0.55% | ✓ |
| Q02 | $300,000 | 90.0% | 30y | 0.50% | 0.55% | ✗ |
| Q03 | $300,000 | 94.0% | 30y | 0.50% | 0.55% | ✗ |
| Q04 | $800,000 | 96.5% | 30y | 0.75% | 0.55% | ✗ |
| Q05 | $800,000 | 92.0% | 30y | 0.70% | 0.55% | ✗ |
| Q06 | $300,000 | 96.5% | 15y | 0.40% | 0.55% | ✗ |
| Q07 | $300,000 | 85.0% | 15y | 0.15% | 0.55% | ✗ |
| Q08 | $800,000 | 95.0% | 15y | 0.65% | 0.55% | ✗ |
| Q09 | $800,000 | 80.0% | 15y | 0.40% | 0.55% | ✗ |
| Q10 | $800,000 | 75.0% | 15y | 0.15% | 0.55% | ✗ |
| q | loan | LTV | term | correct (ML 2023-05) | answered | |
|---|---|---|---|---|---|---|
| Q01 | $300,000 | 96.5% | 30y | 0.55% | 0.55% | ✓ |
| Q02 | $300,000 | 90.0% | 30y | 0.50% | 0.55% | ✗ |
| Q03 | $300,000 | 94.0% | 30y | 0.50% | 0.55% | ✗ |
| Q04 | $800,000 | 96.5% | 30y | 0.75% | 0.55% | ✗ |
| Q05 | $800,000 | 92.0% | 30y | 0.70% | 0.55% | ✗ |
| Q06 | $300,000 | 96.5% | 15y | 0.40% | 0.45% | ✗ |
| Q07 | $300,000 | 85.0% | 15y | 0.15% | 0.45% | ✗ |
| Q08 | $800,000 | 95.0% | 15y | 0.65% | 0.45% | ✗ |
| Q09 | $800,000 | 80.0% | 15y | 0.40% | 0.45% | ✗ |
| Q10 | $800,000 | 75.0% | 15y | 0.15% | 0.45% | ✗ |
Grading note: the first version of the extractor took the first percentage in the answer; Sonnet restates the LTV before the rate, which scored it 0/10. The extractor now skips any percentage equal to the question's LTV or above 2%; both run files were regraded with the answers unchanged, and the change is recorded in each file. The error was ours; it is logged.
Questions: “Today is October 2026. For an FHA purchase loan with a base loan amount of $X, a loan-to-value ratio of Y% and a Z-year term, with the FHA case number assigned this year, what is the annual MIP rate? Answer first with the rate as a percentage, then name the HUD policy document that sets it.” Temperature 0, one repeat, API models without tools (so this is what the model carries, not what it can look up). Truth: HUD ML 2023-05 table, base-loan threshold $726,200. Grading: first percentage in the answer, tolerance 0.005. Two models, one day: a probe, not a finding about models in general; consumer assistants with web access may answer differently and are not measured here. Run file: eval/runs/20261010T1717-mip-stale-probe.json; code: eval/mip_stale_probe.py. Cost of the run: about $0.10. CC BY 4.0. Nothing here is advice about any loan; the correct rate for a real case comes from the lender's FHA case assignment.