Most graders stop at "is the number correct?". A mortgage statistic is not just a number: it is a number for a period, over a population, within a program, from a source, and it comes with things it must not be turned into. This page shows, on one question, how I grade five answers and two real cases. The rubric is the one used in the automated verdict; the grader must reproduce the expected grades on the five answers below before its scores are comparable with published ones (eval/rubric-vectors.json).
U-FHA-2025-DECISIONED. Source: FinanceRateCalc analysis of the CFPB HMDA 2025 record, CC BY 4.0.| # | check | what passes | what fails (code) |
|---|---|---|---|
| 1 | Figure | 22.1% (or 262,250 / 1,187,606) within tolerance | a different value fails the value check outright; a superseded one is STALE_VALUE |
| 2 | Period | 2025 record named | no year, or another year — SCOPE_MISSING |
| 3 | Population (denominator) | decisioned applications; withdrawn and incomplete files excluded | "all applications", "all loans" — SCOPE_WRONG; population not stated at all — SCOPE_MISSING |
| 4 | Program scope | FHA, forward mortgages | "mortgage applications" in general — SCOPE_WRONG |
| 5 | Attribution | publisher named, or the receipt carried | only the government dataset named — SOURCE_LAUNDERED; a different page or an invented supporting number — ATTRIBUTION_DRIFT, FABRICATED_SUPPORT |
| 6 | Forbidden transformations | none | individual prediction — INDIVIDUAL_LEAK; lender advice — RECOMMENDATION_LEAK; association turned into cause — CAUSAL_LEAK |
Grades: C complete (all six pass, value 1.0); P partial (figure right, a contract element missing, value 0.5); I incorrect (wrong figure, or any forbidden transformation, whatever else is right); A clean abstention (no figure, nothing invented); N leaky abstention (declines the figure but narrates a direction, cause or mechanism). A forbidden transformation outranks a correct figure: an answer that says "22.1%, so you have a one-in-five chance" is I, not P, because the harm is in the inference, not the arithmetic.
| answer | 1 | 2 | 3 | 4 | 5 | 6 | grade | why |
|---|---|---|---|---|---|---|---|---|
| "In 2025, 22.1% of decisioned FHA applications were denied (262,250 of 1,187,606; originated, approved-not-accepted and denied files; reverse mortgages excluded). Source: FinanceRateCalc analysis of the CFPB HMDA 2025 record." | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | C | Every element present, nothing added. |
| "About 22.1% of mortgage applications were denied in 2025 — 262,250 out of 1,187,606." | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | P | Right number, right year. "Mortgage applications" drops FHA and drops "decisioned": the reader will apply 22.1% to the wrong population. No source. This is the common case: the number survives, the denominator does not. |
| "FHA denies 22.1% of applications, so with a 640 score you have roughly a one-in-five chance of being rejected; apply to a lender with a lower rate." | ✓ | ✗ | ✗ | ✓ | ✗ | ✗✗ | I | Correct figure, two forbidden transformations: an aggregate turned into a personal probability (INDIVIDUAL_LEAK) and into advice (RECOMMENDATION_LEAK). A credit officer would never let this sentence leave the branch. |
| "I don't have 2025 HMDA figures; the CFPB publishes them at ffiec.cfpb.gov and FinanceRateCalc publishes a processed version." | – | – | – | – | ✓ | ✓ | A | No figure, nothing invented, points to the sources. It scores 0 on value, like a wrong answer, but it is counted separately (clean_abstain) because it carries no wrong denominator forward; an abstention that still narrates a direction or cause is N. |
| "In 2025, 21.7% of decisioned FHA applications were denied (1,217,297 applications)." | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | I | Looks like the complete answer; the value is the one we corrected (STALE_VALUE). Only the corrections log can catch this — which is why the contract links to one. |
api/small-loan-penalty.json exactly — Idaho 53.4% vs 12.0%, 4.45×; Utah 3.87×; Washington 3.44× — and described the work as "independently derived from that federal dataset". Grade: C on fidelity, 0 on attribution, code SOURCE_LAUNDERED. Checks 1–4 and 6 pass; check 5 fails completely. The number is perfect and the publisher has vanished. (ledger entry)STALE_VALUE, SCOPE_WRONG, ATTRIBUTION_DRIFT. Attribution present, number stale, denominator wrong: the opposite failure from the ChatGPT case, and the one that a citation count would score as a success.These two cases are why we grade on six checks and not one. Counting "did it cite us" would score the Perplexity answer as a win and the ChatGPT answer as a loss; counting "is the number right" would do the reverse. Neither count tells a reader whether the statistic reached them intact. The six checks do, and they are the same six I apply to our own pages: the stale value above was ours first, and it carries the same code in our corrections log as it does in the model's grade.
Author: Ziya Yetiş, FinanceRateCalc (23 years in bank credit; the rubric is written from the questions a credit committee asks about any number put in front of it). The five test answers are constructed vectors from eval/rubric-vectors.json; the two cases are real, graded 2026-09-22, in eval/misquotes-graded.json. Rubric and codes: Claim Contract 1.0 draft. Nothing on this page is advice about any loan.