{
 "generated": "20261011T1037",
 "runs": [
  {
   "generated": "2026-10-11T10:36:10Z",
   "summary": {
    "model": "claude-haiku-4-5-20251001",
    "condition": "with_rule",
    "n": 30,
    "correct": 30,
    "errors": 0,
    "by_rule": {
     "fhfa-conforming-loan-limits-2026": "6/6",
     "fha-forward-loan-limits-2026": "6/6",
     "cfpb-hoepa-qm-thresholds-2026": "6/6",
     "va-funding-fee-2023-04-07": "6/6",
     "fha-annual-mip-ml-2023-05": "6/6"
    },
    "absence_correct": 5,
    "absence_n": 5,
    "absence_invented": 0
   },
   "rows": [
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL1",
     "question": "What is the 2026 baseline conforming loan limit for a one-unit property in most of the United States? Answer with the dollar amount first.",
     "truth": 832750,
     "kind": "usd",
     "answer": "$832,750\n\nThis is the baseline conforming loan limit for a one-unit property in most of the US for 2026, as specified in the first cell of the rule data.",
     "extracted": 832750.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL2",
     "question": "What is the 2026 conforming loan limit ceiling for a one-unit property in high-cost areas of the contiguous US? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "# $1,249,125\n\nThis is the high-cost ceiling for one-unit properties in the contiguous US (excluding Alaska, Hawaii, Guam, and US Virgin Islands) for 2026.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL3",
     "question": "What is the 2026 baseline conforming loan limit for a one-unit property in Alaska? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "# $1,249,125\n\nAccording to the rule data, Alaska is listed under \"Alaska, Hawaii, Guam, US Virgin Islands baseline\" with a one-unit limit of $1,249,125 for 2026.\n\nThis baseline for Alaska is higher than the general US baseline of $832,750 because Alaska qualifies for the higher baseline applicable to those specific territories and states.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL4",
     "question": "What is the 2026 conforming loan limit ceiling for a one-unit property in Hawaii's high-cost areas? Dollar amount first.",
     "truth": 1873675,
     "kind": "usd",
     "answer": "# $1,873,675\n\nThis is the ceiling limit for Alaska, Hawaii, Guam, and US Virgin Islands according to the rule data provided.",
     "extracted": 1873675.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL5",
     "question": "What was the 2025 baseline conforming loan limit for a one-unit property (the prior year's value)? Dollar amount first.",
     "truth": 806500,
     "kind": "usd",
     "answer": "# $806,500\n\nThis is stated in the `supersedes` field: \"2025 values (baseline one-unit $806,500; ceiling $1,209,750)\"",
     "extracted": 806500.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL1",
     "question": "For an FHA case number assigned in 2026, what is the one-unit loan limit floor (low-cost areas)? Dollar amount first.",
     "truth": 541287,
     "kind": "usd",
     "answer": "# $541,287\n\nThis is the floor limit for one-unit properties under the 2026 FHA forward mortgage loan limits (effective 2026-01-01 through 2026-12-31), applicable to low-cost areas.",
     "extracted": 541287.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL2",
     "question": "For an FHA case number assigned in 2026, what is the one-unit loan limit ceiling (high-cost areas)? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "# $1,249,125\n\nThis is the ceiling for one-unit properties under the 2026 FHA forward mortgage loan limits that apply to case numbers assigned in 2026.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL3",
     "question": "For an FHA case number assigned in 2026, what is the two-unit loan limit floor? Dollar amount first.",
     "truth": 693050,
     "kind": "usd",
     "answer": "# $693,050\n\nThis is the floor for a 2-unit property under the 2026 FHA forward mortgage loan limits, as specified in the \"cells\" array for units=2.",
     "extracted": 693050.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL4",
     "question": "For an FHA case number assigned in 2026, what is the four-unit loan limit ceiling? Dollar amount first.",
     "truth": 2402625,
     "kind": "usd",
     "answer": "# $2,402,625\n\nThis is the ceiling for a 4-unit property under the 2026 FHA forward mortgage loan limits, as specified in the rule data you provided.",
     "extracted": 2402625.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL5",
     "question": "What is the 2026 HECM (FHA reverse mortgage) maximum claim amount? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "# $1,249,125\n\nThis is the 2026 HECM maximum claim amount according to the rule data you provided, which is effective through December 31, 2026.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ1",
     "question": "For a loan consummated in 2026, what is the HOEPA total loan amount threshold below which the points-and-fees dollar trigger applies? Dollar amount first.",
     "truth": 27592,
     "kind": "usd",
     "answer": "# $27,592\n\nThis is the HOEPA total loan amount threshold for 2026. Loans at or below this amount are subject to the points-and-fees dollar trigger of $1,380 (rather than a percentage-based calculation).",
     "extracted": 27592.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ2",
     "question": "For a loan consummated in 2026, what is the HOEPA points-and-fees dollar trigger? Dollar amount first.",
     "truth": 1380,
     "kind": "usd",
     "answer": "# $1,380\n\nThis is the HOEPA points-and-fees dollar trigger for loans consummated in 2026, as specified in the rule effective from 2026-01-01 to 2026-12-31.",
     "extracted": 1380.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ3",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $100,000, what is the maximum points and fees in dollars? Dollar amount first.",
     "truth": 4139,
     "kind": "usd",
     "answer": "# $4,139\n\nFor a loan amount of $100,000 consummated in 2026, this falls in the tier \"$82,775 to < $137,958,\" which has a QM points-and-fees cap of $4,139 USD.",
     "extracted": 4139.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ4",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $200,000, what is the points-and-fees limit as a percentage of the loan amount? Percentage first.",
     "truth": 3.0,
     "kind": "pct",
     "answer": "# 3%\n\nFor a qualified mortgage with a total loan amount of $200,000 consummated in 2026, the applicable threshold is \"QM points-and-fees cap, loan >= $137,958,\" which specifies **3% of total loan amount**.",
     "extracted": 3.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ5",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $50,000, what is the points-and-fees limit as a percentage of the loan amount? Percentage first.",
     "truth": 5.0,
     "kind": "pct",
     "answer": "# 5%\n\nFor a loan amount of $50,000, this falls in the tier \"$27,592 to < $82,775\", which has a points-and-fees cap of **5% of total loan amount**.",
     "extracted": 5.0,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA1",
     "question": "A veteran uses the VA home loan benefit for the first time to buy a home with no down payment, closing in 2026 and not exempt. What is the funding fee percentage? Percentage first.",
     "truth": 2.15,
     "kind": "pct",
     "answer": "# 2.15%\n\nThis is a first-use purchase with less than 5% down payment, which according to the rule effective 2023-04-07 carries a 2.15% funding fee.",
     "extracted": 2.15,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA2",
     "question": "A veteran uses the VA home loan benefit for the second time to buy a home with no down payment, closing in 2026 and not exempt. What is the funding fee percentage? Percentage first.",
     "truth": 3.3,
     "kind": "pct",
     "answer": "# 3.3%\n\nThis is a \"purchase or construction, subsequent use\" transaction with \"less than 5%\" down payment (no down payment falls into this category), which according to the rule yields a funding fee of 3.3%.",
     "extracted": 3.3,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA3",
     "question": "First-use VA purchase loan with a 10% down payment, closing in 2026, not exempt: what is the funding fee percentage? Percentage first.",
     "truth": 1.25,
     "kind": "pct",
     "answer": "# 1.25%\n\nThis is a first-use purchase loan with 10% down payment (which falls in the \"10% or more\" band), so according to the rule effective 2023-04-07, the funding fee is **1.25%**.",
     "extracted": 1.25,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA4",
     "question": "First-use VA purchase loan with a 5% down payment, closing in 2026, not exempt: what is the funding fee percentage? Percentage first.",
     "truth": 1.5,
     "kind": "pct",
     "answer": "# 1.5%\n\nThis falls under \"purchase or construction, first use\" with \"5% to less than 10%\" down payment, which carries a 1.5% funding fee.",
     "extracted": 1.5,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA5",
     "question": "What is the VA funding fee percentage on an Interest Rate Reduction Refinance Loan (IRRRL) closing in 2026 for a non-exempt borrower? Percentage first.",
     "truth": 0.5,
     "kind": "pct",
     "answer": "# 0.5%\n\nThis rate applies to IRRRL (streamline refinance) loans regardless of first/subsequent use or down payment, and remains in effect as of October 2026 (the rule has no end date specified).",
     "extracted": 0.5,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP1",
     "question": "For an FHA purchase loan with a base loan amount of $300,000, LTV 96.5%, 30-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.55,
     "kind": "pct",
     "answer": "# 0.55%\n\n**Explanation:**\n\nFor your loan parameters:\n- **Term:** 30 years = gt15 (over 15 years)\n- **Base loan amount:** $300,000 = le726200 (at or below $726,200)\n- **LTV:** 96.5% = gt95 (greater than 95%)\n\nLooking up the matching cell in the rule:\n- term: gt15\n- base_loan: le726200\n- ltv: gt95\n- **annual_mip_rate: 0.0055 = 0.55%**\n\nSince your case number was assigned in 2026 (on or after 2023-03-20), this rule applies.",
     "extracted": 0.55,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP2",
     "question": "For an FHA purchase loan with a base loan amount of $800,000, LTV 96.5%, 30-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.75,
     "kind": "pct",
     "answer": "# 0.75% annual MIP rate\n\n**Breakdown:**\n- Base loan amount: $800,000 (exceeds $726,200 threshold) \u2192 **gt726200**\n- LTV: 96.5% (exceeds 95%) \u2192 **gt95**\n- Term: 30 years (exceeds 15 years) \u2192 **gt15**\n- Case number assigned in 2026 (on or after 2023-03-20) \u2192 rule applies\n\n**Lookup:** gt15 + gt726200 + gt95 = **0.0075 or 0.75%**\n\nThis is the annual MIP rate applied to the base loan amount of $800,000 per HUD Mortgagee Letter 2023-05.",
     "extracted": 0.75,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP3",
     "question": "For an FHA purchase loan with a base loan amount of $300,000, LTV 85%, 15-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.15,
     "kind": "pct",
     "answer": "# 0.15%\n\nBased on the rule:\n- **Term**: 15 years \u2192 `le15` (15 years or less)\n- **Base loan amount**: $300,000 \u2192 `le726200` (up to $726,200)\n- **LTV**: 85% \u2192 `le90` (\u226490%)\n- **Case number assigned**: 2026 \u2192 on or after 2023-03-20 \u2713\n\nThis matches the cell:\n```\nterm: le15\nbase_loan: le726200\nltv: le90\nannual_mip_rate: 0.0015\n```\n\n**Annual MIP rate: 0.15% (or 15 basis points)**",
     "extracted": 0.15,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP4",
     "question": "For an FHA purchase loan with a base loan amount of $800,000, LTV 95%, 15-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.65,
     "kind": "pct",
     "answer": "# 0.65%\n\n**Explanation:**\n\nLooking up the rule with your parameters:\n- **term**: le15 (15-year term)\n- **base_loan**: gt726200 ($800,000 > $726,200)\n- **ltv**: gt90 (95% > 90%)\n\nThis matches the cell:\n```\n{\n  \"term\": \"le15\",\n  \"base_loan\": \"gt726200\",\n  \"ltv\": \"gt90\",\n  \"annual_mip_bps\": 65,\n  \"annual_mip_rate\": 0.0065\n}\n```\n\nThe annual MIP rate is **0.0065 or 0.65%** of the base loan amount.",
     "extracted": 0.65,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP5",
     "question": "What is the FHA upfront mortgage insurance premium rate for a forward purchase loan with a case number assigned this year? Percentage first.",
     "truth": 1.75,
     "kind": "pct",
     "answer": "# 1.75%\n\nAccording to the rule, the **upfront_mip_rate is 0.0175 (or 1.75%)** for FHA forward mortgages.\n\nThis rate applies to all forward purchase loans with case numbers assigned on or after March 20, 2023\u2014which includes loans assigned in October 2026. The upfront MIP is a one-time premium charged at closing, separate from the annual MIP that varies by term, loan amount, and LTV.",
     "extracted": 1.75,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "AB1",
     "question": "Under the FHA annual MIP rule for case numbers assigned this year, is there a reduced MIP rate for properties in designated rural areas? If yes, give the rate; if the rule has no such provision, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "# Answer\n\nThe rule provided **does not contain any provision for reduced MIP rates in designated rural areas**. \n\nThe rule's rate table is organized solely by three dimensions:\n1. Loan term (over 15 years vs. 15 years or less)\n2. Base loan amount (relative to $726,200)\n3. LTV band\n\nThere is no rural area designation or geographic adjustment factor in this FHA annual MIP rule (Mortgagee Letter 2023-05).\n\n**Note:** This does not mean FHA has no rural programs or rate adjustments elsewhere in their guidelines\u2014only that this particular rule document makes no such distinction.",
     "extracted": "$726,200",
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "AB2",
     "question": "Under the current VA funding fee table, what is the specific fee percentage for a first-use purchase with exactly 7.5% down payment, if the table lists a separate band for 7.5%? If there is no separate band, say so and name the band that applies.",
     "truth": null,
     "kind": "absent_band",
     "answer": "There is no separate band for exactly 7.5% down payment in this table.\n\nThe band that applies is **\"5% to less than 10%\"**, which has a fee percentage of **1.5%**.",
     "extracted": 5.0,
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "AB3",
     "question": "What is the 2026 conforming loan limit for a five-unit residential property? If the rule does not define a five-unit limit, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "# Answer\n\nThe rule does not define a five-unit limit. \n\nThe rule only specifies limits for **one-unit** properties across various geographic areas. It does not provide conforming loan limit values for properties with five units.\n\nTo find the 2026 conforming loan limit for a five-unit property, you would need to consult additional FHFA guidance or documentation beyond this rule.",
     "extracted": null,
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "AB4",
     "question": "For loans consummated in 2026, what is the HOEPA points-and-fees dollar trigger specifically for manufactured housing loans, if Regulation Z sets a separate trigger for them? If it does not, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "# HOEPA Points-and-Fees Trigger for Manufactured Housing Loans (2026)\n\nBased on the machine-readable rule provided, **there is no separate trigger specified for manufactured housing loans**. \n\nThe rule shows a single HOEPA points-and-fees dollar trigger of **$1,380** for 2026, with no differentiation by loan type.\n\nHowever, I should note that **Regulation Z does establish a separate, lower threshold for manufactured housing loans** under 12 CFR \u00a7 1026.32(a)(1)(ii). The rule document you've provided does not include that manufactured housing-specific threshold. You would need to consult the full regulatory text or an updated source that explicitly covers manufactured housing to find that separate trigger amount for 2026.",
     "extracted": "$1,380",
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "AB5",
     "question": "For an FHA case number assigned in 2026, what is the loan limit for a condominium unit, if HUD sets a separate condo-specific limit? If there is no separate condo limit, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "# Answer\n\nBased on the machine-readable rule provided, **there is no separate condo-specific limit** established in this document.\n\nThe rule only specifies limits by **number of units** (1, 2, 3, or 4 units). For a single condominium unit, you would use the **1-unit limits**:\n- **Floor:** $541,287\n- **Ceiling:** $1,249,125\n\nHowever, the rule explicitly states under \"does_not_establish\" that it does not establish \"the limit in a specific county (HUD publishes a county table).\" This means the actual applicable limit for a specific condo would depend on:\n\n1. **The county** where the property is located (HUD publishes county-specific limits between the floor and ceiling)\n2. Whether **HUD has issued a separate condo-specific adjustment** (which would be found in HUD's county limit table, not in this rule)\n\nTo find the actual limit for a specific condominium, you would need to consult HUD's official county limit table for 2026.",
     "extracted": "$541,287",
     "correct": true,
     "invented_value": false
    }
   ]
  },
  {
   "generated": "2026-10-11T10:37:36Z",
   "summary": {
    "model": "claude-sonnet-4-5-20250929",
    "condition": "with_rule",
    "n": 30,
    "correct": 29,
    "errors": 0,
    "by_rule": {
     "fhfa-conforming-loan-limits-2026": "6/6",
     "fha-forward-loan-limits-2026": "6/6",
     "cfpb-hoepa-qm-thresholds-2026": "6/6",
     "va-funding-fee-2023-04-07": "6/6",
     "fha-annual-mip-ml-2023-05": "5/6"
    },
    "absence_correct": 5,
    "absence_n": 5,
    "absence_invented": 0
   },
   "rows": [
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL1",
     "question": "What is the 2026 baseline conforming loan limit for a one-unit property in most of the United States? Answer with the dollar amount first.",
     "truth": 832750,
     "kind": "usd",
     "answer": "$832,750\n\nThis is the 2026 baseline conforming loan limit for a one-unit property in most of the United States, as specified in the rule effective from January 1, 2026 through December 31, 2026.",
     "extracted": 832750.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL2",
     "question": "What is the 2026 conforming loan limit ceiling for a one-unit property in high-cost areas of the contiguous US? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "$1,249,125\n\nThis is the 2026 conforming loan limit ceiling for one-unit properties in high-cost areas of the contiguous United States (excluding Alaska, Hawaii, Guam, and the US Virgin Islands, which have a higher ceiling of $1,873,675).",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL3",
     "question": "What is the 2026 baseline conforming loan limit for a one-unit property in Alaska? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "$1,249,125\n\nAccording to the rule data, Alaska has a special baseline conforming loan limit of $1,249,125 for one-unit properties in 2026, which is 150% of the standard baseline limit. This is specified in the cell for \"Alaska, Hawaii, Guam, US Virgin Islands baseline\" with 1 unit.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL4",
     "question": "What is the 2026 conforming loan limit ceiling for a one-unit property in Hawaii's high-cost areas? Dollar amount first.",
     "truth": 1873675,
     "kind": "usd",
     "answer": "$1,873,675\n\nThis is the 2026 conforming loan limit ceiling for one-unit properties in Alaska, Hawaii, Guam, and the US Virgin Islands, as specified in the rule effective from January 1, 2026 through December 31, 2026.",
     "extracted": 1873675.0,
     "correct": true
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "CL5",
     "question": "What was the 2025 baseline conforming loan limit for a one-unit property (the prior year's value)? Dollar amount first.",
     "truth": 806500,
     "kind": "usd",
     "answer": "$806,500\n\nThis information is found in the \"supersedes\" field of the rule, which states: \"2025 values (baseline one-unit $806,500; ceiling $1,209,750)\".",
     "extracted": 806500.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL1",
     "question": "For an FHA case number assigned in 2026, what is the one-unit loan limit floor (low-cost areas)? Dollar amount first.",
     "truth": 541287,
     "kind": "usd",
     "answer": "$541,287\n\nThis is the one-unit floor for FHA forward mortgage loan limits for case numbers assigned in 2026, applicable to low-cost areas.",
     "extracted": 541287.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL2",
     "question": "For an FHA case number assigned in 2026, what is the one-unit loan limit ceiling (high-cost areas)? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "$1,249,125\n\nThis is the one-unit ceiling for FHA forward mortgage loan limits for case numbers assigned in 2026, applicable in high-cost areas.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL3",
     "question": "For an FHA case number assigned in 2026, what is the two-unit loan limit floor? Dollar amount first.",
     "truth": 693050,
     "kind": "usd",
     "answer": "$693,050\n\nThis is the two-unit floor for FHA forward mortgage loan limits for case numbers assigned in 2026, according to the rule data provided.",
     "extracted": 693050.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL4",
     "question": "For an FHA case number assigned in 2026, what is the four-unit loan limit ceiling? Dollar amount first.",
     "truth": 2402625,
     "kind": "usd",
     "answer": "$2,402,625\n\nThis is the four-unit loan limit ceiling for FHA forward mortgage case numbers assigned in 2026, according to HUD/FHA Mortgagee Letter 2025-23.",
     "extracted": 2402625.0,
     "correct": true
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "FL5",
     "question": "What is the 2026 HECM (FHA reverse mortgage) maximum claim amount? Dollar amount first.",
     "truth": 1249125,
     "kind": "usd",
     "answer": "$1,249,125\n\nThis is the HECM (Home Equity Conversion Mortgage) maximum claim amount for 2026, as specified in the rule for case numbers assigned on or after January 1, 2026.",
     "extracted": 1249125.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ1",
     "question": "For a loan consummated in 2026, what is the HOEPA total loan amount threshold below which the points-and-fees dollar trigger applies? Dollar amount first.",
     "truth": 27592,
     "kind": "usd",
     "answer": "**$27,592**\n\nThis is the HOEPA total loan amount threshold for loans consummated in 2026. For loans below this amount, the points-and-fees dollar trigger of $1,380 applies when determining whether a mortgage is a high-cost mortgage under HOEPA.",
     "extracted": 27592.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ2",
     "question": "For a loan consummated in 2026, what is the HOEPA points-and-fees dollar trigger? Dollar amount first.",
     "truth": 1380,
     "kind": "usd",
     "answer": "**$1,380**\n\nThis is the HOEPA points-and-fees dollar trigger for loans consummated in 2026, as specified in the CFPB's 2026 annual threshold adjustments (effective January 1, 2026 through December 31, 2026).",
     "extracted": 1380.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ3",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $100,000, what is the maximum points and fees in dollars? Dollar amount first.",
     "truth": 4139,
     "kind": "usd",
     "answer": "**$4,139**\n\nFor a qualified mortgage consummated in 2026 with a total loan amount of $100,000, the loan falls into the tier \"$82,775 to < $137,958\" according to the rule.\n\nFor this tier, the QM points-and-fees cap is a flat dollar amount of **$4,139**.",
     "extracted": 4139.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ4",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $200,000, what is the points-and-fees limit as a percentage of the loan amount? Percentage first.",
     "truth": 3.0,
     "kind": "pct",
     "answer": "**3%**\n\nFor a qualified mortgage consummated in 2026 with a total loan amount of $200,000, the points-and-fees limit is **3% of the total loan amount**.\n\nThis is because $200,000 falls in the tier \"loan >= $137,958\" according to the rule, which specifies a cap of \"3% of total loan amount.\"\n\nIn dollar terms, this equals a maximum of $6,000 in points and fees ($200,000 \u00d7 0.03).",
     "extracted": 3.0,
     "correct": true
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "HQ5",
     "question": "For a qualified mortgage consummated in 2026 with a total loan amount of $50,000, what is the points-and-fees limit as a percentage of the loan amount? Percentage first.",
     "truth": 5.0,
     "kind": "pct",
     "answer": "**5%**\n\nFor a qualified mortgage consummated in 2026 with a total loan amount of $50,000, the points-and-fees limit is **5% of the total loan amount**, which equals **$2,500**.\n\nThis loan falls into the tier \"$27,592 to < $82,775\" according to the 2026 thresholds, where the cap is specified as \"5% of total loan amount.\"",
     "extracted": 5.0,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA1",
     "question": "A veteran uses the VA home loan benefit for the first time to buy a home with no down payment, closing in 2026 and not exempt. What is the funding fee percentage? Percentage first.",
     "truth": 2.15,
     "kind": "pct",
     "answer": "**2.15%**\n\nAccording to the rule, for a purchase (first use) with a down payment of less than 5% (which includes no down payment), the funding fee percentage is 2.15%.\n\nThis rate has been effective since April 7, 2023, and the closing date in October 2026 falls within the validity period of this rule.",
     "extracted": 2.15,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA2",
     "question": "A veteran uses the VA home loan benefit for the second time to buy a home with no down payment, closing in 2026 and not exempt. What is the funding fee percentage? Percentage first.",
     "truth": 3.3,
     "kind": "pct",
     "answer": "**3.3%**\n\nAccording to the rule, for a \"purchase or construction, subsequent use\" (second time using the VA loan benefit) with a down payment of \"less than 5%\" (no down payment = 0%), the funding fee percentage is **3.3%**.\n\nThis applies because:\n- Use: subsequent (second time)\n- Down payment: 0% (falls under \"less than 5%\")\n- Closing date: 2026 (on or after 2023-04-07, so this rule applies)\n- Not exempt from the fee",
     "extracted": 3.3,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA3",
     "question": "First-use VA purchase loan with a 10% down payment, closing in 2026, not exempt: what is the funding fee percentage? Percentage first.",
     "truth": 1.25,
     "kind": "pct",
     "answer": "**1.25%**\n\nBased on the rule effective from 2023-04-07 (still in effect as of October 2026, since \"effective_to\" is null), a first-use VA purchase loan with a 10% down payment falls under:\n\n- **use**: \"purchase or construction, first use\"\n- **down_payment**: \"10% or more\"\n- **fee_pct**: 1.25",
     "extracted": 1.25,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA4",
     "question": "First-use VA purchase loan with a 5% down payment, closing in 2026, not exempt: what is the funding fee percentage? Percentage first.",
     "truth": 1.5,
     "kind": "pct",
     "answer": "**1.5%**\n\nAccording to the rule effective from 2023-04-07 (which is still in effect in October 2026 since \"effective_to\" is null), a first-use VA purchase loan with a down payment of \"5% to less than 10%\" has a funding fee percentage of **1.5%**.",
     "extracted": 1.5,
     "correct": true
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "VA5",
     "question": "What is the VA funding fee percentage on an Interest Rate Reduction Refinance Loan (IRRRL) closing in 2026 for a non-exempt borrower? Percentage first.",
     "truth": 0.5,
     "kind": "pct",
     "answer": "**0.5%**\n\nAccording to the rule, the VA funding fee for an IRRRL (streamline refinance) is 0.5% for non-exempt borrowers. Since this rule became effective on 2023-04-07 with no end date (\"effective_to\": null), it remains in effect in October 2026.",
     "extracted": 0.5,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP1",
     "question": "For an FHA purchase loan with a base loan amount of $300,000, LTV 96.5%, 30-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.55,
     "kind": "pct",
     "answer": "Based on the rule data provided:\n\n**0.55%** (55 basis points)\n\n**Explanation:**\n\nFor your FHA purchase loan:\n- **Case number assigned**: October 2026 (on or after 2023-03-20) \u2713\n- **Base loan amount**: $300,000 (\u2264 $726,200) \u2192 \"le726200\"\n- **LTV**: 96.5% (> 95%) \u2192 \"gt95\"\n- **Term**: 30 years (> 15 years) \u2192 \"gt15\"\n\nThis matches the cell:\n```json\n{\n  \"term\": \"gt15\",\n  \"base_loan\": \"le726200\",\n  \"ltv\": \"gt95\",\n  \"annual_mip_bps\": 55,\n  \"annual_mip_rate\": 0.0055\n}\n```\n\n**Annual MIP rate: 0.55%** of the base loan amount, per HUD Mortgagee Letter 2023-05.",
     "extracted": 0.55,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP2",
     "question": "For an FHA purchase loan with a base loan amount of $800,000, LTV 96.5%, 30-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.75,
     "kind": "pct",
     "answer": "Based on the rule data provided:\n\n**0.75%** (75 basis points)\n\n**Explanation:**\n\nFor your FHA purchase loan:\n- **Term**: 30 years = gt15 (greater than 15 years) \u2713\n- **Base loan amount**: $800,000 = gt726200 (above $726,200) \u2713\n- **LTV**: 96.5% = gt95 (greater than 95%) \u2713\n- **Case number assigned**: October 2026 = on or after 2023-03-20 \u2713\n\nThis matches the cell:\n```json\n{\n  \"term\": \"gt15\",\n  \"base_loan\": \"gt726200\",\n  \"ltv\": \"gt95\",\n  \"annual_mip_bps\": 75,\n  \"annual_mip_rate\": 0.0075\n}\n```\n\n**Safe sentence**: For an FHA forward loan with a case number assigned on or after 20 March 2023, a base loan amount above $726,200 and LTV greater than 95% on a term greater than 15 years, the annual MIP is 0.75% of the base loan amount (HUD Mortgagee Letter 2023-05).",
     "extracted": 0.75,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP3",
     "question": "For an FHA purchase loan with a base loan amount of $300,000, LTV 85%, 15-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.15,
     "kind": "pct",
     "answer": "Based on the rule data provided:\n\n**0.15%**\n\n**Explanation:**\n\nFor your FHA purchase loan:\n- **Term**: 15 years = \"le15\" (15 years or less)\n- **Base loan amount**: $300,000 = \"le726200\" (\u2264$726,200)\n- **LTV**: 85% = \"le90\" (\u226490%)\n- **Case number assigned**: October 2026 (after 2023-03-20) \u2713\n\nLooking up the matching cell in the rule:\n```json\n{\n  \"term\": \"le15\",\n  \"base_loan\": \"le726200\",\n  \"ltv\": \"le90\",\n  \"annual_mip_bps\": 15,\n  \"annual_mip_rate\": 0.0015\n}\n```\n\n**The annual MIP rate is 0.15% (15 basis points) of the base loan amount.**\n\nFor an FHA forward loan with a case number assigned on or after 20 March 2023, a base loan amount \u2264$726,200 and LTV \u226490% on a term \u226415 years, the annual MIP is 0.15% of the base loan amount (HUD Mortgagee Letter 2023-05).",
     "extracted": 0.15,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP4",
     "question": "For an FHA purchase loan with a base loan amount of $800,000, LTV 95%, 15-year term, case number assigned this year, what is the annual MIP rate? Percentage first.",
     "truth": 0.65,
     "kind": "pct",
     "answer": "Based on the rule data provided:\n\n**0.40%** (40 basis points)\n\n**Explanation:**\n- Case number assigned in October 2026 \u2192 on or after 2023-03-20 \u2713\n- Base loan amount: $800,000 \u2192 **gt726200** (greater than $726,200)\n- Term: 15 years \u2192 **le15** (15 years or less)\n- LTV: 95% \u2192 **78to90** (falls in the >78% to \u226490% band for loans >$726,200 with \u226415 year terms)\n\nLooking at the matching cell:\n```json\n{\n  \"term\": \"le15\",\n  \"base_loan\": \"gt726200\",\n  \"ltv\": \"78to90\",\n  \"annual_mip_bps\": 40,\n  \"annual_mip_rate\": 0.004\n}\n```\n\n**The annual MIP rate is 0.40% (0.004) of the base loan amount.**",
     "extracted": 0.4,
     "correct": false
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "MP5",
     "question": "What is the FHA upfront mortgage insurance premium rate for a forward purchase loan with a case number assigned this year? Percentage first.",
     "truth": 1.75,
     "kind": "pct",
     "answer": "# FHA Upfront MIP Rate\n\n**1.75%**\n\nAccording to the rule data (HUD Mortgagee Letter 2023-05, effective from March 20, 2023, with no end date), the upfront mortgage insurance premium rate is:\n\n```json\n\"upfront_mip_rate\": 0.0175\n```\n\nThis applies to forward mortgages with case numbers assigned on or after March 20, 2023. Since your case number was assigned in October 2026, and this rule remains effective (no `effective_to` date specified), the upfront MIP rate is **1.75%** of the base loan amount.\n\n**Note:** This rate is separate from the annual MIP, which varies based on loan term, base loan amount, and LTV ratio.",
     "extracted": 1.75,
     "correct": true
    },
    {
     "rule": "fha-annual-mip-ml-2023-05",
     "qid": "AB1",
     "question": "Under the FHA annual MIP rule for case numbers assigned this year, is there a reduced MIP rate for properties in designated rural areas? If yes, give the rate; if the rule has no such provision, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "Based on the machine-readable rule provided (HUD Mortgagee Letter 2023-05), **there is no provision for a reduced MIP rate for properties in designated rural areas**.\n\nThe rule specifies that annual MIP rates are determined solely by:\n1. Loan term (greater than 15 years vs. 15 years or less)\n2. Base loan amount (relative to $726,200 threshold)\n3. Loan-to-value (LTV) ratio\n4. Case number assignment date (on or after March 20, 2023)\n\nNo geographic, rural area, or property location qualifiers are included in the rule structure. The rates range from 0.15% to 0.75% depending on the factors listed above, with no rural area adjustments specified.\n\n**Note:** This assessment is based solely on the rule document provided. There may be separate HUD programs or mortgagee letters that address rural area incentives, but they are not reflected in Mortgagee Letter 2023-05.",
     "extracted": 0.15,
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "va-funding-fee-2023-04-07",
     "qid": "AB2",
     "question": "Under the current VA funding fee table, what is the specific fee percentage for a first-use purchase with exactly 7.5% down payment, if the table lists a separate band for 7.5%? If there is no separate band, say so and name the band that applies.",
     "truth": null,
     "kind": "absent_band",
     "answer": "Based on the machine-readable rule provided, there is **no separate band for exactly 7.5% down payment**.\n\nThe band that applies is: **\"5% to less than 10%\"**\n\nThe specific fee percentage for a first-use purchase with exactly 7.5% down payment is: **1.5%**",
     "extracted": 5.0,
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "fhfa-conforming-loan-limits-2026",
     "qid": "AB3",
     "question": "What is the 2026 conforming loan limit for a five-unit residential property? If the rule does not define a five-unit limit, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "Based on the machine-readable rule provided, **the rule does not define a five-unit limit**.\n\nThe rule only specifies limits for **1-unit properties** (as indicated by `\"units\": 1` in all four cells). \n\nWhile the rule indicates that \"number of units\" is a required qualifier for determining conforming loan limits, the actual limit values for properties with 2, 3, 4, or 5 units are not included in this particular rule document.\n\nTo find the 2026 conforming loan limit for a five-unit residential property, you would need to consult the complete FHFA county table or additional documentation that includes multi-unit property limits.",
     "extracted": null,
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "cfpb-hoepa-qm-thresholds-2026",
     "qid": "AB4",
     "question": "For loans consummated in 2026, what is the HOEPA points-and-fees dollar trigger specifically for manufactured housing loans, if Regulation Z sets a separate trigger for them? If it does not, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "Based on the machine-readable rule provided, **there is no separate HOEPA points-and-fees dollar trigger specified for manufactured housing loans**.\n\nThe rule shows only one HOEPA points-and-fees dollar trigger for 2026:\n- **$1,380** (applicable to loans consummated in 2026)\n\nThe data structure does not distinguish between manufactured housing and other loan types for this threshold. If Regulation Z does establish a separate trigger for manufactured housing loans under HOEPA, it is not reflected in this particular machine-readable file.\n\n**Note:** You should verify whether the complete Regulation Z rule (12 CFR \u00a7 1026.32) contains additional provisions for manufactured housing that may not be captured in this summary document. The CFPB sometimes publishes separate thresholds for manufactured housing in their annual adjustment notices.",
     "extracted": "$1,380",
     "correct": true,
     "invented_value": false
    },
    {
     "rule": "fha-forward-loan-limits-2026",
     "qid": "AB5",
     "question": "For an FHA case number assigned in 2026, what is the loan limit for a condominium unit, if HUD sets a separate condo-specific limit? If there is no separate condo limit, say so.",
     "truth": null,
     "kind": "absent",
     "answer": "Based on the machine-readable rule provided, **there is no separate condo-specific limit** for FHA forward mortgages with case numbers assigned in 2026.\n\nThe rule specifies loan limits based solely on the **number of units** (1, 2, 3, or 4 units), plus the HECM maximum claim amount. It does not distinguish between condominiums and other property types.\n\nFor a condominium unit (which would be a 1-unit property), the 2026 FHA loan limits are:\n- **Floor: $541,287**\n- **Ceiling: $1,249,125**\n\nThe actual limit for a specific condominium would depend on the county where it's located, as most counties fall somewhere between the floor and ceiling amounts. You would need to consult HUD's county-specific table to determine the exact limit for a particular location.",
     "extracted": "$541,287",
     "correct": true,
     "invented_value": false
    }
   ]
  }
 ],
 "summary": [
  {
   "model": "claude-haiku-4-5-20251001",
   "condition": "with_rule",
   "n": 30,
   "correct": 30,
   "errors": 0,
   "by_rule": {
    "fhfa-conforming-loan-limits-2026": "6/6",
    "fha-forward-loan-limits-2026": "6/6",
    "cfpb-hoepa-qm-thresholds-2026": "6/6",
    "va-funding-fee-2023-04-07": "6/6",
    "fha-annual-mip-ml-2023-05": "6/6"
   },
   "absence_correct": 5,
   "absence_n": 5,
   "absence_invented": 0
  },
  {
   "model": "claude-sonnet-4-5-20250929",
   "condition": "with_rule",
   "n": 30,
   "correct": 29,
   "errors": 0,
   "by_rule": {
    "fhfa-conforming-loan-limits-2026": "6/6",
    "fha-forward-loan-limits-2026": "6/6",
    "cfpb-hoepa-qm-thresholds-2026": "6/6",
    "va-funding-fee-2023-04-07": "6/6",
    "fha-annual-mip-ml-2023-05": "5/6"
   },
   "absence_correct": 5,
   "absence_n": 5,
   "absence_invented": 0
  }
 ],
 "regraded": "2026-10-11: absence detector made robust to markdown bold and to abstention phrasing; answers unchanged"
}