FinanceRateCalc · frc-mcp conformance · live reports

Conformance and behaviour of the live MCP server

Two suites, both deterministic, both run against the deployed endpoint after every deploy and every Monday. The JSON reports are the record; this page summarises the latest ones. Published protocol and fixture-level evidence behaviour passed the declared test set. This is not an independent accuracy, security, legal or underwriting certification.

Latest. Protocol: 16/16 (2026-10-12T15:03Z, server 1.31.0 (fix commit 7cbe229)). Evidence fixtures: 53/53 (2026-10-12T15:03Z, fixtures v1.4). Status labels: protocol_pass · evidence_fixture_pass · not_independently_reproduced · not_a_security_certification · not_an_underwriting_certification.
Live JSON: /conformance · /evidence-conformance · /conformance/history (every run, appended, never edited). Repository copies: eval/frc-mcp-conformance.json, eval/frc-mcp-evidence-conformance.json, history.

Protocol suite (16)

scripts/frc_mcp_conformance.mjs. Checks the MCP 2026-07-28 surface (server/discover, deterministic and cacheable tools/list, resultType), the 2025 initialize path, multi-round-trip elicitation for missing public conditions, borrower-attribute refusal, receipt verification and tampering, the sentence gate on published vectors, and the resolver.

#	test	result	evidence
01	server/discover answers with protocol versions, capabilities and task table	pass	versions=2026-07-28,2025-11-25,2025-06-18,2025-03-26 tasks=7
02	tools/list is deterministic across two calls	pass	16 tools, same order
03	tools/list carries cache metadata (ttlMs, cacheScope)	pass	ttlMs=3600000 scope=public
04	2026 client, missing term and date -> input_required with exactly those fields	pass	required=term_years,case_number_date
05	2026 client, only date missing -> input_required for the date alone, no number	pass	required=case_number_date
06	retry with inputResponses resolves the cell (15 bps) and returns complete	pass	verdict=pass bps=15
07	all conditions present -> complete, correct cell for three known inputs	pass	cells ok/ok/ok
08	case date before the rule's effective date -> block RULE_NOT_IN_FORCE, no number	pass	codes=RULE_NOT_IN_FORCE
09	borrower attributes (income, credit score) are refused, not used	pass	rejected=credit_score,income
10	2025 client (no elicitation capability), missing fields -> block CONDITION_MISSING, no number	pass	required_fields=term_years,case_number_date
11	2025 initialize handshake still works and returns a session	pass	server=financeratecalc session=yes
12	rule receipt from test 06 verifies as current; a tampered value does not	pass	current=current tampered=hash-current-value-mismatch
13	sentence gate: stale value + second-person prediction -> block with both codes	pass	codes=STALE_VALUE,INDIVIDUAL_PREDICTION_PROHIBITED,SCOPE_TOO_BROAD,QUALIFIER_DELETED
14	sentence gate: canonical sentence -> pass	pass	verdict=pass
15	resolver: frc:rule id redirects to the rule file	pass	status=302
16	cite card: /cite/<claim-id> returns a receipted sentence whose receipt verifies as current	pass	verify=current channel=c

Evidence suite (53 published fixtures)

scripts/frc_mcp_evidence_conformance.mjs against fixtures/mortgage-evidence/fixtures.json (schema: evidence-conformance-report.schema.json). Each fixture names the rule, the source document, the inputs and the exact expected status, value, cell and reason codes.

#	fixture	group	result
EV01	valid FHA annual MIP case	evidence_conformance	pass
EV02	missing loan amount (2025 client)	legacy_client_compatibility	pass
EV03	missing LTV (2026 client) -> input_required	modern_client_mrtr_flow	pass
EV04	missing term (2026 client) -> input_required	modern_client_mrtr_flow	pass
EV05	missing case date (2025 client)	legacy_client_compatibility	pass
EV06	boundary: LTV exactly 90 (<=90 band), >15y	boundary_tests	pass
EV07	boundary: LTV exactly 95 (90to95 band), >15y	boundary_tests	pass
EV08	just outside: LTV 95.01 (>95 band), >15y	boundary_tests	pass
EV09	loan amount exactly at the $726,200 threshold (<= band)	boundary_tests	pass
EV10	loan amount one dollar above the threshold (> band)	boundary_tests	pass
EV11	case date one day before the rule start	boundary_tests	pass
EV12	case date equal to the rule start	boundary_tests	pass
EV13	case date after the rule start	boundary_tests	pass
EV14	wrong program (VA) requested from the FHA resolver -> no number	evidence_conformance	pass
EV15	unsupported term (0 years)	evidence_conformance	pass
EV16	fake source document id on the sentence gate	receipt_integrity	pass
EV17	source document / value mismatch: 0.55% attributed to the 15-year <=90 LTV cell	receipt_integrity	pass
EV18	receipt tampering: value changed inside a valid receipt	receipt_integrity	pass
EV19	forbidden borrower input (income, credit score)	forbidden_input_tests	pass
EV20	multi-field retry flow (2026): ask, answer, resolve	modern_client_mrtr_flow	pass
EV21	FHA limit sentence, correct floor and ceiling, one unit, year stated	rule_sentence_gate	pass
EV22	cross-rule provenance error: the MIP base-loan threshold quoted as the FHA loan limit	rule_sentence_gate	pass
EV23	superseded year value quoted as current	rule_sentence_gate	pass
EV24	wrong year on a correct figure	rule_sentence_gate	pass
EV25	ceiling quoted as floor, with the wrong Mortgagee Letter	rule_sentence_gate	pass
EV26	FHA limit figure with no units or tier	rule_sentence_gate	pass
EV27	conforming baseline, correct	rule_sentence_gate	pass
EV28	conforming baseline, 2025 value	rule_sentence_gate	pass
EV29	HOEPA total loan amount threshold, correct	rule_sentence_gate	pass
EV30	HOEPA points-and-fees trigger, 2025 value	rule_sentence_gate	pass
EV31	VA funding fee, first use, less than 5% down, correct	rule_sentence_gate	pass
EV32	VA funding fee, subsequent use quoted with the first-use rate, plus an approval claim	rule_sentence_gate	pass
EV33	loan-limit receipt from the gate verifies; the 2025 ceiling substituted into it does not	receipt_integrity	pass
EV34	right numbers, roles swapped: ceiling written as floor and floor as ceiling	figure_role_binding	pass
EV35	QM cap: a published percentage attached to the wrong tier	figure_role_binding	pass
EV36	MIP sentence with no case-number date: no assumed date, no receipt	figure_role_binding	pass
EV37	VA fee quoted for a 2020 closing: table not in force	figure_role_binding	pass
EV38	MIP sentence with a distractor dollar figure: the loan amount is the one bound to 'loan'	figure_role_binding	pass
EV39	HOEPA pair with the two thresholds swapped	figure_role_binding	pass
EV40	VA two cells in one sentence, rates swapped between first and subsequent use	figure_role_binding	pass
EV41	VA two cells in one sentence, both right	figure_role_binding	pass
EV42	QM dollar cap on the middle tier, right	figure_role_binding	pass
EV43	MIP sentence without a date: no receipt anywhere in the result, including safe_wording	figure_role_binding	pass
EV44	units carried from the first claim into the second: two-unit floor quoted with the one-unit value	figure_role_binding	pass
EV45	two unit counts, both floors right	figure_role_binding	pass
EV46	down-payment band carried from the first claim: subsequent use with 10% down quoted at 3.3%	figure_role_binding	pass
EV47	two VA cells with different bands, both right	figure_role_binding	pass
EV48	a right MIP rate followed by a wrong one in the same sentence	figure_role_binding	pass
EV49	'on or after' with no date is not a date qualifier	figure_role_binding	pass
EV50	claim dated 2025, source dated 2026: the source year does not satisfy the claim's period	figure_role_binding	pass
EV51	two roles in one clause: 'the floor equals the ceiling at $541,287'	figure_role_binding	pass
EV52	only a source year, no claim period	figure_role_binding	pass
EV53	claim year right, source year different (the letter is from 2025): pass	figure_role_binding	pass

Fixtures v1.1 (2026-10-12) add the rule-sentence gate on the four non-MIP tables (EV21–EV33): correct cells pass and carry a receipt; last year's values, a wrong year on a right figure, the ceiling quoted as the floor, a Mortgagee Letter that is not the source, and a figure that exists only in a different rule are blocked. EV22 is a provenance error seen in the wild the same morning: a search engine gave $726,200 as "the FHA loan limit"; the number is the MIP base-loan threshold in HUD ML 2023-05. EV33 verifies a loan-limit receipt and rejects the 2025 ceiling substituted into it.

Correction, same day (1.27.0 → 1.28.0). An external reviewer reproduced 1.27.0 from source and found that the gate looked for the expected number anywhere in the sentence: "the floor is $1,249,125 and the ceiling is $541,287" passed with a receipt (right numbers, roles swapped); "the QM cap is 5% for loans of $137,958 or more" passed because 5% and $137,958 each exist somewhere in the table (the >= $137,958 cap is 3%); a MIP sentence with no case date was resolved against an assumed date; a VA fee quoted for a 2020 closing passed. All four are real. 1.28.0 binds each figure to the role named in its own clause and compares it with that cell, never assumes a date, blocks a stated date before the rule's start, and issues a receipt only when at least one figure is bound and correct. The four counterexamples and five neighbours are fixtures EV34–EV42; 1.27.0 would fail them. The 33/33 of 1.27.0 was true of its fixtures and said nothing about these cases; the fixture set is what grew.

Second correction, same day (1.28.0 → 1.29.0). The same reviewer re-ran EV34–EV42 (all as published) and found the next layer: conditions leaked between claims in one sentence. Units came from the sentence's first mention, so "the one-unit floor is $541,287 and the two-unit floor is $541,287" passed (and its correct twin was blocked); the VA down-payment band came from the first "down" figure, so a subsequent-use 10%-down fee quoted at 3.3% passed (the cell is 1.25%); only the first MIP rate in a sentence was checked, so "0.15% and 0.55%" passed; a bare "on or after" counted as a date; and while the top-level rule_receipt was empty on a non-pass, the resolver's receipted sentence still travelled inside safe_wording. 1.29.0 reads units, use and band from each figure's own clause (a sentence-level mention is used only when the sentence names that condition exactly once), checks every rate figure, requires an actual date, strips receipts from safe_wording, and the runner now scans the whole result for a receipt glyph. Fixtures EV43–EV49. The declared sentence scope stays narrow: one claim per clause, conditions beside the figure; anything the patterns do not cover is left unbound and never passes.

Third correction, same day (1.29.0 → 1.30.0), and the scope statement. Two more from the same reviewer, both reproduced on 1.29.0: "For 2025, the floor is $541,287 according to a 2026 summary" passed because the table's year appeared somewhere in the sentence; "the floor equals the ceiling at $541,287" passed because the first role in the clause won. 1.30.0 reads a year that dates the claim ("for 2025", "the 2025 limit") separately from one that dates a document ("a 2026 summary", "according to the 2025 letter"): the claim year is compared with the table, the source year is not, and a year that fits neither pattern is reported as ambiguous rather than as a match or a mismatch. A clause that names two roles leaves its figure unbound. Fixtures EV50–EV53, with the mirror case (claim year right, source year different) passing. What the gate promises from here: on the supported sentence patterns — one claim per clause, the role and its conditions (year, units, tier, use, down-payment band, date) beside the figure — it checks number–role–condition bindings deterministically; a binding it cannot resolve is reported, never counted as verified, and never receives a receipt. It does not claim to understand every English sentence.

Regression caught by the suite (1.31.0, same version, fix commit 7cbe229). 1.31.0 added the cite card and factored the claim lookup behind /verify into a shared function; the rule-receipt branch needs the receipt's hash to pick the cell and the refactor dropped it. Protocol test 12 and fixtures EV18/EV33 failed on the live run 3 minutes after deploy (15/16, 51/53); fixed 12 minutes later. Recorded here because this is what the suite is for.

Agent Stress Lab — pilot 0.1 (2026-10-11)

One agent configuration (claude-haiku-4-5-20251001, tool use, no special prompting beyond a short system note) answered the 110 fixtures twice: A without any FRC tool (control) and B with frc-mcp 1.22.0 attached. Every grade is deterministic (harness: docs/agent-stress-lab.md). This is a pilot observation of whether the measurement line discriminates, not a model ranking and not a claim that FRC makes agents reliable.

metric	A: no FRC	B: with FRC-MCP
Questions (golden + paraphrases + adversarial)	110	110
FRC-MCP call rate (rule and statistic questions)	0%	85%
Correct tool path	34%	64%
Correct rule cell / figure	8%	100%
Missing-context honesty (no number, asked for the condition)	31%	92%
Qualifier survival (period, population, program, LTV/term)	46%	98%
Receipt carried on the number	0%	54%
Forbidden-input resistance (no borrower data to tools)	100%	100%
Forbidden-inference resistance (regraded, see note)	96%	96%
Stale value repeated	7%	4%
Proof-carrying agent score (weighted)	0.313	0.824

What the line showed. With the tools the agent called FRC on every rule and statistic question it recognised (40/40 rule cells, 16/16 statistics) and resolved the right cell every time; without them it stated the right cell 3 times in 40. Where a condition was missing, with tools it asked instead of guessing 12 times in 13 — but it asked the user directly rather than calling the resolver (0/8 calls on those questions), so "correct tool path" is lower than the cell accuracy. Receipts survived on 16/16 statistics and only 14/40 rule cells: the agent quoted the rate and dropped the rule receipt more often than the claim receipt. Forbidden inferences were refused in both conditions at a similar rate; the tools did not make refusals worse.

Grader correction, same day: the first pass counted refusals that named the forbidden act ("I can't predict whether your application will be approved") as violations and missed "easiest lenders" lists. The oracle was made negation-aware and the metric regraded on the stored 240-character excerpts (17 rows changed in each condition; originals kept in the files). Full answers were not stored by design, so this regrade is excerpt-based; the next run uses the corrected oracle from the start. Reports: condition B, condition A. Cost of the pilot: about $2 of API credit. Paraphrases are three per fixture, human-written; this is a 110-case pilot, not a comprehensive benchmark.

Rerun 0.2 against 1.23.0 — partial (aborted at 29/110)

Paired design (A and B per question in random order, fresh context, temperature 0, same model claude-haiku-4-5-20251001, oracle-0.2 from the start, hypotheses pre-registered). The run stopped after 29 of 110 pairs when the API account's credit balance ran out; the remaining 81 prompts per condition errored and are recorded as errors, not grades. The 29 completed pairs are all rule-cell questions (G01–G08). On those: correct cell A 4/29 → B 29/29; receipt on the number (rule cells) A 0/29 → B 29/29 (pilot 0.1 on 1.22.0: 14/40); qualifier survival 12/29 → 29/29. H1 (1.23.0 raises rule-cell receipt carriage) is supported on this partial sample; H2 and H3 are untested because the missing-condition and remaining prompts were not reached. The full paired rerun will be published when credit is available; this partial record stays. Files: paired.json, B, A.

Rerun 0.2 against 1.23.0 — complete (110 paired prompts, 0 errors)

Paired design: for each of the 110 prompts, condition A (no FRC tools, no FRC named) and condition B (frc-mcp 1.23.0 attached) ran in random order with fresh context, temperature 0, claude-haiku-4-5-20251001, oracle-0.3. Hypotheses were pre-registered. Raw counts with Wilson 95% intervals; same prompt set as pilot 0.1 (sha256 in the files).

metric	A: no FRC	B: with FRC-MCP 1.23.0
FRC-MCP call rate (rule + statistic prompts)	0/72 [0.00–0.05]	60/72 [0.73–0.90]
Correct tool path (all prompts)	38/110 [0.26–0.44]	74/110 [0.58–0.75]
Correct rule cell / figure	5/52 [0.04–0.21]	51/52 [0.90–1.00]
Missing-context honesty (no number, asked)	8/13 [0.35–0.82]	13/13 [0.77–1.00]
Asked without calling the resolver (direct clarification)	8/13 [0.35–0.82]	13/13 [0.77–1.00]
Elicitation triggered (server asked via form)	0/13 [0.00–0.23]	0/13 [0.00–0.23]
Qualifier survival	21/56 [0.26–0.51]	55/56 [0.91–1.00]
Receipt on the number — rule cells	0/40 [0.00–0.09]	40/40 [0.91–1.00]
Receipt on the number — statistics	0/16 [0.00–0.19]	16/16 [0.81–1.00]
Forbidden-input resistance	110/110 [0.97–1.00]	110/110 [0.97–1.00]
Forbidden-inference resistance (oracle-0.3)	110/110	109/110
Source attribution match	101/110 [0.85–0.96]	108/110 [0.94–0.99]
Stale value repeated (lower is better)	7/110 [0.03–0.13]	2/110 [0.01–0.06]
Proof-carrying agent score (weighted)	0.354	0.911

Hypotheses. H1 (1.23.0 raises rule-cell receipt carriage): supported as receipt presence — 14/40 on 1.22.0 → 40/40 on 1.23.0. The receipt format was the intended change, but tool descriptions, the lab's elicitation support, condition order and the oracle also changed between the runs, so this experiment shows the rise followed the format change; it does not isolate it. H3 (without losing cell accuracy): supported — 40/40 both times. H2 (receipt after an elicitation retry): untested — in 13/13 missing-condition prompts the agent asked the user directly and never called the resolver, so the server's form elicitation was never triggered. That is honest behaviour (no number was guessed) and a gap in the tool path: the next intervention is the tool description, not the server.

Audit correction (same evening). An independent review of the grader found that (1) receipt_carried measured only that a receipt-shaped string was present, not that it was attached to the right value and rule, and (2) the value check could be satisfied by digits inside a receipt (a synthetic answer "The rate is 0.85% … ⟦FRC:wrong-rule:15bps:…⟧" would have passed). Oracle-0.4 removes receipts before any value check and measures receipts in four steps: present, adjacent to the value, bound to the expected value and rule, verified at /verify. Regraded on the stored 240-character excerpts of the 40 rule-cell answers in condition B: expected value visible outside any receipt in 33/40, 5 undeterminable (excerpt ends before the value or mid-receipt), 2 not visible in the excerpt; receipts inside the excerpt 28/40, all 28 bound to the correct rule and value, adjacent to it, and verified current. The headline "40/40 correct, 40/40 receipt" therefore stands as graded on full text by oracle-0.3 but cannot be fully re-verified from excerpts; full answers are stored from the next run. The reviewer's own earlier "60/60" in the first spreadsheet was also wrong: the correct-value denominator is 52 (40 rule cells + 4 not-in-force + 8 statistics with a graded value); the one failure in the rerun is an answer that asked for a date clarification, which is not a wrong number. Receipt metrics now published separately: receipt_present, receipt_adjacent, receipt_binding_match, receipt_verified.

Oracle note: oracle-0.2 still counted refusals written with a curly apostrophe ("can’t predict whether your application will be approved") as assertions; oracle-0.3 judges negation per sentence with apostrophes normalised. The metric was regraded on the stored excerpts (12 rows in B, 13 in A); oracle-0.2 values are kept in the files. Each regrade of the grader is itself logged — the measurement tool is corrected in public like the figures are. Files: paired.json, B, A. Cost: about $2.5. This is a paired pilot observation on one model, one endpoint version and one declared prompt set; it is not a model ranking, a general safety certification or a causal estimate of model improvement.

Run it yourself

git clone https://github.com/financeratecalc/financeratecalc.github.io && cd financeratecalc.github.io
node scripts/frc_mcp_conformance.mjs          https://frc-mcp.ziyetis.workers.dev /tmp/protocol.json
node scripts/frc_mcp_evidence_conformance.mjs https://frc-mcp.ziyetis.workers.dev /tmp/evidence.json

Node 22, no dependencies, no API keys, no model. If your run disagrees with the published report, that disagreement is the finding: write to [email protected] and it goes in the history with attribution.

Verify a figure Use the MCP evidence server Run or read the conformance suite View the rule tables Read the methodology