Pre-registered. Written before any outcome data is examined, so the prediction is on record before the result — and the null result publishes the same way as a positive one.
This page renders the protocol document verbatim. It was written before any outcome data was examined, and it is published before any exists, so that the predictions and the decision rules are on record ahead of the result. Any change made after data collection begins is dated inside the document. Loans examined so far: 0.
Status: pre-registration. Written before any outcome data is examined.
Owner: Nik Gak · Date opened: 30 August 2026 · Published: 2 September 2026, before any outcome data was examined · Sample target: 50 (pilot) → 300 (full library) · Standard version under test: RTL-STD-001 v1.0, 56 checks, 20 Sphinx-original
The Sphinx standard contains 56 checks. Not one has been tested against a realised outcome. The entire company rests on a claim that has never been measured: that defects visible in an appraisal predict how far its after-repair value will miss the price the property actually sells for.
This protocol is written before any outcome data is looked at, on purpose. Predictions recorded in advance are the difference between a finding and a story. When the first study publishes, this document publishes with it.
If the primary hypothesis fails, that is a real result and it should be reported.
Version under test. RTL-STD-001 v1.0 as published — 56 checks across five families, 20 of them Sphinx-original, none calibrated. Record the standard version and the extraction pipeline version alongside every result. If the standard changes mid-study, the study is against the version recorded here and any later change is a separate run.
raw_ratio = exit_price / appraised_ARV
hpi_move = county HPI change, effective_date → exit_date (FHFA county series)
adjusted_ratio = raw_ratio / (1 + hpi_move)adjusted_ratio < 1 means the ARV was optimistic once market movement is removed.
Market normalisation is not optional. Without it, 2021 vintages look excellent and 2023 vintages look catastrophic for reasons that have nothing to do with the appraisal.
abs_error = |adjusted_ratio − 1|This distinction is load-bearing. Some checks are expected to predict direction (the value is too high). Others are expected to predict uncertainty (the value could be anywhere). A thin comp set does not mean the ARV is too high — it means the estimate is unreliable in both directions. Testing a variance-predicting check against mean outcome will find nothing and wrongly discard a good check.
Each check below is tagged [D] directional or [V] variance.
adjusted_ratio measures the mechanism. Loss measures the thing the customer cares about. Capture, where the servicing record allows:
extended — did the loan require an extension, and how manydefault_event — any notice of default, forbearance, or foreclosure filingloss_severity — realised loss net of carry and legal, where a loss occurredclean_exit — paid off at or before maturity with no modificationWhy this belongs in the pilot even if the data is patchy. "The scope-finish gap correlates with value miss" is an analyst's sentence. "The scope-finish gap correlates with loans that went sideways" is a credit committee's sentence, and it is worth an order of magnitude more in every conversation that follows.
Extensions are the useful variable here, not defaults. Defaults are too rare at n = 50 to test. Extensions are common, they are the leading indicator of an RTL loan in trouble, and they are usually recorded cleanly. Treat extended as the primary trouble proxy and report defaults descriptively only.
If servicing data is unavailable, record that as an exclusion rather than silently dropping the analysis.
Record retail sale, refinance, wholesale, or REO. Refinance exits have no arm's-length sale price — either exclude them from the primary analysis or use the refinance appraisal with an explicit note. Decide before looking at outcomes and record the decision here.
56 checks against 50 loans is not one experiment; it is 56, and at α = 0.05 roughly three will appear significant by chance alone. Tiering is how the study stays honest.
RTL-V-01 — budget versus comparable finish level [D]
H1: Appraisals where the submitted budget falls short of the finish level implied by the supporting comparables have a lower
adjusted_ratiothan those where it does not.
This is the company. It gets a single pre-registered test at α = 0.05, uncorrected, because it was specified in advance and everything else is secondary to it.
Thirteen of the twenty Sphinx-original checks should fire often enough to be measurable at n ≈ 50–150:
| Check | Predicts | Direction expected |
|---|---|---|
| RTL-V-02 — are the supporting sales actually finished | [D] | lower ratio when comps unrenovated |
| RTL-V-03 — as-is + budget reconciles to ARV | [D] | lower ratio when lift implausible vs budget |
| RTL-V-04 — ARV vs neighbourhood ceiling | [D] | lower ratio when above local 12-month high |
| RTL-V-05 — marketing time vs comp DOM | [V] | wider error when exposure understated |
| RTL-C-03 — comp set depth | [V] | wider error when thin |
| RTL-C-09 — superior comparables omitted | [D] | lower ratio |
| RTL-C-15 — market conditions adjustment | [D] | signed — lower ratio in a declining market, higher in a rising one |
| RTL-A-04 — cross-comparable consistency | [D] | lower ratio |
| RTL-A-07 — round-number clustering | [D] | lower ratio |
| RTL-A-08 — condition adjustment vs C-rating | [D] | lower ratio |
| RTL-A-09 — quality adjustment vs Q-rating | [D] | lower ratio |
| RTL-S-07 — as-is lands on contract price | [D] | lower ratio (anchoring) |
| RTL-S-16 — stated neighbourhood trend vs index | [V] | wider error on direction mismatch |
Benjamini–Hochberg FDR control at q = 0.10 across this family. Report every result, including nulls.
Note on RTL-C-15: it is the only check in the family whose expected effect is signed by market direction rather than uniformly negative. Test it against the signed index move, not as a plain fired/not-fired split, or a rising-market and a falling-market file will cancel each other out.
Seven originals cannot be evaluated at n = 50, for structural reasons rather than lack of interest. Record fire rates; do not test.
| Check | Why not now |
|---|---|
| RTL-C-11 — comparable resale history | Requires deed history on every comparable. Feasible, but adds a large data-collection burden for an expected fire rate under 10%. |
| RTL-S-09 — site value plausibility | Ground-up only. In a 50-loan mixed sample, the eligible subset is likely under 15 files. |
| RTL-S-10 — permit status | Ground-up and structural rehab only. Same subset problem, plus permit data availability varies by municipality. |
| RTL-P-03 — county coverage | Requires an appraiser's assignment history. A 50-loan sample spread across 20–30 appraisers cannot establish any individual's prior distribution. This is the cross-lender pooling problem in miniature and is exactly why the appraiser layer is second, not first. |
| RTL-P-04 — property type experience | Same reason. |
| RTL-P-08 — turn time outlier | Requires a per-appraiser median turn time, which needs volume per appraiser that a 50-loan sample cannot supply. |
| RTL-A-12 — conclusion inside adjusted range | Industry baseline, but listed here because it may fire on nearly every file and therefore carry no discriminating power. Check the fire rate first. |
The three RTL-P checks becoming testable is itself a milestone: it marks the point at which enough appraiser volume has accumulated for the second product layer to exist.
The 36 industry-baseline checks (distance, gross and net adjustment ratios, bracketing, recency, and so on) are hypothesis-generating, not confirming, at this sample size. Report effect sizes and confidence intervals; do not report p-values as evidence. Anything interesting becomes a Tier 2 hypothesis in the next study.
This tier answers a question nobody in the industry has asked: do the conventions already in universal use predict anything at all? A convention that predicts nothing is a publishable finding in its own right, and it is the strongest possible demonstration that the standard has a working discard mechanism.
Checks that will fire on roughly 0–2 files in 50, or that test file integrity with no expected effect on value: RTL-P-01 (licence valid), RTL-P-02 (licence level), RTL-P-05 (disciplinary history), RTL-P-06 (supervisory signature), RTL-P-07 (USPAP certification), RTL-S-11 (photo coverage), RTL-S-13 (sketch present), RTL-S-15 (flood zone), RTL-C-14 (duplicate comps), RTL-C-12 (closed vs pending), RTL-A-11 (support commentary), RTL-S-05 (zoning), RTL-S-08 (highest and best use), RTL-S-14 (adverse site conditions).
Record their fire rates for operational purposes. Do not test them for predictive power, and do not report null results as evidence they fail — they are unmeasured, which is a different thing.
Also record, and set aside, any check firing on more than 90% or fewer than 5% of files. A check that fires on everything has no discriminating power regardless of what it measures.
Pilot: 50 loans. Full library: 300.
n = 50 has power to detect only large effects on a single pre-specified test. That is enough for the go/no-go decision on RTL-V-01. It is not enough to evaluate 56 checks. Run the pilot first; expand only if Tier 1 passes.
Selection rules, all of which matter more than they sound:
Per loan:
loan_id (hashed) · address · origination_date · effective_date · subject_gla · as_is_value · arv · budget_total · project_type · exit_date · exit_price · exit_type · appraiser_name · appraiser_licence · county_fips
Per comparable (three or more per loan):
sale_price · sale_date · c_rating · q_rating · gla · distance_mi · net_adj · gross_adj · condition_adj · market_conditions_adj
Exit price comes from recorded deeds where the loan file doesn't carry it.
budget_psf = budget_total / subject_gla
implied_finish = median over comps of tier_cost(c_rating, q_rating)
gap_psf = implied_finish − budget_psf # positive = underfundedStarting tier lookup — deliberately rough:
| Condition / Quality | $/sf |
|---|---|
| C1–C2 / Q1–Q2 | 95 |
| C2 / Q3 | 75 |
| C3 / Q3 | 50 |
| C3–C4 / Q4 | 30 |
| C4–C5 / Q4–Q5 | 15 |
These numbers do not need to be correct. The primary test is rank-based, so the lookup only needs to be monotonic and consistently applied. Do not spend time refining it before the pilot — that is calibration work, and calibration is what happens after the signal is confirmed to exist.
Sensitivity check: re-run the primary test with tier values shifted ±30%. If the result flips, the finding depends on the lookup rather than the signal, and that must be reported.
A raw correlation between a flag and the outcome is not evidence the flag is informative. The first question any rating agency or diligence reader will ask is whether it beats the obvious alternatives.
Fit the naive model first, before testing any check:
adjusted_ratio ~ price_band + origination_vintage + county + project_type + hold_durationThen report, for RTL-V-01 and for every Tier 2 check, the incremental explanatory power over that baseline — partial correlation, or ΔR² from adding the flag to the model.
This matters more than it sounds. Cheap houses in weak submarkets may plausibly show both a large scope-finish gap and a poor realised ratio for reasons that have nothing to do with each other. If RTL-V-01 adds nothing on top of price band and geography, it is a proxy for expensive-house-in-a-good-market and the entire thesis is a restatement of something already known.
Report both numbers, always: the raw effect and the incremental effect over baseline. A check that survives the baseline is a real finding. One that doesn't is a confound, and saying so is what makes the rest of the study believable.
The product does not output one check. It outputs a composite. Two questions the single-check tests cannot answer:
Does flag count predict better than any individual check?
adjusted_ratio ~ baseline + n_material_flagsIf a simple count outperforms every individual check, that is worth knowing — it would mean the product's value is in breadth of coverage rather than in any one insight, which changes how the standard should be presented.
Are the checks measuring one thing or many?
Produce a correlation matrix of fired/not-fired across the Tier 2 family. High collinearity would mean thirteen checks are wearing the same underlying signal, which is acceptable but must be known before publishing — otherwise the study looks like thirteen independent confirmations when it is one.
Report both. If they conflict with the single-check results, the composite result is the one that describes the product.
At n = 50, a Spearman test at α = 0.05 with 80% power detects approximately ρ ≥ 0.39. Weaker true effects will not reach significance in the pilot regardless of whether they are real.
Two consequences, both worth internalising before results arrive:
At n = 300, the detectable effect drops to roughly ρ ≥ 0.16, which is why the full library evaluation waits for the expanded sample rather than being attempted in the pilot.
Primary [D] test. Spearman rank correlation of gap_psf against adjusted_ratio. Report ρ, p, n, and a scatter. Rank correlation is used specifically because the tier lookup is approximate.
Secondary presentation for a non-technical audience: split at the median gap, compare group mean adjusted_ratio, report the difference with a 95% CI. Blunt, but this is the version that persuades a credit committee.
[V] tests. Compare variance of adjusted_ratio between fired and not-fired groups using Levene's test. Do not use a mean comparison for these checks.
Confounder to rule out. Cheap houses in cheap submarkets may show both a large gap and a poor ratio for unrelated reasons. Repeat the primary test within price tercile. If the effect survives only in the bottom tercile, it may be a price-band artifact rather than a scope-finish effect.
Also report, regardless of outcome: fire rate per check, extraction success rate per field, and count of loans excluded with reasons.
RTL-V-01, primary test:
| Result | Decision |
|---|---|
| ρ ≤ −0.35, p < 0.05 | Proceed. Signal confirmed. Expand to 300 and run the full library. |
| −0.35 < ρ ≤ −0.15 | Ambiguous. Expand to 150 before deciding anything else. |
| ρ > −0.15 | Primary hypothesis fails in this cohort. |
If the primary fails: check rehab and ground-up separately before concluding — plans-and-specs budgets are more precise than rehab scopes and the effect may exist in only one. If it fails in both, report it, and the strategic consequence is that the product reduces to appraiser calibration, which is a weaker business and a different competitive position.
Tier 2: report all thirteen with BH-adjusted q-values, including nulls. Checks that survive become candidates for calibrated thresholds. Checks that don't are flagged for review at n = 300, and if they still show nothing, they are removed from the standard. That removal mechanism is the point of the whole project.
Extraction accuracy is covered by the arithmetic reconciliation and the deed cross-check. Whether the person scoring the files did so consistently is not, and at pilot scale one person is hand-assigning finish tiers and reading C/Q ratings across a multi-week collection window. Drift is likely and invisible.
Procedure. Set aside 15 files at random. Re-score them blind at least seven days after the first pass, without reference to the original values. Report:
gap_psfIf tier agreement falls below roughly 80%, the tier lookup is too subjective to support the finding and needs tightening into an explicit decision rule before the study is valid.
This costs about an hour and closes a hole any careful reader will probe. A study whose central variable depends on one person's judgment, with no reliability estimate, is not publishable.
Draft the abstract before any data is collected — one paragraph per branch of section 7. If a coherent paragraph cannot be written for the null case, the study design has a problem, and it is much better to discover that now.
If the signal is confirmed (ρ ≤ −0.35):
Across [n] exited residential transition loans in [geography], appraisals whose submitted budget fell short of the finish level implied by their supporting comparables realised sale prices [x]% below their after-repair values, against [y]% where the budget matched. The effect persisted after controlling for price band, vintage, county and project type, and after adjusting each loan for market movement over its own hold period.
If ambiguous (−0.35 < ρ ≤ −0.15):
A directionally consistent but statistically inconclusive relationship was observed between the budget-to-finish gap and realised value miss. At this sample size the study detects effects of ρ ≥ 0.39 with 80% power; the observed effect falls below that threshold and no confident conclusion is available in either direction. The sample is being expanded to [n]. Method and full results published here regardless of outcome.
If the hypothesis fails (ρ > −0.15):
Across [n] exited loans, no meaningful relationship was found between the budget-to-finish gap and realised value miss, in either the rehab or the ground-up cohort. The hypothesis that under-funded scopes systematically produce optimistic after-repair values is not supported by this sample. RTL-V-01 is [flagged for review at expanded sample / removed from the standard]. The dataset and method are published so the result can be checked.
The third paragraph is the one that matters most. A standard whose entire premise is that checks get removed when they fail has to be willing to remove its own headline check. If that turns out to be the finding, publishing it is worth more to the long-run credibility of the standard than any positive result — it is the only thing that proves the discard mechanism is real rather than rhetorical.
validation/hypothesis_test.ipynb producing:
This notebook, cleaned and with figures redrawn, becomes Finding 001 — the first published study and the artifact that ends the credibility problem permanently.
Stated up front so it isn't overclaimed later:
RTL-STD-001 v1.0 · published · this page last updated 2 September 2026 · status
The Sphinx standard describes analytical review of an appraisal report. It is not an appraisal, not a valuation, and not a review appraisal under USPAP Standard 3. Outputs are internal analytical inputs for credit decisioning. They are not communicated to appraisers and are not a basis for requesting revision to any report.