A FAIR estimate can be built exactly to spec, reviewed by a competent analyst, and still be wrong, and the model itself won’t tell you that. FAIR (Factor Analysis of Information Risk) gives a risk team a structure. Define the scenario, estimate how often it happens, estimate what it costs when it does, and combine the two into a probability distribution a board can act on. What FAIR does not include is a way to check, after the fact, whether the number that came out the other end was any good. That gap sits in the framework, not in the analysts running it.

The problem with unfalsifiable estimates

If a low-probability loss shows up anyway, the sponsor of the model has an answer ready every time. It was in the tail, a rare event the estimate already priced in. That answer is available whether the model was right or wrong, which means it isn’t actually testing anything. Without a way to compare what a model predicted against what happened, across enough cases to mean something, there’s no way to tell a well-calibrated model from a confidently wrong one.

Calibration is the test that settles it. Take a model that puts a 10% probability on a loss event in a given year: track enough organizations that got that same estimate, and roughly one in 10 of them should experience a loss if the model is calibrated. Fall well outside that range across a large enough sample, and the probabilities are decoration on an argument rather than a measurement of one.

Running that test takes two things most quantification programs don’t have: enough observations to make the comparison meaningful, and visibility into what happened afterward. A vendor that delivers the assessment and moves to the next client typically doesn’t find out whether the loss it modeled showed up, or what it cost when it did, 18 months later. That’s the same logic behind how we build the probabilities in our own models. The loop only closes when the same dataset holds both sides of it, the telemetry that feeds the frequency estimate and the claims outcome that tells you whether the estimate held up.

Where the data thins out

The second problem sits earlier, in the data that feeds loss magnitude. Most FAIR implementations pull severity figures from public breach databases, and those databases only capture what gets disclosed. The UK’s 2025/2026 Cyber Security Breaches Survey, conducted by Ipsos for   the Department for Science, Innovation and Technology and the Home Office, found that among organizations that identified a breach or attack, only 40% of businesses and 36% of charities reported the most disruptive one outside their own organization. The survey’s own authors note that hidden and unidentified attacks likely mean the real figures run higher than what gets counted. A magnitude estimate built on that visible slice starts underweighting severity before an analyst has entered a single assumption.

There’s a practical version of the same problem. A full FAIR exercise means sitting down with subject matter experts to gather the frequency assumptions, then chasing down magnitude data scenario by scenario and building the rest out by hand. That’s slow work, and the timeline stretches with every additional scenario. Most teams are still assembling the inputs by the time the board wants the number, which means nobody asks the calibration question from earlier.

Kovrr, a cyber risk quantification vendor whose own platform is built and marketed as a direct alternative to FAIR, made a similar point in a 2025 comparison of quantification models. Kovrr has an obvious commercial interest in that conclusion. But the claim itself doesn’t need their endorsement: the standard FAIR approach is resource-intensive and reliant on manual inputs, which makes it hard to scale once the underlying exposure changes. None of that makes FAIR wrong as a foundation for cyber risk quantification so much as incomplete; the taxonomy holds up, but there’s no built-in mechanism to keep testing it against something real.

What closes the loop

We laid out the contrast between FAIR and Resilience directly a while back, and the position hasn’t moved since. The two aren’t competing on the same axis, and using one doesn’t rule out the other. For a board reviewing any quantification exercise, the taxonomy behind the number matters less than what happens to it after year one: whether anyone goes back, checks it against what happened, and adjusts the next estimate accordingly. Ask a vendor for that comparison, and the answer says more about the model than the methodology behind it ever will.

Nothing here should be taken as legal, financial, or security advice for your specific situation — see the full disclaimer at cyberresilience.com/disclaimer.