Hindsight tells you how good your judgement is. That is a strong claim, so here is exactly how the number is produced, what it can and cannot tell you, and the point at which the product refuses to tell you anything at all.
The rule: squared error, averaged
Every resolved decision contributes the square of the gap between what you said and what happened. Say 80% and it happens, and you contribute (0.8 − 1)² = 0.04. Say 80% and it does not, and you contribute (0.8 − 0)² = 0.64. Your score is the average across everything you have resolved.
This is the Brier score. Lower is better. Zero is perfect. 0.25 is what you would score by answering 50% to everything, which is the number worth holding on to — it is the line between forecasting and shrugging.
Why this rule and not a friendlier one
The Brier score is strictly proper, which means the only way to make your expected score as good as possible is to state the probability you actually believe. Shading your numbers to look better cannot work — not because we would catch you, but because the arithmetic does not reward it.
That property is the whole reason a scoring rule was chosen rather than invented. A rule that could be gamed would quietly teach people to game it, and then the thing being measured would no longer be their judgement.
Two rules were considered and rejected. The logarithmic score is also proper, but its penalty runs away to infinity as a confident forecast approaches being wrong, so one badly-missed 99% would swamp a decade of careful thinking. “Percent correct” is not proper at all: it rewards saying 51% about everything you think is more likely than not, which is precisely the habit this product exists to cure.
The curve: are your 80%s really 80%?
The single score tells you how well you did. The reliability diagram tells you how you are wrong, which is more useful. It groups your decisions by the confidence you stated and plots, for each group, how often things actually went that way.
If you are well calibrated, the points sit on the diagonal: the things you were 70% sure of happened about 70% of the time. Points below the line mean confidence ran ahead of reality. Points above it mean you were more right than you let yourself believe.
Each point carries a vertical bar. That is a 95% Wilson interval — the range of true frequencies that would plausibly produce what you observed. It is wide when a point rests on five decisions and narrow when it rests on fifty, because those two situations mean very different things and a chart that drew them the same way would be lying by omission.
When the product says nothing
A calibration curve drawn from four decisions is not a weak signal. It is an invented one. So there are thresholds, and below them you see your individual decisions and a count of how far off you are — not a faded number or a provisional estimate.
- 10 resolved decisions before any overall score or the curve
- 20 before the skill score and the calibration breakdown
- 5 within a single domain before that domain gets a figure
Decisions you close as could not be settled are excluded from every figure and reported as their own count. They are never quietly folded in, because the ones that get murky are disproportionately the ones that went badly, and dropping them silently would flatter everybody.
What this cannot tell you
It cannot tell you whether a decision was good. A well-made decision can turn out badly and a reckless one can turn out fine; outcomes are noisy and one of them is never evidence about the thinking behind it. What accumulates over dozens of decisions is evidence about your confidence — whether the feeling of being sure means anything when you have it.
It also cannot tell whether you were honest with yourself when you typed the number. Nothing can. The product only guarantees that whatever you typed is still exactly what you typed.
Checking any of this
The arithmetic lives in one small, dependency-free part of the codebase and is covered by property-based tests: the decomposition identity is asserted to twelve decimal places, the intervals are checked against an independent derivation, and there is a test demonstrating that stating your real belief beats shading it. The reasoning behind each choice is written up as an architecture decision record.