Your team writes confidence numbers — model scores, “likely by Q3”, 70%.
Has anyone ever checked whether the 70% ones actually happen 7 times out of 10?
Drop the CSV below. The report renders on this same screen.
DROP A CSV HERE
or click to pick a file · needs a probability column + an outcome column (0/1)
no data handy? try the sample data
Your data never leaves this machine.
Every calculation runs in this browser. No upload, no server receiving the file,
no analytics. Turn off your wifi once this page has loaded — it still works.
That is not a promise buried in a privacy policy; it is how the page is built.
CSV format
column
contents
header names recognised
required
probability, 0..1 or 0..100, written before the outcome was known
probabilityforecastconfidencepprediksi
required
what actually happened, 0 or 1
outcomeactualresolvedyhasil
optional
market price or consensus — this sharpens the report considerably
market_priceconsensusmarketpasar
optional
name of the decision, used for the “ten worst misses” table
labelquestionnameid
Why the 0.25 baseline is a trap
Almost every calibration tool scores your Brier against 0.25 — the score you get
by saying 50/50 every single time. That sounds reasonable, right up until the event is rare.
Base rate 10%? A model that always answers “no” already scores 0.09. Far below 0.25,
with no skill whatsoever. A report that says “good, under 0.25” in that situation is
praising something that isn't there.
Here you face two harder opponents:
the base rate (br × (1−br) — the Brier of “always guess the average”)
and the market price when that column exists. The hardest available one decides the
verdict. Lose to the market and the verdict reads “LOSES TO THE MARKET PRICE” — not
“beats the base rate”, which would also happen to be true.
Three things this report refuses to do
It won't average away thin buckets
A bucket holding 8 forecasts can look dramatic while being pure noise. Buckets under
30 get a dashed circle and are excluded from the bias figure.
It won't make the verdict kinder than the findings
Including the colours. The Brier card only turns green when you beat the hardest
opponent your data supports — not when you happen to clear a coin flip.
It won't conclude from small data
Below 30 rows the report says so itself: this indicates direction, not conclusions.
Who this is for
ML teams that emit confidence scores and have never checked their calibration
risk, underwriting and credit scoring teams
demand, churn and fraud forecasting
PMs who write “likely to ship in Q3” and want to see their track record
anyone with a confidence column in a database that has never been tested
Limits, stated up front
This is a mirror, not investment advice. No buy/sell recommendations, no one's
money is touched.
Calibration ≠ accuracy ≠ profit. A perfectly calibrated model can still lose money
with bad sizing. That is a different problem.
It needs forecasts written before the outcome was known. Numbers filled in afterwards
measure memory, not calibration.
Take it with you
The downloadable version runs on your own laptop, for data that must not leave the
corporate network. It contains the audit engine, the formulas, two SKILL.md files for
Claude Code, sample data and a sample report. Node is the only requirement —
no database, no network calls.
The tool gives you the numbers. What usually helps more is the reading: why the skew
concentrates in certain ranges, which buckets can actually support a conclusion, and
what changed since last month.