How it works
Nothing here is secret. The interesting engineering in a forecasting system is not the prediction — it is everything built to stop the system from flattering itself afterwards.
The nightly run
One scheduled chain, same time every night, always working on a completed day rather than a live one. Every timestamp in the system is UTC, with no exceptions — a single local-time function once cost five days of silent downtime, which is the sort of lesson that turns into a rule.
-
Collect
Posts on X and news articles scraped from websites arrive from a collector that runs separately from the pipeline, so a bad night for one does not corrupt the other. Market data comes in on its own schedule: equity and options prices, options dealer positioning, statutory releases, and the prices attached to open questions on public forecasting markets.
-
Turn text into structured observations
Each item is matched against a maintained vocabulary of entities and themes and written out as a structured observation: who, what, which theme, which source, when. From here on the pipeline works on records, not prose.
-
Extract only claims that can be graded
This is the strictest gate in the system. A claim is kept only if it carries all three of: a direction, a thing it is about that something measurable can be attached to, and a deadline. Anything missing one of those is discarded rather than guessed at.
It is a brutal filter — see the funnel on the front page — and loosening it is the tempting mistake. A pipeline that fills its ledger with claims it can never resolve looks productive and learns nothing.
-
Register the question
The claim is the input; the question is the object that can be resolved. Registering it separately means one question can gather evidence from several sources, and that the resolution criteria are fixed at registration — before anybody knows the answer.
-
Record what the market already thinks
Where a public price exists for the same outcome, it is snapshotted at the moment the forecast is made and stored alongside it. That price is the benchmark. "Better than nothing" is not a standard worth measuring against; "better than the price you could have traded at" is.
-
Wait
Deliberately the dullest stage. Between registration and the deadline nothing touches the claim. If a forecast can be revised after the fact, the track record built on it means nothing.
-
Resolve
Once the deadline has passed, a resolver grades the question against the record: confirmed, falsified, or inconclusive. Inconclusive is a stored outcome, not a deletion. A system that silently drops the cases it could not settle will always look better than it is.
How trust moves
Every source keeps a running record of how its claims turned out, held as a probability with an explicit uncertainty around it rather than a score somebody assigns. Early on, a source that gets one call right barely moves; after fifty resolved claims the record starts to mean something. That is the whole point of using a distribution instead of a percentage — it knows the difference between good and not yet measured.
Reputation updates run behind circuit breakers, because an automatic weighting system that is wrong at speed is worse than no weighting at all.
The circuit breakers
Always on- A source needs a minimum body of resolved evidence before its weight moves at all.
- Only a small number of changes are allowed per night, so one bad batch cannot rewrite the whole ledger while nobody is watching.
- If a large share of one category collapses in a single run, the update aborts instead of committing. That pattern is nearly always a broken feed, not a sudden outbreak of dishonesty.
- Sources are quarantined, never deleted, and there is a cooldown before a quarantined source can be re-scored.
How the forecasts are scored
Three measurements, because being right is not one thing:
-
Brier score
The standard scoring rule for probability forecasts. Saying "80% likely" and being right scores better than saying "55% likely" and being right; saying "95% likely" and being wrong is punished hard. It removes the hedger's escape route, where every forecast sits near 50% and is never wrong enough to matter.
-
Calibration
Of everything called 70% likely, did roughly 70% of it happen? A forecaster can be well-calibrated and useless, or sharp and badly calibrated. You want to see both numbers, so both are kept.
-
Lead time
How far ahead of the market price the call arrived. This is the one that usually gets left out, and it is decisive. A signal that reaches the same conclusion as the market, two days later, has added nothing. It is an echo, and it is recorded as one.
The rules that stop it marking its own homework
These are the parts worth copying into any system that has to be trusted rather than merely believed.
-
Everything new starts at zero influence
A new source, a new extraction method, a new model — none of them arrive with weight. They run alongside the existing system, producing forecasts that count for nothing, and earn influence only by resolved outcomes. There is no path by which something joins the system because it seemed promising.
-
Three gates, in order, and the third is the sharp one
To go from zero influence to some, a method has to clear a volume floor (enough independent resolved cases that the result is not luck), a value bar (measurably better than the existing baseline, not merely different), and a lead bar (earlier than what is already available). Failing the third means it is accurate but late — and it stays at zero regardless of how good its accuracy is.
-
The rules are sealed before the results exist
The criteria — what counts as eligible, what threshold constitutes an improvement, how cases are grouped so correlated ones cannot be counted twice — are written down and cryptographically hash-pinned in advance. Changing them later changes the hash, which is visible. This is ordinary pre-registration discipline borrowed from clinical trials, and it exists because the easiest way to build a system with a great track record is to decide what counts as success after seeing the numbers.
-
Demotion is automatic and cheap
Anything whose rolling performance drops below the bar returns to zero influence on its own. There is a kill switch that drops every earned weight at once and reverts to the plain baseline. Nobody has to win an argument for that to happen.
How it fails loudly
The failure mode that actually happens in production is not a crash. It is a stage that runs, exits zero, and quietly produces nothing, for weeks, while the dashboards stay green. So:
- Each stage writes a marker before it starts and another when it finishes, so "never started" and "started and died" are distinguishable.
- Each stage writes a manifest of what it actually produced.
- An audit reports not only what failed, but which expected stages left no manifest at all — the gap that a pass/fail check cannot see.
- The funnel report always prints a summary line, even when the answer is zero. A run that measured nothing is visible, not silent.
The bar for any change
Zero new failuresUnderneath all of it sits a test suite of more than 1,200 automated checks. Several of those tests exist to prove that the guards themselves can fail — a safety check that has never been shown to trip is not a safety check, it is decoration.
Two related pages
What counts as a prediction here, and what this whole thing is for.