Runs nightly · grades its own output
A system that grades its own predictions, and lets the grades decide what it trusts next.
Every night Kengly reads the same material a desk analyst would skim and mostly forget: posts on X, news articles from financial and general-news sites, and market data — equity and options prices, options dealer positioning, and the open questions on public forecasting markets. It turns what it reads into claims that carry a direction and a deadline, then comes back later to check which ones held. The grades are not a report card at the end. They are the input: a source that keeps getting it wrong quietly loses its say, and nothing carries weight until the outcomes have earned it.
Most systems that make predictions never find out
Sentiment scores, risk dashboards, AI summaries of the morning's news — they all produce something shaped like a forecast. Almost none of them store it in a form that can later be marked right or wrong. "Sentiment is deteriorating" names no threshold and no date, so there is no future moment at which it becomes false. It can never be wrong, which sounds like a strength and is the entire problem.
"Sentiment around the sector is deteriorating."
No threshold, no referent, no date. There is no future moment at which this becomes false, so it never enters a track record.
A direction, a size, a thing it is about, and a date by which it has to have happened.
Left untouched until that date passes, then graded against what actually occurred — confirmed, falsified, or inconclusive.
What it does, in four steps
-
It reads a firehose
Posts on X, news articles, and price and positioning data arrive as a messy stream on their own schedules. The pipeline sorts, tags and turns them into structured observations, without anyone reading each one.
-
It writes down what it thinks will happen
Not a mood, not a score — a dated, falsifiable claim, alongside the market price of the same outcome where a public price exists.
-
It grades itself against the outcome
When the deadline passes, a resolver checks the claim against the record. Confirmed, falsified, or inconclusive. Inconclusive is written down too, rather than quietly dropped.
-
The weight follows the result
Every source and every method keeps a track record. Something that is usually right earns influence. Something that is usually wrong loses it, automatically, without anyone defending it in a meeting.
What the grading buys, and what skipping it costs
Grading is the expensive part — most of the engineering here is in it rather than in the predicting. Three things follow from it, and none of the three are available to a system that skips it.
The cost of skipping it is not that such a system is wrong. It is that nobody ever finds out either way. You go on weighting a source that has been wrong for a year — not because you decided to, but because nothing ever measured it and told you to stop. An ungraded system does not sit still and stay mediocre; it drifts, and it drifts invisibly, and it keeps producing confident output the whole time.
That is the case for spending the money on the grading, and it is why a forecasting system without one is not a cheaper version of this. It is a different thing that looks the same from outside.
-
The system reallocates its own attention
Every resolved claim moves the record of the source it came from. Over months, the accounts and outlets that keep turning out to be right accumulate weight and the rest lose it — mechanically, without anybody defending a favourite source in a meeting.
-
You can tell an improvement from a change
A modification to the pipeline either moves the scores or it does not. Without grades, every change is argued on plausibility instead, and plausible changes are precisely the ones that quietly make a system worse.
-
Someone who does not trust you can check
The claims are dated, the resolution criteria are fixed before the outcome is known, and the losses are published next to the wins. That is a track record an outsider can audit rather than a number they have to take on faith.
The part most write-ups leave out
Extracting claims is easy. Extracting claims that can actually be graded is where nearly all of them die. From a real run over 1,173 extracted claims:
Claims surviving to a grade
One real run| Claims extracted | 1,173 | 100% |
|---|---|---|
| Had a parseable deadline | 658 | 56% |
| Resolvable against a price | 57 | 4.9% |
| Actually graded | 5 | 0.43% |
Accuracy measurements
Not yet reportable| Measurement | State |
|---|---|
| Brier score | Not yet measurable |
| Calibration | Not yet measurable |
| Lead time vs. market price | Not yet measurable |
Five resolved outcomes cannot support any of these numbers. They are listed because the machinery to compute them exists and runs — not because there is a figure to show. When there is one, it appears here.
That funnel is the single most useful thing the system has produced so far, and it is the sort of number that never appears in a vendor deck, because it is the number that says this is harder than it looks. It is also what tells you where to spend the next month of engineering: not on extracting more, on extracting things that can be scored.
What it has not proven
Kengly does not claim an edge. It runs nightly, it grades what it can grade, and the volume of resolved outcomes is still small — small enough that no honest claim about accuracy can be made from it yet.
That is stated here deliberately. A forecasting system that leads with a track record before it has one is telling you something about its methods, and none of it is good. When there is enough resolved history to say something, it will be said here, with the losses in the same table as the wins.
Why this might matter to you
Strip out the subject matter and what is left is a general shape: take work someone would otherwise do by hand every day, have a system do it instead, and keep an honest measurement of whether the output can be trusted. Most of that machinery — the grading, the track records, the refusal to score yourself in hindsight — has nothing to do with finance and everything to do with whether an automated process deserves to be relied on.
Kengly is the working example
Nothing is sold here and there is nothing to sign up for. The mechanisms are public; read them, or write to the person who built it.