AI-generated reports have a problem: they sound just as convincing when they are right as when they are making things up. A number with no source, a trend nobody measured, a competitor that does not exist. All in the same confident tone.
So I set out to build a research system I could audit.
How it is built
I write a request in plain language and a planner puts together a custom team from 16 specialists. It does not call on all of them, only the ones it needs. Each specialist searches the web and data sources and writes its part.
Then two checks go over the work. One is a verifier that opens the sources and confirms every data point, marking it as verified, corrected, declared estimate or unverified. The other is a deterministic quality judge that checks whether they did what was asked and, if something is missing, sends them back for another try.
The report shows a reliability index: the percentage of data points confirmed against their source. The system also keeps memory between runs. It is built in Python with CrewAI and Gemini, and an average run costs ten cents.
The predictor that tied with the dumbest rule
Since the system kept memory, I added a predictor: by looking at similar situations in the past, it estimated where the price would go over the next 7 days.
But before trusting it I tested it against 8 years of historical data, and the result was uncomfortable: it tied with the dumbest rule possible, "the price always goes up". All that complexity added nothing.
Five theses that fell on the same day
In August the system wrote five claims along the lines of "Bitcoin will stay in this range over the next month". On 19 August Bitcoin rose almost 8% and all five fell together.
Digging into why they fell together, I found something worse than a forecasting error: the theses were impossible from the start. I measured 9 years of history, and a band that narrow had never held for 30 days. Not once in more than 3,000 cases.
The most frustrating part was that the system already had that tool and was not using it on its own claims.
And there was another, more hidden problem. Range theses were being checked on one side only, when they can fail either above or below. That inflated the hit record by about 30 points.
How I fixed it
I changed the rules and put strict filters on the model:
- The predictor got a statistical filter: it only speaks if its edge is real, with less than a 1% probability of being chance. Today it almost never makes a prediction, and that is the honest result.
- Every thesis has to pass a filter before it gets written: if historically it has less than a 20% chance of coming true, it is rejected.
- I fixed how range theses are evaluated so both sides are checked, above and below. That removed the bias from the hit record.
An AI system is not good because it sounds smart, but because it can show when it is right and admit when it is not. Verification, testing against real data and filters that tell the model "no" are what ended up working.
What I took from it
The agentic loop matters so that the LLMs and agents themselves check each other and reach the goal.
The deterministic judge and the source verifier are essential when scraping, because you do not know whether the information they bring back is real. Comparing sources is critical.
And the most valuable part of a project is the one that forces you to admit you were wrong. Do not trust the model because it sounds convincing; build filters and tests with real data that tell it "no".