An assistant that answers from memory and then offers to check is doing the work in the wrong order. It is fast, it is usually right, and the times it is wrong look exactly like the times it is right. The whole point of the rebuild was to invert that: the default for a question about the world is now to go and find out, and to show you what it found.
What shipped
- A grounding check that runs after the answer. The app reads the finished reply and labels any paragraph that asserts a figure, date or named entity without a source. It is not a claim the line is wrong — it is an accurate statement that nothing in the source list supports it.
- Answer, Sources, Steps, Images. The evidence stopped competing with the answer for space.
- Answer-shape enforcement. An opening sentence with no content in it is removed before you see it.
- Related searches from the actual subject. Derived from the entities and figures in the answer, not from a fixed template.
- News, with topics you follow and a search box that searches the web rather than filtering the sixty rows already on screen.
- Exact-answer tools. Arithmetic, units, currency, weather, time and places — the cases where recall is least trustworthy.
The bug
We built a benchmark to check whether the citation rules were landing. It scored close to zero, which was suspicious enough to go looking — and the first thing we found was that the benchmark itself had never sent the system prompt at all. It was grading the model on rules it had never been given.
Fixing that surfaced the real problem. The assembled prompt was about 21,000 characters and the client truncated it at 8,000. The server-generated portion alone was around 14,000, so the cut landed inside it — and everything after it was discarded. That everything was the search posture, the anti-hallucination rules, the persona and the format policy. On every turn, for every user.
The model was being told to cite its sources by a paragraph it never received. The cap was applied to the rules and the context together, so the rules lost.
The fix is small and slightly embarrassing: budget the part that genuinely varies — retrieved pages, search results, memory — and append the policy afterwards, outside the cap. There was already a comment in the file saying two other blocks sat outside the cap "deliberately". The policy should always have been the third.
Why measuring came first
None of that was findable by using the product. The answers looked fine. They were fine, mostly — a good model with no instructions still writes well. What it does not do is reliably cite, and you cannot see the difference between "cited because it was told to" and "happened to cite" without running a few hundred cases and counting.
So the suite stays. 800+ graded cases, run against the live product on a schedule, with transport failures recorded as harness breakage rather than as the model getting it wrong — a benchmark that cannot tell those apart is not a measurement, it is a mood.
Next
The full grounding benchmark against the fixed build, then the rest of the tool set — productivity, OCR and PDF, and eventually a code sandbox that is genuinely contained. Those are on the roadmap, labelled by how real they are.