Notes from building a system that has to be checkable
Design decisions and the reasoning behind them, measurements with their methodology attached, and the things we got wrong before we got them right.
-
Case and article lookups, twice as good
We taught the search agent to tell a lexical query from a semantic one. Case numbers and article references are now found in 99.8% of cases instead of 48.6%, empty results dropped from 32% to 11%, and the agent takes fewer, more targeted steps to get there.
-
A deterministic calculation tool for legal agents
Language models are competent at legal reasoning and unreliable at the arithmetic it produces — and the unreliability concentrates in exactly the figures a client acts on without checking. Why we built a calculation tool instead of letting the model do the sums, how it handles working days across 250 ISO countries and territories, and a measured comparison: 60% of answers correct without it, 100% with it.