Case and article lookups, twice as good

Search over a document set only works if the agent asks it the right kind of question. A case number, a complaint number, an article reference — these are strings that either appear in a document or don't, and finding them is a job for exact-word search. "What arguments did the respondent raise about jurisdiction" has no fixed wording to match against; it's a job for search by meaning. The two calls are different tools, and the agent has to pick correctly before it searches at all.

We reworked how that choice gets made, and measured the effect on a test set of 108 questions, 432 runs in total, backed by more than 2,000 validation runs while the change was being tuned.

Results

Alongside the accuracy gain, the agent does less unnecessary work: 29% fewer searches overall, because it reaches for exact-word search less often in cases that actually call for search by meaning. On case-number lookups specifically, it now gets the same result in 20% fewer steps.

What changed

A query built around a concept is now sent to semantic search as a single sentence, not chopped into keywords first — cutting a question down to its keywords is exactly what turns a meaning-search into an accidental word-search, with all of the meaning-search's precision lost and none of exact-search's guarantees gained. Identifiers — a case number, a citation, a term the user typed exactly — get their own lexical search, run in parallel rather than folded into the same query as the surrounding concept. Each fact the agent needs is now looked up on its own, instead of one query trying to carry several facts at once and returning a passage that partially matches all of them and fully matches none.