A deterministic calculation tool for legal agents

Language models are competent at legal reasoning and unreliable at the arithmetic that reasoning produces. The unreliability is not spread evenly: it concentrates in exactly the figures a client acts on without re-deriving them — a filing deadline, a limitation date, a penalty figure, a total across line items.

A representative failure. Eighteen months of monthly compounding, then a daily penalty capped at a percentage of the balance. The correct total is 3,285,085.55. The model returned 3,285,083.95 — and at temperature zero, returned exactly that five times out of five.

The size of the error is not the interesting part. What matters is that the figure is the sum claimed in a filing, that nothing in the answer looks wrong, and that re-running or asking a second model surfaces nothing, because the mistake is deterministic. Prose gets re-read. Numbers get copied.

Why not simply let the model calculate

The obvious design is to let the model write an arithmetic expression and have the tool evaluate it. We rejected that, for a reason that turned out to matter more than the security argument against executing model-authored strings.

Legal arithmetic is under-determined until a counting convention is fixed, and a bare expression throws that information away. One month from 31 January is either 28 February or 2 March. A notice period either counts the day of service or starts the day after. A deadline landing on a public holiday either moves or does not, and either forward or back. Each choice is defensible, each produces a different answer, and an evaluator hands back a number with the choice invisible inside it.

So the tool is a fixed set of named operations rather than a calculator, and every result states the convention it applied — not in documentation, but in the answer the model receives. A thirty-business-day period comes back with the end date, the weekday it falls on, the fact that weekends were skipped, and the explicit note that the start date was not counted and which day was therefore day one.

A wrong assumption then appears in the answer instead of hiding inside it, which means a lawyer can challenge it.

What it computes

The operations cover the arithmetic that legal work actually turns on: adding and subtracting periods in days, business days, weeks, months or years; the distance between two dates; moving a date off a non-working day under a stated convention; checking whether a set of dates are working days; penalty interest with an optional contractual cap; compound interest; adding or extracting VAT; percentages and percentage changes; sums and averages.

Some of the choices inside that list are the whole point of the exercise. Month arithmetic clamps to the length of the target month, so 31 January plus one month is 28 February — the convention every civil code and contract drafter means, and the one place a naive thirty-day addition goes wrong. The VAT rate has no default, because statutory rates change and a stale default is precisely the silently-wrong number the tool exists to prevent. A percentage change names which figure is its denominator, because a rise from 100 to 150 and a fall from 150 to 100 are the same gap and different percentages. When a penalty cap binds, the result shows the uncapped figure beside the capped one, so the reduction is visible rather than assumed.

Money is handled in exact decimal arithmetic throughout, not floating point, and figures are read the way they actually appear in documents — both European and Anglo grouping conventions, including the non-breaking spaces that arrive when a number is copied out of a PDF. Dates are accepted in two unambiguous formats and refused otherwise: a bare 03/12/2026 is rejected rather than guessed, because a month/day transposition is a wrong deadline and a guess produces one silently.

Working days across jurisdictions

Weekends are trivial. Public holidays are not, and a hand-maintained table of them is a data-entry commitment that quietly goes stale. We build on a maintained public dataset instead: 251 ISO countries and territories, of which 28 have a working week that is not Monday to Friday. Saudi Arabia runs Sunday to Thursday; the UAE moved from a Friday/Saturday weekend to Saturday/Sunday in 2022. The tool reads each country's own working week rather than assuming a European one.

No country list, weekend rule or holiday date is written into our code. Adding a jurisdiction takes no code at all — only its ISO code.

Two decisions sit on top of that data. The first is a concurrency matter: the underlying dataset fills in a year the first time it is asked about, which modifies it in place, and our chat workers run several threads in one process. Two simultaneous first-requests for the same year could see it half-filled and miss a holiday — a wrong deadline, occasionally, and effectively impossible to reproduce from a bug report. We therefore load a fixed range of years once and freeze it, removing the race by construction rather than locking around it. It also made the common path faster.

The second is about the edge of that range. "No holidays that year" and "no data for that year" produce identical arithmetic and completely different confidence, so a date outside the loaded range is reported as uncovered rather than quietly treated as a run of ordinary working days.

Measuring whether it helps

We measured rather than assumed. Ten tasks where the answer is a number, each solved twice by the same model with the same prompt: once answering directly, once with the tool available and executing real calls. The only variable is whether the tool is on the table.

Those ten were selected from a pool of thirty by what they actually measure. Twenty of the thirty were answered correctly on every run by both arms — a task both sides always get right discriminates nothing — so they were dropped. What remains is the eight where an untooled model demonstrably fails, plus two kept for structure: a debt balance that changes between periods, and a twelve-date payment schedule.

The harder part of a benchmark like this is not running it but trusting the expected answers. Ours are derived in a file that never imports the tool, so a bug shared between the tool and its own test cannot hide. The facts underneath are asserted rather than assumed — that a particular date is a Japanese public holiday while the day beside it is an ordinary Friday, that Christmas Day is a US federal holiday and Boxing Day is not. And a separate pass confirms the tool reproduces every expected answer before any model runs, so a later failure is attributable to the model's use of the tool rather than to the arithmetic.

Results

DeepSeek v4-pro, temperature 0, five repetitions per task, 100 runs.

The cost is about 1.6 times the wall-clock time per task, and many more tokens. That second figure reads worse than it is: roughly 95% of them are input tokens, which are cheaper and cacheable, and the tool-using run actually produces fewer output tokens, because it makes short calls instead of reasoning at length about arithmetic it cannot do.

The failure modes are more instructive than the percentage. On the two-thousand-working-day horizon the untooled model missed by more than three months, five times out of five. On a task with a five per cent cap it dropped the cap in three runs out of five. On one deadline question it returned four different answers across five runs at temperature zero — which is at least visibly unreliable, unlike the compounding error above.

What the measurement changed

Every task the agent failed turned out to be a defect on our side rather than a limit of the model, and each was a design lesson rather than a typo.

The sharpest one: the tool does not count the start day of a period, and said so clearly — in the result, which the model reads after the call has already happened. Asked for ten business days from a given date, the model conscientiously worked out the first working day after it, passed that as the start, and the tool skipped a day again. Every answer landed exactly one working day late. Moving that sentence into the description of the parameter, where it is read before the call, fixed it outright.

The most structural: computing a deadline and moving it off a non-working day are a single rule in every civil code we know of, but were two separate operations in our interface — and the second is the one that gets skipped. It is one call now.

One correction ran the other way, which is worth reporting too. Legal periods are written "from 1 January to 20 March inclusive", with both named days inside, while our day count begins the following day. Adding an option to say so directly looked obviously right, and made the agent measurably worse: it then had two levers for one decision and pulled both, over-counting where it had been under-counting. We reverted it. Stating a convention where the model reads it helps; adding a switch for the same convention did not.

The rest were plainer engineering. A lookup keyed by the spelling a caller used rather than the resolved country, so one jurisdiction written two ways disagreed with itself. A formatting path that could throw on a very large figure. The shared-cache race described earlier.

Limits

• 100% is a result on these ten tasks, not a property. The score moved between 83% and 100% during development as defects were found and fixed; the hardest tasks found two that the easier ones never touched.
• Holiday dates come from one dataset. The arithmetic performed over those dates is independently verified; the reference data itself is not.
• Five repetitions is a small sample, and the measured agent is the tool-calling loop rather than the full production assistant with its own prompting.
• The tool performs no legal reasoning. It computes exactly what it was asked. If the model asks the wrong question — a penalty on the gross instead of the net, a skipped step — the answer will be precisely wrong. What changes is that the mistake shows up in a trace and is fixable, rather than varying from run to run.
• Task wording was revised twice for ambiguity, never for score; the expected answers never changed. One finding transfers to anyone writing evaluations: a formula written out in words is measurably more ambiguous than the same formula in notation, and cost one task a perfect score before we noticed.