A deterministic calculation tool for legal agents

Language models are competent at legal reasoning and unreliable at the arithmetic that reasoning produces. The unreliability is not spread evenly: it concentrates in exactly the figures a client acts on without re-deriving them — a filing deadline, a limitation date, a penalty figure, a total across line items.

A representative failure. Eighteen months of monthly compounding, then a daily penalty capped at a percentage of the balance. The correct total is 3,285,085.55. The model returned 3,285,083.95 — and at temperature zero, returned exactly that five times out of five.

The size of the error is not the interesting part. What matters is that the figure is the sum claimed in a filing, that nothing in the answer looks wrong, and that re-running or asking a second model surfaces nothing, because the mistake is deterministic. Prose gets re-read. Numbers get copied.

Why not simply let the model calculate

The obvious design is to let the model write an arithmetic expression and have the tool evaluate it. We rejected that, for a reason that turned out to matter more than the security argument against executing model-authored strings.

Legal arithmetic is under-determined until a counting convention is fixed, and a bare expression throws that information away. One month from 31 January is either 28 February or 2 March. A notice period either counts the day of service or starts the day after. A deadline landing on a public holiday either moves or does not, and either forward or back. Each choice is defensible, each produces a different answer, and an evaluator hands back a number with the choice invisible inside it.

So the tool is a fixed set of named operations rather than a calculator, and every result states the convention it applied — not in documentation, but in the answer the model receives. A thirty-business-day period comes back with the end date, the weekday it falls on, the fact that weekends were skipped, and the explicit note that the start date was not counted and which day was therefore day one.

A wrong assumption then appears in the answer instead of hiding inside it, which means a lawyer can challenge it.

What it computes

The operations cover the arithmetic that legal work actually turns on: adding and subtracting periods in days, business days, weeks, months or years; the distance between two dates; moving a date off a non-working day under a stated convention; checking whether a set of dates are working days; penalty interest with an optional contractual cap; compound interest; adding or extracting VAT; percentages and percentage changes; sums and averages.

Some of the choices inside that list are the whole point of the exercise. Month arithmetic clamps to the length of the target month, so 31 January plus one month is 28 February — the convention every civil code and contract drafter means, and the one place a naive thirty-day addition goes wrong. The VAT rate has no default, because statutory rates change and a stale default is precisely the silently-wrong number the tool exists to prevent. A percentage change names which figure is its denominator, because a rise from 100 to 150 and a fall from 150 to 100 are the same gap and different percentages. When a penalty cap binds, the result shows the uncapped figure beside the capped one, so the reduction is visible rather than assumed.

Money is handled in exact decimal arithmetic throughout, not floating point, and figures are read the way they actually appear in documents — both European and Anglo grouping conventions, including the non-breaking spaces that arrive when a number is copied out of a PDF. Dates are accepted in two unambiguous formats and refused otherwise: a bare 03/12/2026 is rejected rather than guessed, because a month/day transposition is a wrong deadline and a guess produces one silently.

Working days across jurisdictions

Weekends are trivial. Public holidays are not, and a hand-maintained table of them is a data-entry commitment that quietly goes stale. We build on a maintained public dataset instead: 250 ISO countries and territories, of which 28 have a working week that is not Monday to Friday. Saudi Arabia runs Sunday to Thursday; the UAE moved from a Friday/Saturday weekend to Saturday/Sunday in 2022. The tool reads each country's own working week rather than assuming a European one.

No country list, weekend rule or holiday date is written into our code. Adding a jurisdiction takes no code at all — only its ISO code. Nor is there a default to fall back on: if a jurisdiction's own working week cannot be read, the call is refused rather than answered against an assumed Monday to Friday. That assumption is right for most of the world and wrong for exactly the twenty-eight places this machinery exists to get right, which makes it the worst possible thing to guess.

A weekend is not only a property of the country, either. When a government moves a public holiday's day off onto a weekday, the weekend day it was moved from becomes a working day — Russia has had five such Saturdays since 2022, and a deadline crossing one is a day shorter than it looks. The dataset publishes those days; the tool reads them, and a decreed working Saturday beats the weekend it lands on.

Three decisions sit on top of that data. The first is a concurrency matter: the underlying dataset fills in a year the first time it is asked about, which modifies it in place, and our chat workers run several threads in one process. Two simultaneous first-requests for the same year could see it half-filled and miss a holiday — a wrong deadline, occasionally, and effectively impossible to reproduce from a bug report. We therefore load a fixed range of years once and freeze it, removing the race by construction rather than locking around it. It also made the common path faster.

The second is about the edge of that data, and the edge is not one line across the world. Sri Lanka's calendar ends in 2026 and Japan's in 2099; dozens of countries begin at independence rather than at our earliest year. India is subtler still: its fixed national days are generated for any year at all, while its Hindu festivals are tabulated only through 2035, so an answer for 2040 would quietly lose Diwali and keep enough dates to look complete. "No holidays that year" and "no data for that year" produce identical arithmetic and completely different confidence, so each jurisdiction is asked where its own data stops, and a date past that edge is reported as uncovered rather than treated as a run of ordinary working days.

The third is about a kind of staleness no dataset can fix: a public dataset is always behind a government. Russia's transferred days off for a year are set by decree the previous autumn and reach the dataset later — ours stopped at 2025 while our customers were already working through 2026. So a small second layer sits beside it: a table of calendars we have checked ourselves against the published decree, each recording the year it was checked through. Past that year the statutory dates are still applied and the answer says plainly that the moves around them are unconfirmed. Where a date did come from that layer, the result cites the decree rather than the dataset, because a citation that sends a reader to a source not containing the date is worse than no citation at all.

Measuring whether it helps

We measured rather than assumed. Eleven tasks where the answer is a number, each solved twice by the same model with the same prompt: once answering directly, once with the tool available and executing real calls. The only variable is whether the tool is on the table.

Ten of them were selected from a pool of thirty by what they actually measure. Twenty of the thirty were answered correctly on every run by both arms — a task both sides always get right discriminates nothing — so they were dropped. What remains is the eight where an untooled model demonstrably fails, plus two kept for structure: a debt balance that changes between periods, and a twelve-date payment schedule.

The harder part of a benchmark like this is not running it but trusting the expected answers. Ours are derived in a file that never imports the tool, so a bug shared between the tool and its own test cannot hide. The facts underneath are asserted rather than assumed — that a particular date is a Japanese public holiday while the day beside it is an ordinary Friday, that Christmas Day is a US federal holiday and Boxing Day is not. And a separate pass confirms the tool reproduces every expected answer before any model runs, so a later failure is attributable to the model's use of the tool rather than to the arithmetic.

Results

DeepSeek v4-pro, temperature 0, five repetitions per task, 110 runs.

The cost is about 1.6 times the wall-clock time per task, and many more tokens. That second figure reads worse than it is: roughly 95% of them are input tokens, which are cheaper and cacheable, and the tool-using run actually produces fewer output tokens, because it makes short calls instead of reasoning at length about arithmetic it cannot do.

The failure modes are more instructive than the percentage. On the two-thousand-working-day horizon the untooled model was wrong five times out of five, missing by as much as six weeks and returning four different answers across those five runs — at temperature zero. That at least looks unreliable. The compounding task is the opposite case and the worse one: wrong five times out of five as well, but with the same figure every time, and nothing about a stable answer invites a second look. In between sit the tasks it got right three or four times in five, which is the pattern that makes an untooled model feel dependable until it is not.

What the measurement changed

Every task the agent failed turned out to be a defect on our side rather than a limit of the model, and each was a design lesson rather than a typo.

The sharpest one: the tool does not count the start day of a period, and said so clearly — in the result, which the model reads after the call has already happened. Asked for ten business days from a given date, the model conscientiously worked out the first working day after it, passed that as the start, and the tool skipped a day again. Every answer landed exactly one working day late. Moving that sentence into the description of the parameter, where it is read before the call, fixed it outright.

The most structural: computing a deadline and moving it off a non-working day are a single rule in every civil code we know of, but were two separate operations in our interface — and the second is the one that gets skipped. It is one call now.

One correction ran the other way, which is worth reporting too. Legal periods are written "from 1 January to 20 March inclusive", with both named days inside, while our day count begins the following day. Adding an option to say so directly looked obviously right, and made the agent measurably worse: it then had two levers for one decision and pulled both, over-counting where it had been under-counting. We reverted it. Stating a convention where the model reads it helps; adding a switch for the same convention did not.

The rest were plainer engineering. A lookup keyed by the spelling a caller used rather than the resolved country, so one jurisdiction written two ways disagreed with itself. A formatting path that could throw on a very large figure. The shared-cache race described earlier.

Two later findings, and one lesson about our own code

The working-day machinery was audited again some months on, and it had two silent errors in it — the kind that produce a confident answer rather than a failure.

The first: we were reading only half of what the dataset publishes about a transferred holiday. It records the weekday that becomes a day off and, separately, the weekend day that goes to work in exchange. We took the first list and not the second, so 27 April 2024 was a Saturday to us and a working day to everyone in Moscow. Any deadline crossing one of those five Saturdays came back a day late — inside a year we were reporting as fully covered, which is the worst combination available.

The second was a claim rather than a calculation. The tool's own description promised to say so when a year's transfer data was unverified. It had once done that, from a table since removed, and the sentence had outlived the behaviour. So from 2026 onwards it applied Russia's statutory dates, knew nothing about the moves around them, and said "calendar applied". A promise in a tool description is not documentation — it is the thing the model reasons from — and one that has quietly stopped being true is worse than never having made it.

The lesson was about us, and it is the one worth passing on. Fixing the first defect, we wrote a mechanism to reconstruct those working Saturdays from the dataset's internal tables: parsing tuple shapes, walking a private attribute, even reading the text of a warning it emits about one country. It worked, and it was all unnecessary. The dataset publishes exactly that list under a plain name, and a method that answers the whole question directly. Reading the library properly deleted more than two hundred lines and every piece of reverse engineering in them, and left a rule simple enough to state in a sentence: the dataset publishes facts, and our own table declares only what we have checked ourselves.

Limits

• 100% is a result on these eleven tasks, not a property. The score moved between 83% and 100% during development as defects were found and fixed; the hardest tasks found two that the easier ones never touched.
• Holiday dates come from one public dataset, plus a small table of calendars we have checked against a published decree. The arithmetic performed over those dates is independently verified; the reference data itself is not, in either layer. A jurisdiction the dataset does not cover is one the tool does not cover.
• A checked calendar is only checked as far as somebody checked it. Each jurisdiction records the year it was verified through, and past that year the answer says so rather than implying more.
• Five repetitions is a small sample, and the measured agent is the tool-calling loop rather than the full production assistant with its own prompting.
• The tool performs no legal reasoning. It computes exactly what it was asked. If the model asks the wrong question — a penalty on the gross instead of the net, a skipped step — the answer will be precisely wrong. What changes is that the mistake shows up in a trace and is fixable, rather than varying from run to run.
• Task wording was revised twice for ambiguity, never for score; the expected answers never changed. One finding transfers to anyone writing evaluations: a formula written out in words is measurably more ambiguous than the same formula in notation, and cost one task a perfect score before we noticed.