Analysts at a national financial regulator have to check, fund by fund, that each investment fund stays inside the limits its own policy document sets. I led the build of a system that reads the Hebrew policy text with Claude, turns every limit into a structured rule, and checks it against the fund's monthly holdings report. The model translates legal wording into filters. Code does every sum and every comparison.
Every fund publishes an investment policy in free Hebrew text: a floor on government bonds, a ceiling on “other channels as permitted by law”, a rating threshold, a foreign-currency band, no equities at all. Each month the fund manager files a holdings report with one row per security, 64 to 106 rows per fund in the pilot set. Checking one against the other meant reading the legal wording, working out which rows it covers, and adding up exposures by hand. Then the analyst moved on to the next fund.
The diagram is wider than the screen: swipe sideways to see all of it →
Each run is a job with a status (start, preliminary pass, primary pass, completed), a timed duration and a step-by-step log in the database. The text of every stage (raw, cleaned, model output) is written to S3, so any result can be traced back to the exact words the model saw.
In the first version Claude extracted the limits, extracted the actuals and compared them. 27 of 72 checks came back “cannot determine”, and one row called 1.6% “within” a 10% minimum. Now the model only turns a clause into a JSON filter over known values (asset types, trading venues, currencies, rating range). C# applies it, sums exposure and compares it to the limits. Trade-off: the schema has to grow with new kinds of clause (10 dimensions in the prototype, 12 in the MVP).
Some clauses depend on data the holdings report doesn't contain: maturity date, domestic affinity, credit-risk group. The model marks those filters, the engine stores no verdict, and the UI shows Untested. Trade-off: a lower coverage number, but never a false “compliant”.
Pass one isolates and cleans the investment-policy section of the filing. Pass two returns typed restrictions: source clause, type (exposure, holding, permitted), minimum, maximum and wording. Trade-off: two model calls per document, in exchange for much smaller and cleaner input to the pass that matters.
Prompts and their six parameters (max tokens, temperature, top-k, top-p, thinking mode, thinking budget) live in the database, one row per prompt type. That made moving from Claude 3.5 to 3.7 with extended thinking a configuration change. To catch drift, a reference check compares each extraction with a known-good set on count, minimum, maximum, source clause, type and required keywords.
Most of the design came from things that failed on real filings. A selection:
Policy documents arrived as scanned PDFs, so I ran Hebrew OCR at 300 dpi and asked the model to extract limits, extract actuals and compare them. 38% of checks ended in “cannot determine”, and the model got the comparisons themselves wrong.
Took arithmetic away from the model entirely and redesigned around one narrow task: clause in, filter out.
Clauses such as “assets other than government bonds and/or deposits” can't be written as a single filter.
Filters became an array (their union), each filter can be inverted, and the whole array can be inverted. The model picks the structure; code computes the set.
Equity and foreign-currency exposure come from summary lines in the report, not from adding up security rows. The row sum gave the wrong number.
Added a “single asset type” filter that reads the summary line directly.
Holdings mix S&P-style and Moody's-style ratings, and “A and above” has to include A- and A3.
Mapped both 26-grade scales to a single ordinal index, held in configuration. Rating ranges compare indexes, not strings.
Across 3 pilot funds and 28 restrictions, the prototype ran with 0 errors. 14 of the 21 exposures the regulator had measured matched within 0.05 percentage points.
Traced each miss. Most needed maturity, domestic-affinity or credit-risk data the holdings report doesn't carry, and one reference value was itself flagged for review. That finding became the Untested path, plus a list of external data sources for the next phase.
In production the policy is an HTML filing, and the policy text sits in a named form field whose name changed between form versions.
Parsed with AngleSharp and fell back between the two field names. A missing field returns an explicit “no policy text” status instead of an empty analysis.
Holdings reports are tab-separated, in a legacy Hebrew code page, with header variants between fund managers. For two asset types, one column carries a different meaning.
Eight header-normalisation rules run before typed CSV mapping, plus a per-asset-type column correction. Rows that don't belong to a known fund are dropped and logged.
Moving to Claude 3.7 with extended thinking, Bedrock refused requests that set temperature, top-k or top-p.
The provider sends sampling parameters only when thinking is off. When it's on, it sets temperature to 1 and a token budget, chosen per prompt type in the database.
These are synthetic clauses and numbers written for this page. They contain no client data or code. Each one shows what the model returns and what the code then does with it.
Free scoping call · 30 minutes · No commitment
Write a few lines: the process or system, what hurts today (time, errors, cost or an AI pilot that stalled) and who uses it. After a short call you get a written plan and a fixed quote within ≈3 days.
Opens a pre-filled email · reply within one business day
The model reads. The code counts.
All projects