In late February the compliance team at a Santiago-based AGF asked for help. They were triaging roughly 1,200 inbound regulatory documents per month — circulars, supplier KYC packs, fund-document requests — by hand. Each document had to be classified into one of six types, routed to the right operator, and logged against the relevant audit trail.
The triage was eating a third of the team’s bandwidth.
We agreed to build a classifier in two weeks. We delivered it in nine days. This is the build log.
Day 1 — The brief
The first thing we did was not write code. We sat with the compliance team for two hours and built the eval set. Two hundred real documents, manually labeled across six categories, with a 30-document holdout for final eval.
This is the most important hour of any AI Lab engagement. If you can’t agree on the eval criteria, you can’t agree on the answer. We’ve watched teams spend twelve weeks on a model and then disagree about whether it works.
The six categories ended up being:
- Regulatory circular — official communication from CMF / UAF
- Supplier KYC — onboarding documents from custodians and counterparties
- Fund document request — investor relations document pulls
- Internal escalation — issues raised by the compliance team to legal
- Routine acknowledgment — confirmations, receipts, low-priority notes
- Other / human review — escape hatch for ambiguous cases
The holdout set was the inspector — anything we couldn’t get above 92% accuracy on, we didn’t ship.
Day 2 — First pass
A sensible engineer would have started with a fine-tuned classifier. We started with vanilla Claude 3.7 Sonnet and a careful prompt. The reason: classification problems with rich text and explicit category definitions are exactly where modern instruction-tuned models shine, and you don’t want to spend a week training something a prompt can do.
First-pass accuracy on the eval set: 88%. Below threshold. But the failure modes were illuminating.
Category 1 (Regulatory circular): 96% accurate
Category 2 (Supplier KYC): 91% accurate
Category 3 (Fund document): 89% accurate
Category 4 (Internal escalation): 71% accurate ← bad
Category 5 (Routine ack): 95% accurate
Category 6 (Other): (tail of failures)The model couldn’t reliably distinguish category 4 from categories 2 and 3. The failure mode wasn’t capability — it was a definitional ambiguity in the spec. Some “supplier KYC” documents were internal escalations because legal needed to weigh in.
Day 3 — Re-spec, not re-train
Instead of fine-tuning, we re-spec’d the categories. Category 4 became “anything that requires sign-off from a named operator” — orthogonal to the document type. The other five remained content categories.
The classifier now ran in two passes: classify the document type (1, 2, 3, 5 or 6), then classify whether sign-off is required (yes/no). Two simpler classification calls instead of one harder one.
Eval accuracy after the re-spec: 94% on document type, 96% on sign-off. Above threshold. We didn’t fine-tune anything.
The classifier wasn’t broken. The taxonomy was. Spending a day fixing the taxonomy was worth more than a week of model work.
Days 4–6 — The harness, the UI, the audit trail
With the model behavior locked in, the rest was engineering. Three things to build:
1. The harness. Eval set runs on every commit. Catches regressions automatically. Ours runs in 11 seconds against a sample, 6 minutes against the full holdout. The full run gates production deploys.
2. The triage UI. A simple queue view for the compliance team. Each document shows the model’s classification, its confidence and the source RAG citation that drove the decision. Operators can override; overrides feed back into a re-eval set we use for monthly drift checks.
3. The audit trail. Every classification logged with the model version, the prompt version and the input document hash. Operator attribution on every override. The compliance team’s auditors can trace any decision to its inputs in under a minute.
Days 7–8 — Eval, stress, deploy
The full holdout eval ran cleanly. We then ran adversarial inputs (truncated documents, multi-language documents, scanned PDFs with OCR errors) and noted where the classifier degraded. Below 60% confidence, the document goes to human review automatically — even if the classifier had a guess.
Cutover to production happened on day 8. We deployed gradually: first 10% of incoming documents, monitored for 24 hours, then 100%.
Day 9 — Live in production
Production load on day one was 47 documents. The model processed them in under 30 seconds. Operator overrides on the first day: two. Both were edge cases the eval set hadn’t covered, so they went into the eval set for next month’s drift check.
A month later, the numbers settled at:
- 78% reduction in manual triage time
- 94% accuracy on document type (held since day one)
- 0 hallucinations live (RAG with provenance prevents the model answering without a source)
- 6 / 6 model deploys passed the eval harness gate
What we’d do differently
Two things, neither of them dramatic.
Build the eval-set pipeline before the prompt. We hand-built the eval set on day one. Next time we’d build a small tool that lets the compliance team add eval cases as they appear. We did this manually for the first month and it worked, but the tool would have been worth a day’s work.
Ship the override flow earlier. The override flow shipped on day 7. Two days earlier would have given us more signal for adversarial cases during the eval phase.
The classifier is still running. It runs about 1,500 documents per month now. It’s never been re-trained. The eval set has grown to 340 cases. Drift remains within tolerance.
Diego Vargas builds AI agents at finnerve’s AI Agents. If you’re sizing an AI engagement and want to talk through the eval-first approach, start a conversation.