Machine Learning
Fine-tuned Legal-BERT to 80.75% F1, then RAG-scored risk against a real contract
Upload a PDF contract and five agents work through it in sequence: parse, classify every clause into one of 41 legal categories, score its risk against a reference corpus, explain the flagged ones, and generate a colour-coded PDF report — validated against a real freelance agreement, not a synthetic test.
Stack
- Python
- PyTorch
- OpenAI
- React
- TypeScript
80.75%
weighted F1
81.76% accuracy, 1,316-example held-out validation set
41
legal clause categories
CUAD taxonomy, 11,839 training clauses
9H · 6M · 1L
risk flags, real contract
validated end-to-end, live API calls
The brief
Overview
Built with one teammate (Abdullah Ghaffar) as an academic submission for Generative AI + Process Mining coursework: a 5-agent asynchronous pipeline that takes a PDF contract and produces a full, colour-coded risk report — classification, risk scoring, and LLM-grounded explanation, end to end.
Validated against a real document, not a synthetic one: the finished pipeline was run end to end against my own real freelance contractor agreement, from live OpenAI and ChromaDB Cloud calls, producing a genuine 16-clause report (9 High, 6 Medium, 1 Low).
Architecture
The five agents
Parse, classify, score, explain, report — each a distinct agent, streamed to the frontend as it runs.
Parse the PDF — contract text extracted into clauses
Classify each clause — 41 CUAD legal categories — see the classifier note below
RAG-score risk — all-MiniLM-L6-v2 embeddings vs. a curated CUAD-sourced clause corpus, in ChromaDB Cloud
Explain flagged clauses — GPT-4o-mini, with a cost-optimisation fast-path — see below
Generate the report — a colour-coded, LLM-grounded PDF
Per-agent, per-substep progress streams to the React frontend over native WebSockets, backed by a self-written asyncio.Queue pub/sub layer — a deliberate zero-dependency substitute for Celery/Redis, not "LangChain ReAct orchestration."
How the models actually work
The classifier and the risk scorer
Fine-tuned Legal-BERT — `nlpaueb/legal-bert-base-uncased`, fine-tuned on 11,839 CUAD legal clauses across 41 categories (single-label softmax, 3 epochs on a Colab T4 GPU) — 80.75% weighted F1 / 81.76% accuracy on a 1,316-example held-out validation set, logged from the training run itself.
Keyword-first, BERT on disagreement — The classifier is a 32-category keyword-heuristic layer first; the fine-tuned model’s label is added only when the keyword layer hasn’t already caught it, capped at 3 labels per clause — a 3-tier fallback chain, not a probabilistic fusion of two models.
RAG risk scoring — `all-MiniLM-L6-v2` embeddings queried against a curated, CUAD-sourced reference-clause corpus in ChromaDB Cloud, averaging the top-3 cosine-similarity distances to flag each clause Low, Medium or High risk.
A cost-optimisation fast-path — A clause within 0.20 cosine distance of a standard reference clause skips the GPT-4o-mini call entirely, via a deterministic templated explanation and a SHA-256-keyed 24-hour cache — cutting both latency and per-document API spend on the common case.
Why a hand-written pub/sub layer, not Celery/Redis
A 5-agent async pipeline with live progress streaming is exactly the kind of thing Celery and Redis exist for. Why build a plain `asyncio.Queue` layer instead?
The whole pipeline needed to run with no external infrastructure — no Redis instance, no message broker — for a coursework submission that has to be runnable on whatever machine grades it.
FastAPI’s own `BackgroundTasks.add_task()` plus a self-written `asyncio.Queue` pub/sub layer gets the same per-agent, per-substep progress streaming a Celery/Redis setup would, at zero deployment cost.
The alternative
Celery workers with a Redis broker, as the design docs originally specified.
What it cost
The hand-written layer doesn’t survive a process restart mid-job the way a real message broker would — an acceptable trade for a project this size, not a claim it would scale past one.
The honest number on latency
Outcome
Selected results
80.75%
weighted F1
81.76% accuracy, held-out validation set
41
legal clause categories
CUAD taxonomy
211–247s
real end-to-end run
against a live 16-clause contract, not simulated
9H · 6M · 1L
risk flags produced
from live OpenAI + ChromaDB Cloud calls
Retrospective
What I would take to the next build
Log real numbers before writing the pitch — The logged 211–247 second run, and the 80.75% F1 from the training notebook, are what this case study stands behind — a design-doc target is not a result until it’s measured.
A zero-dependency substitute can be the better engineering story — The self-written pub/sub layer isn’t a downgrade from Celery/Redis — it’s the correct answer for a pipeline that has to run with no external infrastructure, and it’s a more interesting one to explain.
Built with one teammate; my own verified contribution is the Legal-BERT fine-tuning notebook, the RAG risk-scoring agent, and the LLM cost-optimisation fast-path.