Black Knight, Inc. was bringing the power of AI to the mortgage industry. AIVA is an AI that understands business grammar, learns skills, and acts as an alternative workforce for mortgage lenders — capable of learning new skills in days as it trains. One of its skills is Compile Loan: it organizes borrower, vendor, lender, and closing documents into a complete, indexed file, reading, classifying, and entering key data directly into the loan origination system. AIVA takes the first pass.
But in a regulated workflow, nothing the AI does can stand on its own. AIVA processes thousands of documents an hour, and every transaction lands in an operator’s queue to be verified and approved before the loan can move forward. The operator is the bottleneck by design — and per-loan review time was the cost that put the economics of the whole approach in question.
The screen operators had gave them none of AIVA’s work to build on. No queue depth, no visible confidence, no distinction between what had been checked and what hadn’t. Operators were re-deriving conclusions the AI had already reached.
The AI had already done the work. The interface made operators do it again.
The Approach
My hypothesis was that review time was an interface problem, not a model problem. If the screen surfaced AIVA’s confidence, highlighted exactly what it had already done, and made correction a single gesture, operators would stop re-deriving and start verifying — and per-loan review time would drop without touching the model.
I designed two operator surfaces against that hypothesis — Classify Documents and Data Extract — on one shared principle: the operator’s job is to correct AIVA’s labels, never to start from scratch. With a supervisor’s second-round review behind them, they make up the human-in-the-loop trust layer: confidence thresholds, verification, misclassification correction, and an audit trail that feeds every operator correction back into the model — the same record compliance relies on. I did this work as the sole designer, and carried it through the Black Knight acquisition.
AIVA’s accuracy and classification are rendered on every document set, scannable at a glance. An operator decides where to spend attention before opening anything — the low-confidence work announces itself instead of hiding in a queue that looks uniform.
Correction as a single gesture
Feedback controls sit inline with the thing being corrected. Fixing a misclassification is one action, not a detour — which is what makes correcting cheaper than re-deriving, and what keeps the training loop fed with real operator judgement.
Queue state visible without opening anything
Operators can see how many files are waiting, and tagged documents are visually distinct from untagged ones. Progress is a property of the screen rather than something the operator has to hold in their head across a shift.
Designed for the physical act of scanning
Sizable thumbnails, rapid scrolling, zoom in and out, and partial zoom on a single region of a page. At thousands of documents an hour, the ergonomics of reading are the product — a half-second of squinting per page is the difference between the target and missing it.
One audit trail serving compliance and training
Every transaction records whether AIVA was right and whether an operator intervened. That single record satisfies the regulated workflow and improves the model — rather than building a compliance log and a feedback pipeline as two separate systems.
Classify Documents
Classification is the first pass over a loan packet — AIVA sorts pages into document types, and an operator confirms or corrects the sort. It is the highest-volume screen in the workflow, so it set the ceiling on the whole review time.
Old design
the interface operators started with
What the redesign had to fix
Operators can see at a glance how many files are in the queue.
Highlight AIVA’s accuracy and classification on each set of documents for quick scans.
Quick controls for operator feedback.
Operators can go back through their work to verify or correct their classifications.
Tagged documents are visually distinct from untagged ones — labeled or moved to a "done" state.
Rapid scrolling through pages with sizable thumbnails, plus zoom in and out for a readable view.
Partial zoom so operators can expand the important region of a page.
New designs
1 / 7Classify Documents — redesigned
demo — Classify Documents in action
Data Extract
Once pages are classified, AIVA pulls the values that matter — borrower income, insurance terms, closing figures — and writes them into the origination system. Here an operator isn’t judging a category but a number, and a wrong number carries further than a wrong label.
Old design
data extraction, before the redesign
What the redesign had to fix
Operators can see how many files are in the queue and approve without hunting for data points.
Highlight AIVA’s accuracy and extracted data points on each page for quick scans.
Quick controls for feedback that help train AIVA by correcting errors.
Rapid scrolling through pages, with alternate data options surfaced when applicable.
New designs
1 / 4Data Extract — redesigned
demo — Data Extract in action
Supervisor Review & Audit Trail
After operators correct AIVA and give feedback, a supervisor performs a second-round verification. The audit trail of every transaction shows how often AIVA was correct and how often an operator had to give manual feedback — serving both the regulated workflow and the training loop that makes AIVA better over time.
1 / 8Supervisor review & audit trail
Outcome
2operator surfaces redesigned — Classify Documents and Data Extract
1audit trail for compliance and training — operator corrections feed back into the model
2019the year this trust layer shipped
The redesign reframed AI from a tool the operator used to a partner the operator verified. The result is a UI vocabulary I’ve reused in every AI product since — Clinical Study Master’s source attribution, KAM Field Advisor’s structured responses, the Trinity Design System’s AI module. Different surfaces, same trust layer.
