Document classification and routing for a document-heavy back office

The client received several thousand documents a day by post, email and upload — about forty distinct types, many of them near-identical forms differing in one heading — and a team sorted them by hand into processing queues. Sorting took roughly two and a half full-time people, and a misrouted document typically surfaced three days later at the wrong desk, restarting its clock. The work is dull and the error rate got worse in the afternoon, which is exactly the shape of problem worth automating, provided the automation knows when it is guessing.

Machine learning engineer, the only one on the project, working with a backend engineer and the operations team lead. I owned the model, the evaluation, and the routing service, and I ran the shadow period.

I owned everything from the scanned image to the routing decision: preprocessing and deskewing, text extraction, the classifier, and the confidence calibration that decides whether a document is routed automatically or sent to a human. Calibration was the part I spent the longest on, because a classifier that is ninety-four percent accurate and has no idea which six percent it is wrong about is not deployable here. I also built the retraining pipeline and the drift monitoring, and I wrote the evaluation protocol that operations sign off against each quarter.

I set an explicit abstain threshold and tuned it so that automatic routing runs at a much higher precision than the model's headline accuracy, sending everything below the threshold to the manual queue. The client initially wanted every document routed, and the argument that won was showing them the cost of a misroute — three days — against the cost of a manual sort, which is about forty seconds. I also chose a straightforward text-based classifier over a document-image model, because ninety percent of the signal is in the words and the image approach was far harder to explain to an auditor and to retrain. For the two document types that genuinely look alike and differ by layout, I added a small set of layout features rather than switching the whole approach.

Documents contain personal data, so nothing left the client's environment — training and inference both run on their own machines, which ruled out several hosted options and set the model size ceiling. Historical labels came from the manual sorting process and carried its errors, so part of the early work was cleaning about two thousand mislabelled examples before any number meant anything. Operations could not absorb a change in their tooling, so the service had to write into the existing queue system exactly as a human sorter does.

About seventy-eight percent of documents route automatically, at a measured precision of 99.4 percent on that subset over the last two quarters. The remaining twenty-two percent go to a manual queue that is now roughly half a person's work instead of two and a half. Misroutes fell from an internal estimate of around three percent to under half a percent overall, counting the manual queue. Drift monitoring has flagged two real shifts, both after the client's own forms changed, and retraining took about a day each time.

The operations team lead relabelled the historical set with me and defined what a costly misroute actually is, which set the threshold. A backend engineer built the queue integration and the service scaffolding. The client's compliance officer reviewed the evaluation protocol and required the quarterly re-sign-off, which has been useful rather than a burden.