Support ticket routing for a telecoms operator

Tickets arrived from several channels and were routed by keyword rules that had been added to for years, so a ticket mentioning a common word could land in a queue that had nothing to do with it. Misrouted tickets bounce between teams, and each bounce adds most of a day, which shows up directly in the resolution times the operator reports on. Roughly a fifth of tickets were being reassigned at least once. The business wanted fewer bounces; the support managers, who I only understood later, wanted to stop being blamed for other teams' queues.

Machine learning engineer, the junior of two on the model side, with a data engineer and a backend engineer on the same squad, over about six months. I did the data preparation and evaluation, and roughly half the modelling, under a senior colleague who reviewed everything and let me make my own mistakes first. I also ran the fortnightly session with the support team leads where we reviewed misroutes, which I was thrown into and which turned out to be the most valuable part of the job.

I owned the evaluation set and the labelling process, which is the part I would now say was the whole project. Historic routing data is not ground truth, because it records where a ticket ended up rather than where it belonged, so I built a labelled set from resolved tickets with the reassignment history used to work out the correct destination, then had support leads adjudicate the ambiguous ones. I owned the feature preparation and the confidence thresholding, and I wrote the fallback that sends a low-confidence ticket to the old rules rather than guessing. The serving side belonged to the backend engineer.

The decision I argued for and won was routing only above a confidence threshold and falling through to the existing rules below it. My first instinct — and the squad's initial plan — was to replace the rules outright, which the evaluation showed would be better on average and much worse in the tail, and the tail is where the complaints come from. Setting the threshold per destination team rather than globally came out of the review sessions, because a mistake into one particular queue costs far more than a mistake into another. I also pushed back on adding the customer's account value as a feature, which was suggested and which I thought would quietly encode something we would not want to defend.

Ticket text contains customer details, so all of the work happened inside an environment with no external access and I could not use anything hosted outside it, which ruled out several approaches and made the project simpler than it would otherwise have been. Any change to routing had to be approved by all six team leads, which is why the rollout took as long as it did. We released to one channel at a time over eleven weeks with a per-team switch that a lead could throw themselves, and one team did throw it in week three and stayed off for a fortnight until we fixed a pattern of misroutes affecting them.

Reassignment rate went from about twenty-one per cent to nine over the quarter after full rollout, and average time to first correct owner dropped by roughly four hours. The fallback path still handles around fifteen per cent of tickets and I no longer think of that as a failure. The genuine lesson for me was in the review sessions: the model's aggregate accuracy barely moved after the first month, while trust moved a great deal, and the difference was entirely about giving each team a switch and showing up every fortnight with their misroutes.

My senior colleague let me build the first version badly and then asked the questions that made me rebuild the evaluation set, which was the right way to teach me. Six support team leads adjudicated about eleven hundred ambiguous labels between them over three weeks. The backend engineer built the serving path and the per-team switch that made the cautious rollout possible.