Cluster consolidation at a media streaming company

Over about four years each product team had stood up its own cluster, which was reasonable at the time and had become nine clusters with nine sets of versions, nine ingress arrangements, and nine ways to get logs. The cost was partly money — a lot of idle capacity reserved per team — and mostly people: patching took a week of somebody's month, upgrades happened at nine different times, and an incident meant working out whose cluster it was before you could start. A security review that required a base image change across everything made the situation concrete, because doing it nine ways took a month.

Platform engineer, one of four, over roughly nine months. I was not the lead but I owned the migration mechanics and ran most of the individual workload moves, which meant working with about a dozen product teams whose priorities were not this. My manager handled the political side, which I was glad of. There was a fortnightly forum where team leads could object to their migration window, and I learned to arrive with a rehearsed rollback rather than reassurance.

I owned the shared cluster configuration — namespaces, quotas, network policy between tenants, and the ingress arrangement that let a workload keep its existing address while moving underneath. I built the migration runbook and then automated most of it, so that moving a workload became a pipeline someone could run rather than a day of my attention. I owned the traffic-shifting mechanism used to move a service gradually and to send it back within a minute, which we used in anger four times. I did not own the observability rollout that happened alongside, though I made sure the two did not collide.

I argued for consolidating into three clusters rather than one, split by risk rather than by team — production, non-production, and one for the workloads with regulatory constraints. One cluster is cheaper and simpler and was the initial plan; three meant we never had to have the conversation about whether a test workload could take down a production one, and it made the blast radius arguments easy in a room full of people who were nervous. I also insisted that no team be asked to change their manifests as a condition of moving, and instead absorbed the differences into the shared configuration. That made my work considerably worse and is the only reason the migration finished.

The company streams live events, so there were fixed blackout periods around anything scheduled, and one quarter was effectively closed. Two workloads could not move at all for licensing reasons tied to their hosts and stayed on a tenth cluster which is still there, documented. Everything had to be done without a maintenance window customers would notice, so every move was a traffic shift rather than a stop-start. And I was doing this alongside on-call, which meant the schedule assumed I would lose about one day a fortnight, which it correctly did.

Nine clusters became three plus one exception, and reserved capacity dropped enough to cut the monthly infrastructure bill by roughly a third — the number that got quoted upward, though the honest saving is smaller once you count the migration effort. Patching went from a week a month to about a day. The base image change that started all this now takes one pipeline run. Two migrations went wrong: one service lost its scheduled jobs because they lived outside the manifests we were moving, and one had a hard dependency on a node-local path nobody had documented, which we found by breaking it.

Our platform lead did the persuading that made the windows possible and took the meetings I would have handled badly. A colleague built the tenancy policy set and its test suite. Several product team engineers walked me through their workloads before their move, and the ones who did are the ones whose migrations were boring, which I have since used as an argument for making that walkthrough mandatory.