Cutting the cost of an over-provisioned analytics environment

The analytics environment had grown for three years without anyone owning its bill, and finance had escalated it because it had crossed a threshold that required a director's sign-off every month. Nobody could explain the number. The environment held scheduled jobs, several long-running services, and a lot of storage, and the teams using it had each added things without removing anything, because removing something you did not create is a risk with no reward. My job was to make the number explainable first and smaller second, in that order, which was the useful framing my manager gave me on day one.

Cloud engineer, working mostly alone for about four months with my manager reviewing weekly and a data engineering lead as my main counterpart. There was no team and no sprint; it was a piece of work with a target and a lot of conversations. I also had to present findings to a finance business partner monthly, which forced me to learn to talk about this in money rather than in resources, and I think that made me more effective than any technical thing I did.

I owned the whole exercise: tagging and attribution so every resource had an owner, the analysis of what was actually being used, and the removals. The attribution was two thirds of the work — about forty per cent of spend was on resources with no tag and no obvious owner, and finding out what each was involved reading job definitions, checking access logs, and in eleven cases turning something off in a controlled way and waiting to see who complained. I also built the recurring report that keeps the attribution honest, which is the only part of this project that will still be delivering value in two years.

I decided to fix attribution before touching anything, which felt slow and was resisted — there were obvious oversized machines that could have been shrunk in week one for a visible win. But cost work without attribution turns into an argument, and I wanted every subsequent removal to have a named owner who had agreed to it. The second decision was to right-size scheduled workloads by moving them to capacity that starts and stops, rather than by picking smaller always-on machines: most of the waste was time, not size. I rejected reserved-capacity commitments as a first move, because committing to a footprint you are about to halve is how you lock in the problem.

I could not degrade the overnight reporting that the business reads at eight in the morning, so any change to those jobs had to be proven on a parallel run before it replaced anything. Data retention on several stores is set by policy and was not mine to question, which took a large chunk of storage cost off the table permanently. And there was a rule from my manager that I liked: nothing gets deleted without a two-week period where it is stopped but recoverable, which cost us some savings and meant that the two things I got wrong were undone in an afternoon.

Monthly spend on the environment came down by about forty-seven per cent over four months and has stayed within ten per cent of that since, which matters more than the initial drop. Every resource now has an owner and there is a monthly report that names anything untagged, which has kept new sprawl small. Two of my changes were wrong: I stopped a job that ran quarterly and looked idle, and I shrank a machine that was fine for eleven months of the year and not in December. Both were recoverable because of the stop-before-delete rule, which is the lesson I took from the whole thing.

The data engineering lead knew which jobs mattered and vouched for or vetoed most of my proposed removals, which made the work possible at speed. The finance business partner taught me to present this as a monthly run rate rather than a saving, which changed how the work was received internally. A colleague on the platform team reviewed the stop-and-wait procedure before I used it on anything shared.