Credential rotation across a platform team's services

The services had accumulated static credentials in environment configuration, some of them years old, some shared between services, and at least four that appeared in more than one place including a wiki page. Nobody could answer what would need to change if a credential leaked, and the honest answer was a weekend and a lot of guessing. An internal review flagged it, and separately a departing contractor prompted a rotation that took two people three days and broke two services, which made the case better than any document could have.

Backend engineer on the platform team, four of us, and I picked this up as the owner because I had complained about it most. It ran about ten months alongside other work, never more than half my time. There was no product owner and no roadmap slot; I got it done by making it a standing item and doing one service per sprint, which is the only way work like this ever finishes.

I owned the approach and the per-service migration. That meant building the mechanism for short-lived credentials issued at start-up and refreshed in the background, writing the library that hides it from application code, and then doing the actual conversion service by service — including the four written before I joined, in a language the team barely uses, where the work was mostly proving I had not broken anything. I also owned the inventory: a list of every credential, what uses it, and when it was last rotated, which did not exist and which I maintained by hand for the first three months before automating it.

I decided to do it incrementally with both mechanisms live rather than as a coordinated cutover, which the team's technical lead initially resisted because it meant carrying two paths for the best part of a year. But a coordinated cutover across thirty services needs a window nobody would ever grant, and the failure mode is everything at once. I also chose to make the library fail loudly at start-up if it cannot get a credential, rather than falling back to the static one — the fallback is what turns a migration into a permanent dual system, and I have watched that happen elsewhere.

Everything had to happen without downtime on services other teams depend on, so each migration was a deploy, a soak, and a follow-up deploy to remove the old path. Four services could not be changed at all during a quarter-long feature freeze in their domain and were done afterwards. Two credentials belonged to a vendor whose rotation process is a support ticket with a five-day turnaround, so those were scheduled around. And I had to keep the whole thing low-drama, because a project about credentials attracts attention that makes people slower.

Twenty-eight of thirty services now hold no long-lived credentials; the two that remain are documented with a reason and a date. A rotation went from a three-day exercise with breakage to something that happens automatically and that we test by forcing it monthly. The inventory, which was the accidental deliverable, gets used more than the mechanism does — it is the thing people open when they are asked a question about access. One service did break during migration, for forty minutes on a Tuesday, because I had missed a background worker that read the credential once at start-up and never again.

The team's technical lead disagreed with the incremental approach and reviewed every one of the thirty changes anyway. A colleague wrote the monthly forced-rotation test after watching me do it by hand for a while and getting tired of watching. The vendor's support contact was slow but ultimately walked me through their rotation process, which is not documented anywhere.