Audit trail service for a payments product

When something went wrong with a payment, answering "who changed this and when" meant asking three teams and reading three log formats, and the answer often did not exist because the interesting action had only ever been logged as a side effect. That is uncomfortable during an incident and unacceptable during a dispute, where the company has to show what happened. There was also a compliance requirement that certain administrative actions be recorded in a way that could not be quietly edited, which nothing then in place satisfied. The ask was one place, one format, and a credible claim that entries had not been altered.

Senior backend engineer and technical lead for the service, about eight months, with two other engineers joining for the second half. I owned the design, the migration plan for existing event producers, and the conversations with the four teams whose code had to change — which is where most of the difficulty was, because I was asking busy teams to do work with no visible benefit to their own roadmap. I reported to an engineering manager and presented the design to an architecture review board twice.

I owned the service itself: the event schema, the write path, the hash chaining that makes tampering detectable, and the retention and query sides. I wrote the client library that the producing teams integrate, and I deliberately made it small and boring because I wanted adoption more than I wanted features. I ran the migration of the six existing log sources, which involved reading a lot of other people's code to find the actions that mattered and were not being recorded at all — about a third of what we ended up capturing had never been logged anywhere.

I chose a per-stream hash chain with periodic anchoring over signing every entry individually. Individual signatures are simpler to explain to an auditor and much slower at our write rate; the chain gives the same practical property, which is that you cannot alter an old entry without invalidating everything after it. The compliance reviewer initially wanted signatures, and what changed her mind was a demonstration of detecting a tampered entry rather than an argument about cryptography. I also decided the service would reject a malformed event rather than accept and normalise it, which caused friction with two teams during integration and is the reason the data is queryable at all.

This is a payments product, so the write path could not add meaningful latency to a customer-facing transaction and could not fail one — which pushed us to asynchronous writes with an at-least-once guarantee and a reconciliation job, rather than the synchronous design I would have preferred. Retention is set by regulation, not by us, and is long enough that storage cost was a real design input. We also had a change freeze covering a peak trading period, which meant the last two producing teams integrated in the following quarter and the trail was incomplete until then, a fact we documented rather than glossed.

Every administrative and money-moving action in the product now lands in one trail; the last count I saw was around forty million entries a month. Answering a dispute question went from a cross-team request measured in days to a query, which the operations team does themselves. During one incident the trail showed a configuration change nobody remembered making, which shortened the investigation from hours to about twenty minutes and was the moment the service stopped needing to be justified internally. The tamper-detection has never fired in anger, which is the correct outcome and an awkward thing to put on a slide.

The compliance reviewer pushed hard on the tamper-evidence design and made it better by refusing to accept my first two explanations. The two engineers who joined built the query side and the retention tooling. One of the producing teams' leads did the unglamorous work of finding every unlogged action in their service, which took him a fortnight and is the reason the trail is actually complete for that domain.