Payments Ledger Service

The client had grown a payments product on top of a balances table that was updated in place, so there was no way to answer why a merchant's balance was what it was. Support spent hours per dispute rebuilding history from logs, and the finance team's monthly reconciliation against the processor took three people about four days. Regulators were also starting to ask questions the system couldn't answer. They needed an auditable ledger that could reconstruct any balance at any point in time, and it had to be introduced without freezing an operating business.

I was the senior backend engineer on a team of four, working with a product manager and one of the client's finance leads. The ledger design and the migration were mine; the API surface and the merchant-facing screens belonged to two other engineers. I also ended up as the person who explained the model to non-engineers, which meant a lot of whiteboard time with the finance lead until we agreed on what an entry actually represented.

I wrote the ledger service — the append-only entry model, the posting rules, the idempotency layer around every write, and the balance projection that support and merchants read from. I owned the backfill that reconstructed about three years of history from the old balances table plus processor statements, which took three attempts before the numbers matched. I built the daily reconciliation job that compares our entries against the processor file and files a discrepancy record rather than silently correcting itself. I did not own the payout scheduling or anything on the merchant dashboard.

I pushed for entries to be immutable with corrections modelled as reversing entries, rather than allowing edits with an audit log. Edits with an audit log are simpler and the team preferred them, but they let a bad write and a bad audit entry happen together, and the whole point was that finance could trust the ledger without trusting us. I chose Kafka for fan-out to reporting but kept the ledger's own writes in Postgres transactions, because I didn't want correctness to depend on a broker's delivery semantics. We also made every posting API idempotent on a caller-supplied key, which added friction for the client teams and eliminated an entire class of duplicate-charge incident.

The system was live throughout, handling money, so anything I shipped had to be reversible in a single deploy. The backfill could only run against a read replica during a four-hour overnight window, which is why it took three attempts spread over three weeks rather than one long weekend. We also inherited a processor integration nobody could change and whose statement format shifted twice during the project without notice.

Reconciliation went from three people for about four days to one person for roughly half a day, mostly reviewing the discrepancies the job flags. Dispute investigations that used to take hours of log archaeology became a query, and support stopped escalating them to engineering entirely — that queue went from a handful a week to none by the third month. In the first six weeks the daily job surfaced a rounding mismatch on one payout type that had been quietly accumulating for over a year, which nobody had noticed.

The client's finance lead was effectively a co-designer; she rejected my first two models for reasons I hadn't considered and she was right both times. A colleague built the merchant-facing balance screens on top of my projections and caught several places where my API made sense to me and to nobody else. Our product manager took the regulatory conversations off my plate, which I appreciated more than I said at the time.