Log pipeline and alerting rebuild for a hosting provider

The provider had logs on every machine and almost nothing centralised, with retention set per host by whoever built it. After an intrusion on a customer-facing service, the investigation took four days and ended inconclusive because the authentication logs for the relevant window had rotated out on two of the three hosts involved. The failure was not detection, it was that there was nothing to look at afterwards. Management asked for a monitoring product; what was needed was collection, retention and a small number of alerts anyone would act on.

Infrastructure and security lead on this, working with two platform engineers. I set the scope, ran the design, and did the collection and alerting work myself.

I owned the collection layer across roughly four hundred hosts, the retention policy including the split between hot and cold storage, and the alert rules. The rules are the part I care about — I wrote about thirty and deleted eighteen of them within the first two months because nobody acted on them. I also owned the access model for the logs themselves, which matters because centralising logs means building a very attractive target, and I did the tabletop exercise we used to test whether the pipeline actually answered investigation questions.

I chose to centralise collection but keep parsing minimal at ingest, storing raw lines alongside a small set of extracted fields. The alternative — parsing everything into structured events up front — is nicer to query and means that on the day you need a field nobody thought to extract, it is gone. I set retention by log category rather than uniformly: authentication and audit logs are kept far longer than application chatter, which cut the storage bill by about two thirds versus a flat policy and was the only way the long retention got approved. I also deliberately kept alerting small and tied every rule to a written response step; a rule with no response is a rule that trains people to ignore the channel.

Storage budget was fixed and was the binding constraint on retention, which is what drove the per-category split. About sixty legacy hosts could not run the collection agent at all, so those ship over syslog with a documented gap in fidelity. Customer data protection rules meant certain log fields had to be redacted at the source rather than in the pipeline, which was slower to implement and is the only correct place to do it.

Median time to reconstruct an authentication event went from days to about twenty minutes, measured against the tabletop scenarios we wrote after the incident. Authentication and audit retention went from a per-host guess averaging under two weeks to a guaranteed thirteen months. Of the twelve alert rules that survived, all twelve have fired for something real at least once, and the on-call team stopped muting the channel — which they had, quietly, before I started.

Two platform engineers rolled out the collection agents and handled the legacy hosts. Our on-call lead reviewed every alert rule and was the one who told me plainly that eighteen of them were noise. A customer's auditor reviewed the retention design and asked the questions that made me split by category.