Streaming Engagement Pipeline

The editorial team was making decisions about what to promote on the homepage using numbers that were a day old, which for a news operation is roughly useless — by the time a story shows as trending, it isn't. The ad team had a different but related problem, needing delivery figures close enough to real time to pace campaigns. Both were being served by the same nightly batch, and both had built their own workarounds involving a third-party analytics tool with different numbers. They wanted one pipeline serving both within a couple of minutes of an event.

I was the senior data engineer and effectively the technical owner, working with one other engineer and a platform engineer who managed the cluster. I made the architecture decisions and did most of the implementation on the streaming side. I also spent an unglamorous amount of time on the event schema negotiation with the web and mobile teams, which determined whether any of it would work.

I owned the streaming topology — ingestion from the client apps, sessionisation, the windowed aggregations that produce engagement metrics, and the sinks feeding both the editorial dashboard and the ad system. I built the schema registry and the compatibility rules that stop a client release breaking the pipeline. I owned the lakehouse tables where everything lands for later batch analysis, and the reprocessing path that rebuilds an aggregate from raw events when we get the logic wrong. I did not own the client-side instrumentation or either consuming application.

I chose to keep one pipeline serving both consumers rather than the two the teams initially asked for. Two pipelines meant two sets of numbers and we already had that problem. The cost was a harder negotiation about event schemas, which took about six weeks and was worth it. I made all aggregations recomputable from the raw event log stored in Iceberg tables, which meant accepting the storage cost and gaining the ability to fix a metric definition retroactively — we've done that four times. I also set late-arriving event handling to a two-hour watermark with explicit late-event correction, rather than dropping them; mobile clients queue events offline and dropping them would have systematically undercounted commuters.

Mobile clients release on a slow cycle, so any event schema change has to be backward-compatible for months and old app versions are still sending the original format years later. Traffic is extremely spiky around breaking news, up by a factor of twenty within minutes, so the pipeline had to be sized for peaks nobody can schedule. There were also strict rules about retaining anything that could identify a reader, which shaped sessionisation more than any technical consideration did.

Editorial went from day-old numbers to a dashboard around ninety seconds behind live, and they now change homepage placement several times a day based on it. Ad pacing moved onto the same figures, which ended a long-running argument between the two teams about whose numbers were right — the ad team's monthly reconciliation against the third-party tool went from a recurring dispute to a formality. During a major news event in the first year the pipeline held through a traffic spike about eighteen times baseline without intervention, which was the first real proof it worked.

The other data engineer built the sinks and most of the operational tooling, and was the one who insisted on the reprocessing path when I was ready to skip it. The platform engineer handled cluster sizing and autoscaling, which I would have got wrong. A senior editor sat with us for two afternoons and cut the dashboard metrics from about twenty to six, which made it something people actually use.