What a CDN taught me about AI agents

2016

Two log lines and a subtraction

2016 felt like the year the internet's biggest audiences arrived over streaming all at once. March Madness, the Olympics, a general election: crowds that had always belonged to broadcast were suddenly pouring through our edge nodes, live, with no tolerance for buffering. I was working the CDN, and during one live March Madness game we needed to know, right then, whether users were feeling latency.

The answer was embarrassingly small. Two log lines on an ingest server. Subtract their unix epoch timestamps, and the delta tracked what users were feeling: a pulse, computed by hand. Around it we ran a polling script that walked the CDN nodes round-robin for live health checks. Nothing in the tooling of the day was built to answer that question at game speed. Plain bash, a subtraction, and a loop.

The Olympics brought different trouble: volumes past anything manual analysis could scale to. There we graduated to SQL that carved windows of observability into massive logs, queries that isolated issues no one would have found by looking.

Streaming kept climbing after that, and the tooling we built to keep up grew into a discipline of its own: system design, triage, volume handling, uptime. But the seed of it all was my first real vision of scale — millions of people watching one game through fleets of machines we polled one at a time, the health of that entire experience compressing into a number small enough to hold in your head. A system's pulse can be two timestamps and the will to subtract them.

2020

The launch I couldn't stand next to

Four years later the pandemic took away the room. I had moved from hands-on-keyboard into management that stayed close to the keyboard, and my company was preparing to deliver a major global console launch, planned and executed over online-only communication, end to end.

Launches like that used to be conducted in person. A war room, a shoulder tap, the read you get on a problem from the way someone stands up. All of it was gone at once, which put the full weight of the story on telemetry. If the graphs didn't carry it, there was no hallway where it could be recovered. The exposure had changed too: global traffic instead of one event's edge nodes, resiliency as a planning discipline instead of a night-of scramble, the same instruments now speaking to planners and decision-makers, not just to the engineer who wrote the polling script.

The launch held. From the CDN side I watched the first run-throughs appear online within hours of the downloads completing, on every continent at once. No dashboard showed that part, and none showed the other thing I took from it: what a team pulling in one direction can actually move.

More and more of what used to be observed in person would now have to be observed through instruments, and the flood was only starting. When nobody can stand in the room, telemetry becomes the room.

The years after pulled me into security, where the instruments meet adversaries: bots at global scale, attack vectors that follow geopolitics as much as technology, practices that have to evolve or lose. Every era so far had asked is it alive? This one added: and is it lying to you?

2026

The day we shipped agentliveness

When AI arrived I met it with trepidation, then outright rejection. What finally broke the rejection wasn't a demo. It was the grudging admission of the oldest cliché in this business: change is the only constant.

So I started small, deliberately: first principles before frameworks, deriving the new field from what a decade of operating systems at scale had already taught me.

The old question surfaced immediately, wearing new clothes. An agent can hold a green checkmark and still be gone. And here the second question catches up with the first: a fresh timestamp sitting on top of a dead producer isn't just a stale signal — it's a signal that lies.

So the subtraction grew up. Running my own fleet of scheduled agents taught me, failure by failure, what the new era actually demands. A counter that silently reset to zero every run taught me that cross-run state must survive process death by construction. A file written by a newer schema, parsing as plausible garbage, taught me that version drift has to fail loud. A fresh timestamp over an analysis that never advanced taught me — again — that freshness is not liveness. A degraded verdict written to a log nobody read taught me that detection without a response channel is an unseen incident. And a notifier that crashed the very engine it was guarding taught me that the alarm must never take down the run it protects. Five failures, three verbs: persist, detect, respond.

Every one of those fixes was built the new way — with assisted coding tools, an agent at my elbow helping me build the mechanics that watch agents. That recursion is the learning arc: I'm not studying agentic engineering from the outside, I'm acquiring it the same way I acquired the CDN — by operating the thing, breaking it, and turning each break into a primitive.

One morning this June I distilled those lessons into a small reliability harness, scrubbed its private history, and pushed it public. It had lived for weeks under a different name. The morning it went live I renamed it, because the old name didn't state the lesson: agentliveness. At its core it is the 2016 move with a decade of scar tissue attached: don't just check the timestamp, prove the producer is still producing.

I'm leaving the rest of this chapter open on purpose. The path I'm walking is being laid as I walk it, and this note is the first in a series: working the new era out in public, from first principles, as it happens.