Is this thing alive?
One question, walked through all three moves. It looks trivial. It has quietly broken every system I've run for a decade — CDNs, engines, agents — always the same way.
What would actually prove aliveness?
Start from zero. A dashboard shows a green check and a timestamp. What does that timestamp prove? Only that the thing succeeded once, then. Every claim of "alive" built on it is an inference — and it fails in exactly three ways I've been bitten by, each one now a root-caused mechanism:
- RCA-017Fresh output over a dead producer. The file looks recent; the job that writes it was never even loaded. Authoring a schedule is not running one.
- RCA-021Green exit codes that shipped nothing. A publish job "succeeded" daily for two weeks — pushing an unchanged ref. Success signals can be no-ops.
- RCA-019The watcher died with the watched. A health check running inside the process it monitors cannot report that process's death.
So the derivation lands on a contract, not a feeling: declare an expected interval; stamp every output with generated_at; and define age > 2× interval = incident — mechanically, loudly, with the check living outside the thing it checks. Try the contract:
This page takes its own site's pulse — right now
Not a mock. Your browser is about to apply that exact contract to the three live data feeds behind this site, the same files its homepage reads. If an engine on my desk died this morning, you'll see it here before I do anything about it:
Same math, richer view: the full status page — including the fleet's own gates.
The version you can use at work tomorrow
A green checkmark tells you the last time something worked — not that it works now. The gap between those two is where systems quietly die. You don't need my tooling to defend yourself; you need three questions:
1 — When did it last actually run?
Not "is the light green" — find the timestamp. If there is no timestamp, that's your first finding.
2 — Would anyone know if it died tonight?
Who — or what — notices, and does that watcher live outside the thing it watches? A process can't report its own death.
3 — What counts as "too old"?
If nobody has written down a number, "stale" is a matter of opinion — and opinions don't page anyone. Pick an interval; alarm at twice it.
That's the whole method, once around: reason to a contract instead of a feeling, demonstrate it against something real, demystify it so anyone can carry it home.