The dangerous failures all returned exit code 0
Every postmortem you read is about something that fell over. A process OOMs, a container restarts, an alert fires, someone gets paged. Those are the easy ones. The system told you.
The failures that actually cost us were the ones that returned exit code 0. A script ran on schedule, wrote nothing useful, reported success, and did that every night until someone went looking. There is no alert for "the job did nothing," because from the outside a job that does nothing and a job that does its work look identical: both finish, both exit clean, both leave a green line in the log.
Here are four of ours. All of them are in this repo's own documentation, which is the only reason I can write them down accurately. No product pitch — these are the bugs, and the general shape of the defence at the end is the part worth your time.
1. The backup that backed nothing up
The classic. We shipped nightly restic backups to offsite object storage,
wired the systemd timer, watched the unit go green, and moved on.
The first time we actually attempted a restore, from a throwaway office, the
drill caught two separate silent-failure bugs in backup.sh. Neither had ever
produced a failing run.
The first: repo credentials arrive in an env file, and the script sourced it.
Sourcing puts variables in the shell. restic is a child process — it reads
the environment, and a plain source leaves the values shell-local. So restic
never saw RESTIC_REPOSITORY at all.
# set -a: restic runs as a child process and needs these EXPORTED — a plain
# source leaves them shell-local and restic dies with "specify repository".
set -a
source /etc/litwindow/backup.env
set +a
The second: nothing ever ran restic init. A restic repository does not exist
until something creates it, and no part of the pipeline created it. The fix is
three words of shell, and it is the shape of fix that should make you
suspicious of everything nearby:
# First run on a fresh instance: the repo does not exist until someone inits
# it, and nothing else ever does. cat config is the cheap existence probe.
restic cat config >/dev/null 2>&1 || restic init
Note what both bugs have in common. Neither is a logic error in the backup
strategy. The strategy was fine. In both cases the script ran, and the
outcome the script existed to produce did not happen. set -euo pipefail was
already at the top of the file and did not save us, because from bash's point
of view nothing failed.
The only thing that found this was trying to restore. The drill itself is now the artefact I trust: 722 MiB snapshot restored in 11s; SQL dump gzip-valid, volume tar intact, tenant.env intact. Not "the backup ran." The bytes came back and were the right bytes.
Your backups are not backups until you have restored one. Everybody says this. Almost nobody has done it, and the two bugs above are exactly what you find on the day you finally do.
2. The cron job that had been crashing for days
We run a sweeper every five minutes. It does several unrelated things: flag instances whose agent has gone quiet, retry provisions that were parked waiting on host capacity, and enforce the billing lifecycle.
Its first pass queried for stale heartbeats — a compound query on status and last-heartbeat time. Firestore requires a composite index for that, and the index had never been created. The query threw. The exception was uncaught, so it took down the whole scheduled function, which meant the capacity retries and every bit of lifecycle enforcement after it never ran either.
This had been happening on every cycle for days. What surfaced it was the first live end-to-end lifecycle test, not any monitoring, because nothing in the system alerts on absence. There was no error to see: the scheduled function had an error rate, sure, but nobody was watching a graph for a thing that had been quietly working for weeks.
The fix is two halves, and the second half is the transferable one.
First half: create the index. Boring, necessary, and now part of review — any new compound query has to declare its index need before it merges.
Second half: stop letting one broken thing kill the things behind it. Each pass of the sweeper is now isolated, so a failure is scoped to the pass that caused it:
// Each pass is isolated: one broken query or provisioner hiccup must never
// stop the passes after it (learned live: a missing composite index in pass 1
// silently killed capacity retries and lifecycle enforcement for days).
async function pass(name: string, fn: () => Promise<void>): Promise<void> {
try {
await fn();
} catch (err) {
console.error(`sweeper pass ${name} failed:`, err);
}
}
That is not a clever pattern. It is eight lines. But it converts one class of outage — "everything downstream of the first bug stopped" — into a much smaller one, and it means the log now names the specific pass that broke instead of showing you a stack trace from whichever thing happened to be first in the file.
The honest caveat: catching and logging is still not alerting. A broken pass now fails alone rather than taking the others with it, and it says so in the log. Something still has to read that.
3. The volume mounted at the wrong path
This one is my favourite because the container is healthy the entire time.
The OpenClaw image runs as user node, whose home is /home/node. We mounted
the persistent state volume at /root. Docker cheerfully created the mount.
The container started. The health check passed. The agent worked.
And every byte of config and state the agent wrote went to /home/node/.openclaw
— inside the container's writable layer, not the volume. Which is fine, right
up until the container is recreated for any reason at all: an image update, a
compose change, a reboot. Then it comes back with a completely empty state
directory and no memory of anything, still perfectly healthy.
volumes:
# Image runs as user `node` (home /home/node); mounting at /root loses all
# state and config (learned the hard way on our own first box).
- openclaw_data:/home/node/.openclaw
There is no error mode here to catch. The mount succeeded. The write succeeded. The health check was answering an HTTP endpoint, which tells you the process is up and tells you exactly nothing about whether its state is durable. A liveness probe cannot distinguish amnesia from health, because amnesia is healthy by every measure a liveness probe has.
The general version: a health check that does not touch the thing you actually care about is a health check for something else.
4. The rebuild that broke every new machine
We rebaked the golden image to add an unrelated opt-in feature. The bake succeeded. The image published. The feature worked.
The npm inside the Paperclip image floats with its unpinned upstream node
base, and the newer npm that came along hard-errors with EACCES on the
root-owned files the container entrypoint leaves in its cache directory. That
killed the onboarding step in firstboot — for every new provision, not just ones
using the new feature. The bug had nothing to do with what we were shipping. It
would have bitten any rebake of that image at that moment.
# Newer npm hard-errors (EACCES) on the root-owned files the container
# entrypoint leaves in /paperclip/.npm. Own the cache before onboarding
# runs as node.
docker compose ... exec -T -u root paperclip \
sh -c 'mkdir -p /paperclip/.npm && chown -R 1000:1000 /paperclip/.npm' \
|| echo "WARN: npm cache chown failed"
The silent part is not the EACCES — that is loud, on the machine it happens
to. The silent part is the gap between "the build exited 0" and "a machine built
from this actually comes up." Those are different assertions, and only one of
them was being made. A green build is a statement about the builder, not about
the artefact.
That bad image was deleted from the fleet default rather than patched in place, which is the right call, but only because someone provisioned a fresh box and watched it boot.
What do these silent failures have in common?
All four are the same shape: an operation that can no-op and still exit 0.
Sourcing an env file no-ops as far as the child process is concerned. A backup to a repository that does not exist no-ops. A query that throws inside an uncaught handler no-ops the entire schedule. A write to the wrong path no-ops across a restart. A build no-ops the question you actually cared about.
None of them are exotic. That is the point. You will not find them by being smarter, because they are all trivially obvious in hindsight, and none of them are visible from the thing that's supposed to be watching.
How do you catch a failure that reports success?
Four rules, all of which we learned by getting them wrong first.
Prove the outcome, not the invocation. "The backup script ran" is not evidence. "722 MiB restored, gzip valid, tarball intact" is evidence. The question is never did the thing run, it is can I produce the result the thing exists to produce. For anything that matters, the drill is the test — you either restore, or you don't have backups.
Assert the post-condition in the script itself. The restic cat config || restic init line is exactly this: a cheap probe that the precondition holds,
inline, on every run. Anywhere your script assumes something exists, check it.
The check usually costs one line and pays for itself the first time.
Isolate each unit of work. One broken pass should degrade one pass. This is cheap to add before you need it and expensive to discover you needed.
Alert on absence-of-success, not presence-of-error. This is the big one and
the one we are still finishing. Every failure above was invisible to
error-based monitoring, because there was no error. What catches them is a
heartbeat with a deadline: the backup reports backup_ok, and something on the
other side notices when that stops arriving. Errors tell you about the failures
your system anticipated. Silence tells you about the ones it didn't.
Being honest about our own gap here: our sweeper flags a missed heartbeat and writes an event, but customer-facing dead-worker alerting is still a TODO in the code with the email transport already live and unwired. It's on the list. Writing this post is partly what put it there.
For what it's worth, this is most of what litwindow actually sells — heartbeats, restarts, budget stops and restore drills, rather than features. But none of the four bugs above required a platform to find. They required someone to ask what would prove this worked, and then go and prove it.