The Config File That Wouldn't Stay Fixed
I’ve got a homelab service that’s been rock solid for months: a Cloudflare Tunnel connecting about thirty internal services to public subdomains, so I can reach everything from anywhere without opening a single port on my router. It’s the kind of infrastructure you stop thinking about, until it breaks twice in one week, for two completely different reasons, and both times the actual fault was hiding in a place I wasn’t looking.
The first outage: a duplicate connector
Everything started returning 404s. Not “service down” errors, 404s, the kind Cloudflare returns when a hostname simply isn’t recognised. My first instinct was DNS. I went into the Cloudflare dashboard, found the CNAME records for the affected subdomains, and started second-guessing them. That was the wrong move, and it cost me time I didn’t need to spend.
The real problem was on my own infrastructure. I’d been running a second, independent tunnel connector on a VPS as a resilience measure, the idea being that if my home connector dropped, the VPS one would keep things alive. Except it was configured with only two of the thirty-odd routing rules the home connector had. Cloudflare doesn’t care which connector answers a request; it round-robins between whichever ones are alive. With two connectors sharing the load, that meant roughly half of all requests were landing on the connector that only knew about two services, and getting a 404 for everything else.
The fix was straightforward once I found it: stop and remove the redundant connector, accept that the home connector is the single source of truth. But the real lesson was in what I’d nearly broken along the way. CNAME records for a proxied Cloudflare Tunnel hostname are correct by construction: they just point at the tunnel, and the tunnel’s own config decides where traffic actually goes. Editing them is treating a symptom as a cause. I wrote that down as a rule for myself: for a locally-managed tunnel, the config file is authoritative, full stop. Don’t go near DNS.
The second outage: a config file with two homes
Four days later, a completely different failure. One specific hostname, the one that lets an AI assistant reach my homelab’s automation server, started resolving to the wrong destination entirely. Everything else was fine.
Tracing it back, a routine diagnostic script had gone looking for the tunnel’s config file, found an old backup, over a month stale, sitting in a default location, and silently restored it, overwriting the current, correct config in the process. The live config, the one actually being used by the running service, lived somewhere else entirely, and nothing about the script’s behaviour would have told you that.
That’s the pattern that got me twice in one week, in different disguises: there were two files with a plausible claim to being “the” config, and something (me, or a script acting on my behalf) picked the wrong one.
What actually changed
The fix wasn’t just restoring the correct file. It was making the ambiguity impossible to fall into again:
- Any diagnostic or automation script that touches tunnel config now has to reference the exact live path, not “wherever a config file happens to be.”
- Any restart of the tunnel service now gets an explicit
ingress validatepass against that exact path before the restart, not after. - Config changes get backed up with a timestamp before editing, so a rollback is one command instead of a reconstruction.
Small process, and it’s paid for itself already. Less than two weeks later, while validating a routing change for an unrelated project, I ran the standard cloudflared tunnel ingress validate command with no path specified, and it happily validated against a stale backup file from over a month prior, not the live config. That’s not a one-off fluke; it’s the tool’s actual default behaviour: point it at nothing in particular and it’ll find a config file to validate against, not necessarily the one your service is actually running on. The only reason it didn’t cause a third outage is that the new rule caught it: I’d made a habit of passing --config with the explicit live path every time, precisely because “the tool’s default” and “the file that matters” had already proven, twice, not to be the same thing.
The infrastructure lesson generalises further than Cloudflare Tunnels. Anywhere a system has more than one place a config could live, a default location and a real one, a backup and a live copy, that ambiguity is a bug waiting for the right (wrong) sequence of events to trigger it. The fix isn’t vigilance. Vigilance runs out. The fix is removing the ambiguity so there’s only one file it’s possible to mean.