418 reloads a day: the bug one customer found twice

Every incident writeup worth reading has a number at the center of it. Ours is 418 -- the count of reverse-proxy configuration reloads on one app host in one day, sitting in the host's journal the whole time, found on the second investigation because the first one stopped at a plausible answer. A customer found this bug. When we told him it was fixed and it was not, he found it again.

What follows is the honest version, in order: the report, the diagnosis we got wrong, the number, the mechanism, the fix, and what we changed about how we investigate.

The report

A customer running Uptime Kuma on our platform wrote in with a clear, specific complaint. His dashboard kept disconnecting. The live view would drop, the page would reconnect a moment later, and any unsaved edits to a monitor were gone. It happened often enough that configuring monitors had become a race against the next drop. He suspected our reverse proxy, and he said so in the ticket.

He was right on the first try. It took us two.

The diagnosis we got wrong

The timing lined up with something we already knew about: we had been running heavy internal testing on his host that day. The disconnects he described sat inside the same window. Two facts that fit together made a story, and the story became a diagnosis -- load. Test workloads were competing with customer apps, so we moved them off his host, tightened the rules about where internal testing is allowed to run, and told him it was fixed.

None of those actions were wrong. Test workloads should not have been sharing a host with paying customers, and the rules needed tightening whether or not they had anything to do with his ticket. But that is exactly what made the diagnosis dangerous: it let doing something responsible stand in for finding the cause. The first plausible explanation that fits the timing is a hypothesis. We skipped the step where you prove it, because the hypothesis arrived with a fix we could ship the same afternoon, and shipping a fix feels like progress.

The second report

The next day he wrote back. Still happening. Same disconnects, same lost edits, and this time no internal testing anywhere near his host, because we had just moved it all off.

It is worth pausing on what that reply costs the person sending it. Most people who report a bug, get told it is fixed, and then hit it again do not write a second ticket. They conclude the platform is flaky, quietly lower their expectations, and eventually leave. A second report with the same precision as the first is a gift, and the correct response to a gift is to take it seriously.

The number

The second investigation started where the first one should have: on the host, in the journal, with no hypothesis to defend. We counted how many times the reverse proxy had reloaded its configuration that day.

418.

Four hundred and eighteen configuration reloads in a single day, on a host where the routing had barely changed. Roughly one every three and a half minutes, around the clock, including hours when no deploy, no domain change, and no customer action of any kind had occurred. Nothing about the load theory survives contact with that number.

The mechanism

Two components, each defensible on its own. The first is reconciliation: the control panel pushes the complete routing map to every app host every five minutes. That is deliberate self-healing -- a host that was unreachable during an update, or that drifted for any reason, converges back to the correct state within five minutes with no human involved. We would build that loop again tomorrow.

The second is the host agent that receives the push. Its job is to make the proxy configuration on disk match what the panel sent. It did that by writing the file and reloading Caddy on every push -- even when the configuration that arrived was byte-for-byte identical to what was already on disk. Every five-minute heartbeat became a reload, whether anything had changed or not.

Each reload is graceful in the formal sense: in-flight requests are allowed to complete, new requests pick up the new configuration, nothing is dropped mid-response. For an ordinary request/response app, a graceful reload is invisible -- a page fetch that finishes in two hundred milliseconds will practically never straddle one, and the next fetch simply lands on the new config. But graceful to in-flight requests is not graceful to a connection that is designed to stay open. A websocket that has been alive for forty minutes is precisely the long-lived thing a reload finishes off, and each of those hundreds of reloads severed every long-lived connection on the host.

Uptime Kuma holds one websocket open for the entire editing session; the live dashboard and your unsaved edits ride on it. Chat apps and live-updating UIs are in the same class. That is why exactly one customer saw the bug and nobody else on the host noticed anything: the churn was harmless to every app except the kind that keeps a connection open, and he was running the one app on that host that did.

There is an irony here we did not miss. The app that caught our infrastructure misbehaving was itself a monitoring tool -- not through any of its checks, but by being the victim.

The fix

As small as the bug deserved. The agent now compares the pushed configuration against what is already on disk before acting. Identical: it does nothing. Different: it writes the file and reloads, exactly as before. The reconciliation loop is untouched -- the panel still pushes the full map every five minutes, and the self-healing property still holds -- but the push is now idempotent at the last step instead of only in intent.

Reloads on that host went from hundreds a day to only when routing genuinely changes: an app deployed, a domain attached, a route removed. We then checked the same measurement across every host in the fleet and verified zero unnecessary reloads anywhere. Long-lived connections now last as long as the apps holding them want them to, which is the whole point of holding one.

What we took from it

• A timing correlation is a hypothesis, not a diagnosis. The internal-testing story fit the window, implied a fix we controlled, and confirmed a rule we already believed in. All three properties made it attractive; none of them made it true. The journal held the real answer the entire time, and reading it cost less than the wrong fix did. Prove the mechanism before writing "fixed" in a ticket.
• Long-lived connections are the canary for periodic churn. Any recurring infrastructure activity that is "harmless" -- graceful reloads, rolling restarts, credential rotations -- earns that adjective against request/response traffic, because that is what gets tested. The claim needs to be tested against a connection that stays open for an hour, not against curl. If your platform holds websockets anywhere, they will report this class of bug first, usually via a confused customer.
• A customer who reports the same bug twice is worth more than a monitoring stack. Nothing we ran flagged 418 reloads, because every individual reload was normal, logged, and successful. The signal came from a person persistent enough to write "still happening" after being told it was fixed. When someone hands you that, the minimum you owe them is a real investigation the second time, and a thank you. He got both.

The dashboard stays connected now. The number worth remembering is not 418 -- it is two, the number of times one customer had to report the same bug before we found it. The goal for the next one is one.