Green, healthy, and consuming nothing

The service looked perfect. After a routine host reboot it came back in seconds, the process was up, and its /health endpoint answered a clean 200 OK. Every dashboard that watched it was green.

It was also doing nothing. Of the 46 message-queue consumers that service runs — the workers that turn queued events into emails, WhatsApp messages, push, and SMS — 45 were dead, and had been dead since the instant the process started. Nothing was being consumed. Notifications were silently piling up in the broker, unsent, behind a health check that swore everything was fine.

The gap between "the process is up" and "the process is doing its job" is the whole story. It's a gap a naïve health check is structurally blind to — and it opened here through a race nobody had staged for.

The race: the service booted faster than the broker

Here's what actually happened, and it's a boot-order race with a very specific trigger.

Each of those 46 consumers dials the message broker itself at startup, independently. And the failure handling was the fatal part: if the dial fails, the consumer logs a warning, disables itself, and never tries again. That's a perfectly reasonable-looking line of code — until the broker isn't there at the one moment every consumer reaches for it.

On this reboot, the broker wasn't there. The host came up, the container runtime brought everything back with restart: always, and the notification service won the race — it was accepting traffic while the broker was still initializing. The broker's AMQP listener didn't open until 38 seconds after boot. By then it was far too late: all 46 consumers had already fired their single dial into a port that wasn't listening, taken ECONNREFUSED, and switched themselves off for the life of the process.

45 of 46. One consumer happened to dial a fraction later, into the window after the listener opened, and lived — which is the tell that this was a timing race, not a config error. Change the boot timing by a few seconds and it's 46 of 46, or 12 of 46. Non-determinism like that is the fingerprint of a startup-ordering bug.

"But we have depends_on"

The reflex objection is the right one: isn't this exactly what container startup ordering is for? We had a health-gated dependency declared — the notification service was supposed to wait for the broker to be healthy before starting.

The subtlety that bites you: startup-ordering conditions only govern the first up. When the container runtime brings containers back on its own — a host reboot with restart: always — it restarts them all at once and ignores those conditions. The ordering you carefully declared is honored exactly when you're watching (a manual deploy) and quietly skipped exactly when you aren't (a 3 a.m. host reboot). So the protection you think you have evaporates in the one scenario — unattended recovery — where you actually need it.

That's the trap worth internalizing: dependency ordering at first-boot is not the same as resilience to the dependency being slow. They look identical on a good day.

The health check that couldn't fail on the real failure

Why did this run for as long as it did before anyone noticed? Because the signal that should have screamed was defined so it could only ever read green.

The /health endpoint reported on the thing that was trivially true — the HTTP process is running — and said nothing about the thing that had actually failed: are the consumers connected and consuming? A liveness probe that checks "am I up" will always pass in this failure, because the process is up. It's just idle. It's the same shape as a delivery metric stamped at the wrong moment, or a dashboard that graphs disk and CPU while the real outage lives one layer beneath: a green number that is structurally incapable of representing the bad case.

A health check earns its keep only if there is a real failure it can turn red. If you can't point at the value it would report when the job is broken, it isn't monitoring the job — it's monitoring the wrong thing and calling it health.

The fix: gate startup on the broker actually answering

The consumers failed because they assumed the broker would be there the instant they reached for it. So the fix is to stop assuming and wait for it — gate consumer startup on the broker actually answering, with a bounded, backing-off probe, before a single consumer dials:

async function waitForBroker(url) {
  const deadline = Date.now() + WAIT_TIMEOUT_MS;   // ~2 minutes, not forever
  let attempt = 0;
  while (Date.now() < deadline) {
    attempt++;
    try {
      await probeBroker(url);                       // one bounded connect + close
      return true;                                  // broker is listening — go
    } catch {
      const delay = Math.min(5000, 250 * 2 ** (attempt - 1));  // capped backoff
      await sleep(delay);
    }
  }
  return false;   // fall through to the old degraded behaviour, don't crash-loop
}

A 38-second-late broker is now a non-event: the service probes, backs off, probes again, and starts its consumers the moment the broker answers — usually within a second or two of it coming up.

Two details in that probe earned their keep, and both are the kind of thing you only find by reading the client's fine print:

• The client library sets no handshake timeout of its own. A broker that accepts the TCP connection but never completes the AMQP handshake — a half-open port during startup — leaves connect() pending forever, which would hang startup straight past the deadline. So each probe is wrapped in its own timeout and raced against it, guaranteeing every attempt either succeeds or fails within a bounded window.
• An abandoned probe can still connect afterwards. When a probe times out, the underlying connection attempt may still succeed a moment later — and if you've walked away, that live connection (and its heartbeat timer) leaks for the life of the process. So a timed-out probe registers a callback to close the socket if it ever completes. A guard that leaks the thing it was guarding isn't a fix.

Honest about the altitude

This gate kills the trigger — a broker that's slow to accept connections at startup. I want to be clear that it does not, by itself, kill the class.

The class is "a consumer whose life is tied to a broker connection it can't survive losing." The startup probe gets all 46 consumers connected on boot, but the failure handling that started this — dial once, and disable on loss — is still there for a connection that drops mid-life. A broker restart an hour later would still take the fleet down the same way. The real fix for the class is per-consumer reconnect with backoff (a supervised connection that re-establishes itself and re-consumes), plus a health check that actually reports consumer connectivity so the next occurrence turns something red instead of staying green. The startup gate is the seatbelt this specific crash earned; the reconnect work is the airbag the class still needs.

And it was caught on a demo environment, not in production — a reboot there surfaced the race before a customer ever felt it. That's the good version of this story. The bad version is the same bug, the same green health check, discovered from a customer asking why they never got their email.

Takeaways

• "The process is up" is not "the process is working." A liveness check that can't distinguish a busy service from an idle one will pass straight through a total outage. Health-check the job — are the consumers connected and consuming — not just the pulse.
• Dependency ordering at first-boot is not resilience to a slow dependency. Health-gated startup conditions govern a manual up and are ignored when the runtime restarts everything itself on a host reboot. Assume your dependency can be late, and wait for it in code.
• A dial-once-and-give-up client is a landmine. If a consumer disables itself on the first connection failure and never retries, a dependency that's a few seconds late costs you that worker until the next deploy. Gate startup on the dependency answering; reconnect on loss.
• Non-deterministic breakage is a race signature. 45 of 46, not 46 of 46, is the tell. When the count of what breaks changes run to run, stop looking for a bad config and start looking at startup timing.
• Read the client's fine print before you trust its defaults. No handshake timeout meant a half-open broker could hang startup forever; an abandoned connect could still succeed and leak. The robustness lived in the two edge cases, not the happy path.
• Fix the trigger, then name the class. A startup gate ends this recurrence; a consumer that can't survive losing its broker is the larger bug. Ship the seatbelt, but write down the airbag.