The Boot Race That Killed 45 of 46 Consumers

Green, healthy, and consuming nothing

The service looked perfect. After a routine host reboot it came back in seconds, the process was up, and its /health endpoint answered a clean 200 OK. Every dashboard that watched it was green.

It was also doing nothing. Of the 46 message-queue consumers that service runs (the workers that turn queued events into emails, WhatsApp messages, push, and SMS) 45 were dead, and had been dead since the instant the process started. Nothing was being consumed. Notifications were silently piling up in the broker, unsent, behind a health check that swore everything was fine.

The gap between "the process is up" and "the process is doing its job" is the whole story. It's a gap a naïve health check is structurally blind to, and it opened here through a race nobody had staged for.

The race: the service booted faster than the broker

Here's what actually happened, and it's a boot-order race with a very specific trigger.

Each of those 46 consumers dials the message broker itself at startup, independently. If the dial fails, the consumer logs a warning, disables itself, and never tries again. That's a perfectly reasonable-looking line of code, until the broker isn't there at the one moment every consumer reaches for it.

On this reboot, the broker wasn't there. The host came up, the container runtime brought everything back with restart: always, and the notification service won the race: it was accepting traffic while the broker was still initializing. The broker's AMQP listener didn't open until 38 seconds after boot. By then it was far too late: all 46 consumers had already fired their single dial into a port that wasn't listening, taken ECONNREFUSED, and switched themselves off for the life of the process.

45 of 46. One consumer happened to dial a fraction later, into the window after the listener opened, and lived, which is the tell that this was a timing race, not a config error. Change the boot timing by a few seconds and it's 46 of 46, or 12 of 46. Non-determinism like that is the fingerprint of a startup-ordering bug.

"But we have depends_on"

The reflex objection is the right one: isn't this exactly what container startup ordering is for? And it was set up that way. The container config told the notification service to wait until the broker reported healthy before starting, which is the standard way to say start this only once that dependency is ready.

The subtlety that bites you: startup-ordering conditions only govern the first up. When the container runtime brings containers back on its own (a host reboot with restart: always) it restarts them all at once and ignores those conditions.

The ordering you carefully declared is honored exactly when you're watching (a manual deploy) and quietly skipped exactly when you aren't (a 3 a.m. host reboot). So the protection you think you have evaporates in the one scenario, unattended recovery, where you actually need it.

That's the thing: dependency ordering at first-boot is not the same as resilience to the dependency being slow. They look identical on a good day.

The health check that couldn't fail on the real failure

Why did this run for as long as it did before anyone noticed? Because the signal that should have screamed was defined so it could only ever read green.

The /health endpoint reported on the thing that was trivially true (the HTTP process is running) and said nothing about the thing that had actually failed: are the consumers connected and consuming? A liveness probe that checks "am I up" will always pass in this failure, because the process is up. It's just idle. It's the same shape as a delivery metric stamped at the wrong moment, or a dashboard that graphs disk and CPU while the real outage lives one layer beneath: a green number that is structurally incapable of representing the bad case.

The fix: gate startup on the broker actually answering

The consumers failed because they assumed the broker would be there the instant they reached for it. So the fix is to stop assuming and wait for it. Gate consumer startup on the broker actually answering, with a bounded, backing-off probe, before a single consumer dials:

async function waitForBroker(url) {
  const deadline = Date.now() + WAIT_TIMEOUT_MS;   // ~2 minutes, not forever
  let attempt = 0;
  while (Date.now() < deadline) {
    attempt++;
    try {
      await probeBroker(url);                       // one bounded connect + close
      return true;                                  // broker is listening, go
    } catch {
      const delay = Math.min(5000, 250 * 2 ** (attempt - 1));  // capped backoff
      await sleep(delay);
    }
  }
  return false;   // fall through to the old degraded behaviour, don't crash-loop
}

A 38-second-late broker is now a non-event: the service probes, backs off, probes again, and starts its consumers the moment the broker answers, usually within a second or two of it coming up.

That safety has a cost: the service now waits on the broker instead of racing ahead, so a genuinely dead broker delays startup by up to the timeout, about two minutes, before it falls through to the old degraded behavior rather than crash-looping.

Two things to consider in that probe:

• The client library sets no handshake timeout of its own. A broker that accepts the TCP connection but never completes the AMQP handshake (a half-open port during startup) leaves connect() pending forever, which would hang startup straight past the deadline. So each probe is wrapped in its own timeout and raced against it, guaranteeing every attempt either succeeds or fails within a bounded window.
• An abandoned probe can still connect afterwards. When a probe times out, the underlying connection attempt may still succeed a moment later, and if you've walked away, that live connection (and its heartbeat timer) leaks for the life of the process. So a timed-out probe registers a callback to close the socket if it ever completes. A guard that leaks the thing it was guarding isn't a fix.

Honest about the altitude

This gate kills the trigger: a broker that's slow to accept connections at startup. I want to be clear that it does not, by itself, kill the class.

The class is "a consumer whose life is tied to a broker connection it can't survive losing." The startup probe gets all 46 consumers connected on boot, but the failure handling that started this (dial once, and disable on loss) is still there for a connection that drops mid-life. A broker restart an hour later would still take the fleet down the same way. The real fix for the class is per-consumer reconnect with backoff (a supervised connection that re-establishes itself and re-consumes), plus a health check that actually reports consumer connectivity so the next occurrence turns something red instead of staying green. The startup gate is the seatbelt this specific crash earned; the reconnect work is the airbag the class still needs.

And it was caught on a demo environment, not in production: a reboot there surfaced the race before a customer ever felt it. That's the good version of this story. The bad version is the same bug, the same green health check, discovered from a customer asking why they never got their email.

Takeaways

• "The process is up" is not "the process is working." A liveness check that can't distinguish a busy service from an idle one will pass straight through a total outage. Health-check the job, are the consumers connected and consuming, not just the pulse.
• Dependency ordering at first-boot is not resilience to a slow dependency. Health-gated startup conditions govern a manual up and are ignored when the runtime restarts everything itself on a host reboot. Assume your dependency can be late, and wait for it in code.
• A dial-once-and-give-up client is a landmine. If a consumer disables itself on the first connection failure and never retries, a dependency that's a few seconds late costs you that worker until the next deploy. Gate startup on the dependency answering; reconnect on loss.
• Non-deterministic breakage is a race signature. 45 of 46, not 46 of 46, is the tell. When the count of what breaks changes run to run, stop looking for a bad config and start looking at startup timing.

Sources

• RabbitMQ: Networking and listeners. Where the broker binds and listens for AMQP connections (port 5672), the listener that opened 38 seconds late here.
• Node.js: Common system errors. ECONNREFUSED is what a consumer got when it dialed that port before the listener was up.