The "DELIVERED" That Never Arrived

Every message said DELIVERED. Half of them bounced.

By its own records, the notification service for a platform I worked on was flawless: very nearly 100% of the email it sent was marked DELIVERED. It was a clean, reassuring number, and it had been a lie since the day the field was added.

What sent me looking wasn't the delivery graph. That looked perfect, which is exactly the problem. It was the bill (the SES run-rate had jumped 4–5× in a week) and pulling that thread unravelled two independent defects and one badly-designed metric that had been quietly hiding both. By the end I'd found a steady 52% bounce rate the dashboard couldn't show, a production email archive that had been dead for days without a single alert, and an invoice running at almost exactly double what it should, all downstream of the same small assumption.

"Accepted" is not "delivered"

The assumption is an easy one to make. When you call the SES SendEmail API and it returns 200 with a MessageId, that response does not mean the message was delivered. It means SES has accepted the message for delivery. It's now queued inside SES, which will attempt delivery and, some seconds to minutes later, find out whether a real mailbox accepted it, rejected it (a bounce), or flagged it as spam (a complaint).

The service recorded delivery at the wrong moment:

const res = await ses.sendEmail(params);   // resolves the instant SES ACCEPTS
// res.MessageId now exists, but the message has only been queued,
// not delivered. Nothing here knows the real outcome yet.
await Notification.updateOne(
  { _id },
  { status: "DELIVERED", messageId: res.MessageId }   // ❌ recorded far too early
);

Read that closely and the bug isn't "sometimes wrong." The status field had no state to represent a bounce. It was stamped DELIVERED at the one moment every message looks identical (the moment SES says "got it") and then never updated again. A metric that is written before the outcome exists isn't measuring the outcome. It's measuring that you tried.

The 52% you couldn't see

Because delivery was booked at acceptance, the true outcome (which arrives afterwards, and only if you're listening for it) was thrown on the floor. And the true outcome was ugly: a steady 52% bounce rate, meaning roughly 37,000 bounces a day were being filed in the database as DELIVERED.

A 52% bounce rate is not a quality-of-life problem; it's an existential one. Mailbox providers and SES itself watch that number, and a sender sitting anywhere near it gets throttled, then suppressed, then blocked. The reputation of the whole sending domain bleeds out. It had been sitting there for who-knows-how-long, and the only reason nobody panicked is that the graph that would have shown it was defined in a way that could only ever read 100%.

One hardcoded BCC, three separate failures

So where were 37,000 bounces a day coming from? Every outbound email carried a hardcoded BCC to a single fallback address: an old idea for keeping a copy of everything the system sent, an informal "archive." That one line of config was, on its own, responsible for three unrelated production problems:

1. It doubled the bill. SES bills per recipient, and a BCC is a second recipient. Every send was two. I measured the ratio of billed recipients to intended ones and it came back at exactly 2.000. Literally half the SES invoice was the BCC, and no configuration value could turn it off.
2. It generated essentially all the bounces. That archive mailbox had itself hard-bounced and been placed on SES's suppression list. So every single message, delivered to its real recipient or not, also went to an address guaranteed to bounce. Two recipients per send, one of them always failing: that alone puts a ~50% floor under the bounce rate. There's your 52%, almost to the point.
3. It silently killed the "archive." The archive was that mailbox. The day SES suppressed it, the copy-of-everything stopped arriving, and because delivery was booked at acceptance, nothing recorded that it had stopped. The archive had been dead for days before anyone noticed.

One line. It doubled the invoice, manufactured the bounce rate, and quietly ended the archive it was supposed to be, and the metric that should have screamed about all three had been designed to smile through it.

The bill had a second problem, too

While I was in there, the 4–5× surge that started the whole investigation turned out to have its own separate cause. A recurring sweep re-sent a reminder for every open item on a fixed schedule, with no staleness bound, so it re-nagged the entire backlog, over and over. Measured over a full unbiased cycle, it was firing 52,491 messages per run: 73.2% of all outbound email.

It hadn't always been loud. The job had gone dormant after an infrastructure cutover and then been silently switched back on by an unrelated deploy, and its two supposed safety guards were both inert by construction:

• A 1-day cooldown on a job that only ran every 2 days: a guard that can never once fire.
• A lifetime per-user cap that, once hit, muted a user forever instead of pacing them.

Guards that can't trigger are worse than no guards, because they read as protection in code review. Both had to be rebuilt around a real recency window.

The fix: measure arrival, not acceptance

The remediation had two halves (stop lying, and stop overspending) and the satisfying part is that the truth was already there, just unused.

Reconcile from real events, not the API response. SES already publishes every Delivery, Bounce, Complaint, and Open to an SNS topic feeding an SQS queue, a feed that had been provisioned at some point and then discarded. I wired it back onto the records: the status is now written when the event arrives, overwriting the optimistic "delivered-on-accept" with what actually happened.

// SES → SNS → SQS. Reconcile the record from the event, not the send call.
for (const event of await pullFromSqs()) {
  await Notification.updateOne(
    { messageId: event.mail.messageId },   // sparse-indexed join key
    { status: event.eventType }            // DELIVERED · BOUNCE · COMPLAINT · OPEN
  );
}

Two details in that consumer earned their keep:

• I mounted the event consumer outside the main message-consumer startup, so delivery truth keeps reconciling even during a broker outage, the moment you'd least want to also go blind on deliverability. Failed pulls are left on the queue for SQS redrive rather than dropped.
• I gave the messageId join key a sparse index. Without it, every inbound event would collection-scan the entire notification history to find its record, turning the fix into its own performance incident.

Then, kill the two cost defects at the source. The BCC became opt-in with no default, which retired the bill-doubler and the bounce floor in a single change. The runaway sweep got a real recency window, its cooldown widened from 1 day to 14, and a rolling 30-day per-user cap instead of a lifetime one. I also pinned the SES configuration set explicitly, rather than leaning on an invisible identity default that could change out from under us.

Together those took the SES run-rate from ~$215/mo toward a projected ~$25–40/mo, about 85% lower, retired the 52% bounce rate outright, and, for the first time, made DELIVERED mean delivered.

I had to retract my own first RCA

There's an epilogue I'd rather not include and will anyway, because it's the most useful part. My first writeup of this incident was substantially wrong. Working from too small a sample, I'd blamed the surge on missing idempotency and warned of a different deliverability risk than the real one. I'd already published it.

What corrected me was boring discipline: re-measuring over a full, unbiased window covering one complete cycle of the job instead of a convenient slice. The real numbers (the 2.000 billing ratio, the 73.2% share, the BCC as the bounce source) disproved my own conclusion. So I rewrote the RCA, corrected the cost estimate against the actual invoice, and kept the wrong version in an appendix rather than quietly deleting it, so the next person can see how a plausible story fell apart against better data.

An incident report is a metric too. It's just as capable of being confidently, silently wrong.

Takeaways

• Record outcomes when they happen, not when you request them. An API 200 from a delivery provider means accepted, not arrived. If you stamp success on the acknowledgement, your success metric is measuring effort, and it can never show a failure.
• A metric with no failure state is decoration. "Near-100% delivered" was structurally incapable of reading anything else. Before you trust a green number, ask what value would represent the bad case, and whether the code can even produce it.
• Wire up the bounce/complaint feedback loop from day one. SES hands you Delivery/Bounce/Complaint/Open events for free over SNS→SQS. Not consuming them doesn't make the bounces go away. It just means you find out from your suppression list instead of your dashboard.
• Audit the cheap-looking config lines. A single hardcoded BCC doubled the bill, manufactured a 52% bounce rate, and killed an archive. Small lines can have large, invisible blast radii.
• A guard that can't fire is worse than none. A 1-day cooldown on a 2-day job reads as a safety net in review and is a no-op in production. Prove your guardrails can actually trigger.
• Measure over a full cycle before you conclude, and correct the record in public. My first RCA was wrong because my sample was biased. When better data overturns you, rewrite it and leave the old version visible. The retraction is worth more than the mistake cost.