Learning, Building, and
Documenting
Field notes on DevOps, SRE, and keeping production fast, reliable, and cheap. Browse the archive or subscribe via RSS.
Right-Sizing a Production MongoDB Cluster: M50 → M40
You can't just click "downgrade" on a production database. Here's the log forensics, index surgery, and aggregation rewrites that shrank the working set enough to take a MongoDB Atlas cluster from M50 to M40 — a full tier down, slow-query time down ~43%, and latency that improved through the cut.
A Zero-Downtime DigitalOcean → AWS Migration (50k Users)
Moving a live 50,000-user production platform off DigitalOcean and onto AWS without a single customer noticing — rebuilt entirely in Terraform and flipped over with one reversible DNS cutover, with the old stack kept warm as a real rollback net.
The "DELIVERED" That Never Arrived
A notification database was reporting DELIVERED for mail that never arrived. Delivery was recorded the instant SES accepted the API call — not when the message landed — so ~37,000 bounces a day were filed as successes, hiding a steady 52% bounce rate and a bill quietly running at double. The fix was to stop trusting acceptance and wire the real delivery events back on.
The Logged-Off Desktop That Killed 62 of 66 Production Processes
62 of 66 processes on a single Windows box went dark — while disk, RAM, and CPU all read green. The real cause was a runaway terminal exhausting the Windows commit limit, crashing the desktop compositor, and logging off the session every process lived inside. An incident I debugged end to end over AWS SSM, with no RDP.