One switch, thrown once, in public

A zero-downtime DigitalOcean to AWS migration reads like one line on a sprint board. In practice, moving a live production platform — one serving 50,000+ users — off DigitalOcean and onto AWS meant standing up an entire environment in a new cloud and then flipping every one of them onto it without a single one noticing. I was the only DevOps/SRE on it, so there was also nobody to hand the pager to if the cutover went sideways.

The whole project bends around one moment: the DNS cutover — a switch you get to throw exactly once, in public, on live traffic. So the real design goal was never "build AWS." It was build AWS in a way that lets me flip everyone onto it with the old stack still warm behind me — a net I could fall back into if the new floor didn't hold.

The setup we'd outgrown

The DigitalOcean environment wasn't wrong when it was built. It grew organically — provisioned by hand, one decision at a time, until its shape was whatever the last change left behind. For a platform that size, that had hardened into three liabilities:

• No real isolation. Compute, cache, and database weren't cleanly segmented, so the blast radius of any one mistake was larger than it had any right to be.
• Manual provisioning. Nothing was reproducible. "How is this configured?" could only be answered by logging in and looking — which is another way of saying it couldn't be answered at all.
• A cost and reliability profile that didn't fit the platform — no multi-AZ story, no in-region redundancy.

The most glaring piece was the network shape: the layers weren't cleanly isolated, so credentials were doing work that network boundaries should have been doing. What we needed was the opposite of organic: a reproducible, multi-AZ, in-region environment where database, secrets, and compute were properly separated — reached without an outage for customers who were mid-transaction while I worked.

Rebuild it as code, not by hand

Rule one: no click-ops. If the migration was going to be trustworthy, the new environment had to exist as code before it existed as infrastructure. Everything went into Terraform and Ansible — Terraform to describe the AWS footprint, Ansible to configure what ran on it. The payoff isn't tidiness; it's that a reviewed, version-controlled environment can be rebuilt on demand — the same property that makes a rollback survivable.

The architecture:

• A multi-AZ EC2 Auto Scaling Group behind an Application Load Balancer, with all compute in private subnets. The ALB is the only thing with a public face; the instances that run the platform have no route in from the internet, and spreading the ASG across availability zones means a single AZ failure sheds capacity instead of taking the site down.
• ECR for container images, so every deploy pulls a known, immutable artifact rather than building on the box.
• ElastiCache (Valkey) for caching, with TLS and AUTH enforced — no more soft, unauthenticated target sitting next to production data.
• AWS Secrets Manager for credentials, so nothing sensitive lives in an AMI, an env file, or a Terraform state you have to remember to encrypt.

Segmentation was the theme. In the old world, everything could more or less reach everything. In the new one, each layer — load balancer, compute, cache, secrets, database — is its own boundary, and crossing it is an explicit rule.

Take the database off the public internet

If I had to point to the one change that did the most for the security posture, this is it — and it's the highest-value move in almost any cloud migration: make sure the database isn't reachable from the public internet at all. A database whose only defense is credentials is one leaked connection string away from a very bad day.

In the new architecture the database sits behind VPC peering plus an explicit allowlist: it only accepts traffic arriving over a private peering connection from a named set of sources, and nothing else can even open a socket. Credentials become the second line of defense instead of the only one — exactly the control an auditor wants to see, rather than a public endpoint gated by a password.

A health check should answer one question

This one you only learn by watching a system fail. An ALB decides whether to keep sending traffic to an instance by hitting a health endpoint — and if that endpoint fails, it pulls the node out of rotation. The check had drifted into reporting the message broker's state, which meant a transient queue hiccup — something customers would never have felt — could mark healthy instances unhealthy and yank the whole fleet out from under the load balancer.

So I decoupled the health signal from the broker. The check now answers one question only: can this instance serve requests? A backed-up queue is a queue problem, not an availability outage — and the load balancer should never confuse the two.

The cutover: draining DigitalOcean to AWS with the old stack warm

This is the moment the whole project was built around. The mechanics are unforgiving: DNS is not instant. A record's TTL decides how long resolvers keep serving the old answer, so for a window of time both values are in the wild — some customers resolving to AWS, some still to DigitalOcean. You don't cut traffic over; you drain it. Lowering the TTL ahead of the change is what shrinks that window from hours to minutes, in either direction. Respect that, and a safe cutover falls out of it:

• Keep the old DigitalOcean stack warm. This is the net. The legacy environment stayed ready the whole time — not powered down, not scaled to zero, but able to take 100% of traffic back the instant a record pointed at it again. As the change propagated, traffic drained from DigitalOcean to AWS with DigitalOcean held on standby behind it.
• Watch the drain, not the clock. There was no single held-breath instant. "Abort" meant re-point the record and let the low TTL carry everyone home — not scramble to rebuild something I'd already torn down.

That's the whole trick to a zero-downtime DNS cutover: make forward and backward symmetric. If going back is as cheap as going forward, the flip stops being a gamble and becomes an observation.

And I didn't trust the fallback on faith. I ran the failure path for real before the switch — an actual restore drill that validated a ~5-minute recovery objective (RTO) end to end. Not "the docs say we can recover," but a stopwatch on a real restore, confirming the number before anything depended on it. Only once the new environment had proven stable under live traffic did I stand the old one down, retiring the legacy DigitalOcean stack four days early — a call I could make confidently precisely because the rollback net had been real, not theoretical.

What it unlocked

• A reproducible, multi-AZ production environment, fully in code. "How is production configured?" now has an answer that lives in a repository, and the whole thing can be rebuilt from it.
• Zero customer-facing downtime. All 50,000+ users changed clouds and none of them had to know.
• A materially hardened security posture — private subnets, a VPC-peered database behind an explicit allowlist, TLS/AUTH on the cache, secrets in a real store — the kind of foundation a later security review leans on instead of scrambling to retrofit.
• A foundation, not a finish line. With production finally expressed as code, the follow-on work had something solid to stand on: GitHub Actions CI/CD across the service stack, blue-green deploys via an ALB target-group swap (~2-minute deploys, ~3-minute rollbacks), and Grafana/Loki/Promtail observability baked into the ASG launch template. The cost program that followed — killing NAT egress with VPC peering and right-sizing the MongoDB cluster among them — trimmed the monthly bill by roughly a fifth, with no reliability regression.

Takeaways

• Design the migration around the rollback, not the launch. The environment that's easy to fall back from is also the one that's safe to cut over to.
• DNS is a drain, not a switch. Lower the TTL early, keep the old stack warm, and let propagation do the work — both answers live in the wild for a while, so plan for that window instead of pretending it doesn't exist.
• No click-ops. A reproducible Terraform/Ansible environment isn't tidiness for its own sake; it's what makes a rollback survivable in the first place.
• Take the database off the public internet. VPC peering plus an allowlist makes credentials your second line of defense instead of your only one.
• Decouple health checks from your dependencies. An ALB should ask "can this node serve requests?" — not "is every downstream system happy?"
• A recovery objective you haven't drilled is a number you made up. Run the real restore, put a stopwatch on it, and confirm the RTO before production depends on it.