A Zero-Downtime DigitalOcean → AWS Migration (50k Users)

One switch, thrown once, in public

A zero-downtime DigitalOcean to AWS migration reads like one line on a sprint board. In practice, moving a live production platform, one serving 50,000+ users, off DigitalOcean and onto AWS meant standing up an entire environment in a new cloud and then flipping every one of them onto it without a single one noticing. I was the only DevOps/SRE on it, so there was also nobody to hand the pager to if the cutover went sideways.

The whole project bends around one moment: the DNS cutover, a switch you get to throw exactly once, in public, on live traffic. So the real design goal was never "build AWS." It was build AWS in a way that lets me flip everyone onto it with the old stack still warm behind me, a net I could fall back into if the new floor didn't hold.

The setup we'd outgrown

The DigitalOcean environment wasn't wrong when it was built. It grew organically: provisioned by hand, one decision at a time, until its shape was whatever the last change left behind. For a platform that size, that had hardened into three liabilities:

• No real isolation. Compute, cache, and database weren't cleanly segmented, so the blast radius of any one mistake was larger than it had any right to be.
• Manual provisioning. Nothing was reproducible. "How is this configured?" could only be answered by logging in and looking, which is another way of saying it couldn't be answered at all.
• A cost and reliability profile that didn't fit the platform: no multi-AZ story, no in-region redundancy.

The most glaring piece was the network shape: the layers weren't cleanly isolated, so credentials were doing work that network boundaries should have been doing. What we needed was the opposite of organic: a reproducible, multi-AZ, in-region environment where database, secrets, and compute were properly separated, reached without an outage for customers who were mid-transaction while I worked.

Rebuild it as code, not by hand

Rule one: no click-ops. If the migration was going to be trustworthy, the new environment had to exist as code before it existed as infrastructure. Everything went into Terraform and Ansible: Terraform to describe the AWS footprint, Ansible to configure what ran on it. The payoff is that a reviewed, version-controlled environment can be rebuilt on demand, the same property that makes a rollback survivable.

The architecture:

• A multi-AZ EC2 Auto Scaling Group behind an Application Load Balancer, with all compute in private subnets. The ALB is the only thing with a public face; the instances that run the platform have no route in from the internet, and spreading the ASG across availability zones means a single AZ failure sheds capacity instead of taking the site down.
• ECR for container images, so every deploy pulls a known, immutable artifact rather than building on the box.
• ElastiCache (Valkey) for caching, with TLS and AUTH enforced, no more soft, unauthenticated target sitting next to production data.
• AWS Secrets Manager for credentials, so nothing sensitive lives in an AMI, an env file, or a Terraform state you have to remember to encrypt.

Segmentation was the theme. In the old world, everything could more or less reach everything. In the new one, each layer (load balancer, compute, cache, secrets, database) is its own boundary, and crossing it is an explicit rule.

Take the database off the public internet

If I had to point to the one change that did the most for the security posture, this is it, and it's the highest-value move in almost any cloud migration: make sure the database isn't reachable from the public internet at all. A database whose only defense is credentials is one leaked connection string away from a very bad day.

In the new architecture the database sits behind VPC peering plus an explicit allowlist: it only accepts traffic arriving over a private peering connection from a named set of sources, and nothing else can even open a socket. Credentials become the second line of defense instead of the only one, exactly the control an auditor wants to see, rather than a public endpoint gated by a password.

A health check should answer one question

This one you only learn by watching a system fail. An ALB decides whether to keep sending traffic to an instance by hitting a health endpoint, and if that endpoint fails, it pulls the node out of rotation. The check had drifted into reporting the message broker's state, which meant a transient queue hiccup, something customers would never have felt, could mark healthy instances unhealthy and yank the whole fleet out from under the load balancer.

So I decoupled the health signal from the broker. The check now answers one question only: can this instance serve requests? A backed-up queue is a queue problem, not an availability outage, and the load balancer should never confuse the two.

The cutover: draining DigitalOcean to AWS with the old stack warm

The mechanics of the cutover are unforgiving: DNS is not instant. A record's TTL decides how long resolvers keep serving the old answer, so for a window of time both values are in the wild: some customers resolving to AWS, some still to DigitalOcean. You don't cut traffic over; you drain it. Lowering the TTL ahead of the change is what shrinks that window from hours to minutes, in either direction. Respect that, and a safe cutover falls out of it:

• Keep the old DigitalOcean stack warm. The legacy environment stayed fully powered the whole time, able to take 100% of traffic back the instant a record pointed at it again. As the change propagated, traffic drained from DigitalOcean to AWS with DigitalOcean held on standby behind it.
• Watch the drain, not the clock. There was no single held-breath instant. "Abort" meant re-point the record and let the low TTL carry everyone home, not scramble to rebuild something I'd already torn down.

That's the whole trick to a zero-downtime DNS cutover: make forward and backward symmetric. If going back is as cheap as going forward, the flip stops being a gamble and becomes an observation.

And I didn't trust the fallback on faith. I ran the failure path for real before the switch: an actual restore drill that validated a ~5-minute recovery objective (RTO) end to end. Not "the docs say we can recover," but a stopwatch on a real restore, confirming the number before anything depended on it. Only once the new environment had proven stable under live traffic did I stand the old one down, retiring the legacy DigitalOcean stack four days early, a call I could make confidently precisely because the rollback net had been real, not theoretical.

What it unlocked

• A reproducible, multi-AZ production environment, fully in code. "How is production configured?" now has an answer that lives in a repository, and the whole thing can be rebuilt from it.
• Zero customer-facing downtime. All 50,000+ users changed clouds and none of them had to know.
• A materially hardened security posture (private subnets, a VPC-peered database behind an explicit allowlist, TLS/AUTH on the cache, secrets in a real store), the kind of foundation a later security review leans on instead of scrambling to retrofit.
• A foundation, not a finish line. With production finally expressed as code, the follow-on work had something solid to stand on: GitHub Actions CI/CD across the service stack, blue-green deploys via an ALB target-group swap (~2-minute deploys, ~3-minute rollbacks), and Grafana/Loki/Promtail observability baked into the ASG launch template. The cost program that followed, killing NAT egress with VPC peering and right-sizing the MongoDB cluster among them, trimmed the monthly bill by roughly a fifth, with no reliability regression.

Takeaways

• Design the migration around the rollback, not the launch. The environment that's easy to fall back from is also the one that's safe to cut over to.
• DNS is a drain, not a switch. Lower the TTL early, keep the old stack warm, and let propagation do the work. Both answers live in the wild for a while, so plan for that window instead of pretending it doesn't exist.
• No click-ops. A reproducible Terraform/Ansible environment is what makes a rollback survivable in the first place.
• Take the database off the public internet. VPC peering plus an allowlist makes credentials your second line of defense instead of your only one.
• Decouple health checks from your dependencies. An ALB should ask "can this node serve requests?", not "is every downstream system happy?"
• A recovery objective you haven't drilled is a number you made up. Run the real restore, put a stopwatch on it, and confirm the RTO before production depends on it.