How to design redundant cloud connectivity

Why this matters

How to design redundant cloud connectivity comes down to counting failure domains. Two circuits that share a conduit, a router or a carrier are one circuit with two invoices. Count how many separate things have to break before traffic stops. If the answer is one, you have a single point of failure wearing a backup's clothes.

Partial failure is the harder problem. A link that stays up but drops packets will pass a ping test and still break sessions. Half-open TCP connections, retransmits and a routing table that changes every 30 seconds hurt more than a clean cut, because nothing fails over and nobody gets paged.

Tie redundancy to what you actually sell. An order pipeline that stops means revenue stops that minute. A nightly batch can wait until morning. Write those numbers down before you buy anything, because they drive everything after: how many paths, how fast detection must be, and where the money goes.

Redundancy has layers: the physical path, the router, the carrier, the cloud gateway, the region, the application. You rarely need all six. You do need to know which ones you have, so you can tell a design from a hope.

Before you start

Start with a written target per system. Recovery time is how long until service returns. Recovery point is how much data you can lose. A payments path might need seconds. An internal wiki can wait an hour. One number for the whole estate is how you overbuild the wiki and underbuild checkout.

Inventory what you have. For each circuit, note the carrier, the circuit ID, the building entry point, the meet-me room, the rack, the terminating device, the port speed and the contract term. Ask each carrier whether the second path is genuinely diverse, and get the answer in writing with the route described.

Check lead times and cloud side limits before you commit. Dedicated circuits take weeks to provision and longer to become diverse. Cloud VPN gateways have tunnel counts, BGP ASN and route limits that you can hit at scale. Look at the quotas on the account you will actually use.

List the failures you are designing against: fiber cut, carrier outage, router failure, cloud edge issue, configuration error. The last one is the most common, and extra links do not help against it. Version control, staged changes and out-of-band console access do.

How to do it

Choose the shape first. Active passive is simpler: one path carries traffic, the other waits, and failover is a routing change. Active active spreads load across both circuits, but it needs symmetric routing or careful state handling. A stateful firewall that sees only the outbound half of a flow will drop the return traffic, and you will lose a day finding out.

Diversity is a physical property, not a contract clause. Two carriers can share a duct into your building. Ask for separate entry points, separate meet-me rooms and separate last-mile paths. On the cloud side, terminate each circuit on a different device and, where the budget allows, a different edge location.

Make detection fast and failover boring. BGP with BFD notices a dead peer in milliseconds. Without it you wait for the hold timer, which defaults to 180 seconds. Use local preference to set the primary path, keep AS path prepending as a second lever, and add a hold-down so a flapping link does not churn the table. Log every session transition.

1. Map every failure domain. Draw the path from each site through each carrier, building entry, router, cloud edge and gateway to the workload. Mark every shared conduit, rack, power feed and device. Anything shared is a single point of failure.
2. Set a recovery target per system. Write down how long each service can be down and how much data it can lose. Use those numbers to decide where redundancy goes. Resist one target for everything.
3. Order diverse circuits and confirm the path. Buy from different carriers where the budget allows, and ask for separate building entry points. Get the diversity claim in writing with the route described. Check the lead time before you promise a date.
4. Terminate on separate hardware. Give each circuit its own router, its own power feed and its own configuration backup. Cross-connect the routers so either can reach both paths. Keep a console you can reach when the network is broken.
5. Build BGP failover with fast detection. Use local preference for the primary path and BFD for sub-second detection. Set a hold-down so a flapping link does not churn the table. Verify ASN and prefix limits on the cloud side before cutover.
6. Set up the cloud gateways and tunnels. Use both tunnels on each cloud VPN connection, or separate virtual interfaces on a dedicated circuit. Place them on different gateway instances where the provider supports it. Send tunnel state to your monitoring.
7. Add application level failover. Health-checked DNS with a short TTL, a global load balancer, or client retries with backoff. Remember that long-lived connections ignore DNS, so close and reopen them on a path change.
8. Break it on purpose. Shut an interface, kill a BGP session, disable a tunnel, during business hours with users on the system. Measure time to first successful request and to full throughput. Fix what you find and run it again next quarter.

How to check it worked

Test from the outside in, with real traffic. Shut the primary interface during business hours and time how long until a user request succeeds. BGP convergence is not the number you want. Time to first successful request, and time to full throughput on the backup, are.

Check both directions. Trace the path out and back, confirm the return traffic uses the link you expect, and read the firewall state table and NAT translations. Asymmetric routing after failover is the usual culprit when things are slow but not broken.

Watch the data plane, not just the control plane. A BGP session can be up while the tunnel behind it is dead, so run synthetic probes from outside the network every 30 seconds. Test MTU too: a VPN path often carries 1500 bytes where the direct path carried 9000, and large packets vanish when path MTU discovery is blocked. Send do-not-fragment pings at rising sizes to find the real limit, then clamp MSS.

Repeat the failure test on a schedule and record the result next to the design. A backup path that has never carried production traffic is a guess.

Mistakes to avoid

Two circuits from one carrier. It feels like redundancy and it fails like a single link when that carrier has a bad day. Same story for two links through one conduit, one patch panel or one power feed.

Sizing the standby for the best case. If the backup carries only a fraction of the primary bandwidth, failover succeeds and the service collapses under load. Size each path to carry critical traffic alone, or accept the reduced service and write down what it looks like.

Active active without shared state. Stateless web tiers handle it fine. Session-based applications, stateful firewalls and anything that pins a client to a server need affinity or shared state, or users get logged out mid-session when a path shifts.

Letting the standby rot. Untested configs drift, certificates expire, BGP peers get decommissioned, and then the real failover happens. If you want a second opinion on the design before you sign circuit contracts, the Expert architecture session at $450 from Cloudverg is one option among others.

Frequently asked questions