Google's London Zone Outage: Plan for Capacity, Not Just Zones

On October 20, a power failure in a single Google Cloud zone in London, europe-west2-b, turned into a regional problem. Google's status page listed 38 affected products, from Compute Engine, Persistent Disk and Google Kubernetes Engine to Cloud SQL, Cloud Storage, BigQuery, Pub/Sub and Cloud Load Balancing. According to Google's incident report, primary impact lasted 6 hours and 8 minutes, from 23:00 US/Pacific on October 19 (07:00 in London), with VM recovery continuing until 05:08 Pacific.

Google published its incident report on October 27. It is worth reading in full, because the root cause is less interesting than what happened next: the outage started in one zone, but the effects reached the other two zones in the region and several regional services. If your resilience plan is "we're deployed across zones, so we're fine," this incident is a useful test of that assumption, whichever cloud you run on.

What failed, according to Google

The incident report describes a chain of ordinary events that lined up badly:

• The site was already on one utility feed. The breaker for utility feed B had been locked out for repairs after a generator load test on October 14.
• Feed A then failed. An underground cable fault took out the remaining utility feed.
• The generators never started. The automation system's undervoltage sensing was wired on the line side of the isolated feed B breaker, so it still saw voltage and never triggered the generators.
• The batteries ran out. Once the UPS batteries drained, servers, networking and storage in the zone lost power.

Google has committed to repairing feed A, running full generator load transfer tests, reviewing and testing the automation code, validating the same configuration at similar sites worldwide, and freezing planned maintenance at those sites while requiring elevated approval for non-standard configuration changes. On the cloud side, it plans automated alerting for location-impacting incidents and broader infrastructure event testing.

Why one zone became a regional problem

Zones exist to contain this kind of failure. Google's own documentation says it designs zones "to minimize the risk of correlated failures" and recommends "deploying applications across multiple zones and multiple regions." So why were zones a and c affected?

The incident reports name several reasons:

• Demand surges. Workloads failing over from zone b hit the surviving zones at the same time. The preliminary report cites compute capacity exhaustion in the unaffected zones, and the final report says some failovers failed because of capacity constraints.
• Cross-zone dependencies. The preliminary report points to "synchronous control-plane validation dependencies across zonal boundaries." Calls to components that were offline caused timeouts and queuing elsewhere.
• Storage recovery. Some failovers ran into storage replication difficulties, and storage devices that shut down uncleanly needed lengthy recovery before services could come back.

During the incident, Google's suggested workaround was to move affected workloads to another zone or region "if feasible." Later updates said Google was prioritizing existing workloads and that customers "may not be able to yet create new workloads." That last line is the one to remember. A recovery plan that depends on launching fresh capacity in a healthy zone during a regional incident is competing with everyone else trying to do the same thing, against a control plane that is itself under strain.

Static stability: build so failover needs nothing new

AWS has a name for the design pattern that addresses this, and it applies on any cloud. The AWS Well-Architected Framework's reliability guidance (REL11-BP05) warns against bimodal behavior, where a workload behaves differently in normal and failure modes, "for example, relying on launching new instances if an Availability Zone fails." The recommended approach is to "provision enough instances in each Availability Zone to handle the workload load if one AZ were removed" and use load balancer or DNS health checks to shift traffic away.

The guidance is candid about the trade-off: running spare capacity costs more than launching it on demand, and teams have to weigh reliability against cost for each workload. It also notes that "by using more Availability Zones, the amount of additional compute you need for static stability decreases." In a three-zone region, surviving the loss of one zone means the remaining two must carry the full load, so each needs headroom above its normal share.

On Google Cloud specifically, the incident report is explicit that Compute Engine zonal resources "require customers to provision failover or cross-zone recovery resources if needed." Google's documentation adds two details that matter for planning:

• Zonal resources stay zonal. Zonal persistent disks can only be attached to instances in the same zone. If your recovery plan involves reattaching data in another zone, you need regional disks, replication or snapshots set up in advance.
• Reservations can hold capacity. A Compute Engine reservation reserves capacity in a specific zone and "provides a high level of assurance of capacity" for a matching VM configuration. Reserved resources are billed at the same on-demand rate as running VMs, with applicable discounts, for as long as the reservation exists. Google lists disaster recovery as one of the uses.

Reservations are not free insurance, and not every workload needs them. But for the handful of systems where hours of downtime are unacceptable, paying for guaranteed capacity in a second zone (or a second region) is usually cheaper than discovering during an outage that the capacity isn't there.

Look for the dependencies you didn't draw

Google's own platform had cross-zone dependencies that amplified the failure. Most enterprise architectures do too, and they rarely appear on the diagram:

• A database primary in one zone, with application tiers in three, means a single zone failure still forces a database failover.
• Shared services such as NAT, private connectivity, message queues, caches, CI runners or identity integrations may be pinned to one zone or depend on regional control planes.
• Retry logic without backoff can turn a partial failure into a load spike on the surviving zones, the same pattern Google described with timeouts and queuing.
• Monitoring, paging and runbook tooling hosted in the same region as the workload can fail with it.

A practical checklist for the next 30 days

1. Read the incident report with your platform team. Map each failure mode (capacity exhaustion, cross-zone control-plane calls, storage recovery) to your own critical workloads.
2. Classify workloads by tolerance. Decide which systems need to survive a zone loss with no manual action, which can tolerate hours, and which can wait for the provider.
3. Check N+1 capacity. For tier-one systems, confirm the surviving zones can carry full load today, not after autoscaling succeeds. Consider reservations where capacity is scarce, such as GPU or large machine types.
4. Audit zonal resources. List zonal disks, single-zone databases, and services with a single zonal endpoint. Decide whether each needs regional replication, cross-zone recovery resources, or an accepted risk.
5. Revisit the region question. For systems where a regional degradation is unacceptable, document what a cross-region failover would require, and how long it would take.
6. Run a zone-loss exercise. Drain or isolate one zone in a non-production environment and measure what actually happens, including how your monitoring and on-call tooling behave.
7. Update the runbook. Write down who decides to fail over, how traffic is shifted, and what you do if you cannot create new resources for several hours.

Single-zone failures will keep happening on every provider; Google's report shows how a routine maintenance state and one misconfigured sensor can line up. The difference between a bad morning and a business outage is usually decided long before the incident, in capacity and dependency choices. Baygramm helps organizations design and test Google Cloud architectures that hold up when a zone goes dark, and builds the cloud visibility to spot single points of failure before they show up on a status page. If you'd like a second set of eyes on your failover plan, get in touch.