Incident post-mortem analysis: us-central1 service disruption on August 19, 2026

Root causeRoot cause

The direct cause of the incident was a storm-related event at the data center building that simultaneously disabled the building management system and the chilled-water cooling plant. With active cooling lost, the data halls overheated within about two hours, and servers, network switches and rack power systems shut down on thermal protection. The facility operator restored cooling in manual bypass mode, and the thermal environment was stabilized about six hours after the storm-related event.

Two factors extended the customer-visible impact well beyond the facility recovery:

Detection depended on equipment-level symptoms. The storm-related event disabled the building management system, which is also the system that monitors the facility, so no facility-level alert reached either the facility operator or us. Our engineering team detected the anomaly via secondary equipment-level thermal indicators and proactively escalated to the facility operator to initiate manual bypass protocols. On our side, regional dashboards show the temperature rise starting around 09:15 UTC in retrospect, but no alerting existed on facility or room temperature trends; the first signals engineers acted on were component over-temperature alerts at 10:15 UTC. As a result, there was no practical window to shed heat load in a controlled way: by the time it was considered, about a quarter of the affected servers had already shut down.

A region-scale cold start required manual work in several recovery paths.

• The regional infrastructure control plane did not recover unattended after power was restored and required about two hours of manual restoration before dependent services could begin recovering.

• Automatic VM recovery is designed as a single attempt per instance and is rate-limited to protect the platform during isolated failures. The platform launched automatic recovery for roughly a third of all running instances in the region; about a third of these attempts succeeded, and the rest failed, mostly on storage-attachment errors caused by stale disk attachment state on abruptly powered-off hosts. Failed attempts were not retried automatically, and the rate limits had to be raised manually mid-recovery.

• A bug in the managed disks hot-plug functionality left some secondary data disks unattached after recovery. Managed Kubernetes cluster control planes store their state on such disks, so hundreds of recovered control-plane instances came back without their data disk, and their clusters stayed down until the bug was identified at 18:17 UTC and a controlled mass restart was applied. The bug has since been fixed and the fix deployed to all regions.

• Regional monitoring depended on components inside the affected region, which reduced responders' visibility during the acute phase and slowed recovery decisions.

Incident response outcomesIncident response outcomes

The facility-level response was fast once the event was detected: the incident was declared within minutes of the paging alerts, the facility operator was engaged immediately, cooling was switched to manual mode within an hour of the incident declaration, and hardware was progressively re-energized as soon as the thermal environment allowed, with control-plane racks prioritized.

The platform-level response surfaced the following lessons:

Facility monitoring must be independent of the systems it monitors. A single storm-related event disabled both the cooling plant and the building management system that would have reported the failure, so neither the facility operator nor Nebius received a facility-level alert. This was the main detection gap; the action plan addresses both the operator-side and the Nebius-side part of it.

Environmental data must be covered by alerting. The temperature rise was visible on our dashboards from about 09:15 UTC, but no alert was configured on facility or room temperature trends; the alerts that fired an hour later reported component overheating, a consequence rather than the cause.

Region-scale recovery must be a rehearsed procedure. Improvements made after earlier incidents worked as intended: control-plane racks were re-energized first, network gateways were restored within about two hours of power returning, and object storage recovered automatically. Because the broader region-scale recovery program remains in progress and has not yet reached the stage of full-scale recovery drills, the restoration sequence across dependent layers was determined and coordinated manually during the incident. The incident also defined the next scope for this program: the longest part of the recovery was restoring customer resources (virtual machines, disk attachments, Managed Kubernetes control planes), and we are extending the drills to cover that layer.

Recovery automation must be designed for mass failure, not only for isolated failure. Single-attempt recovery, conservative rate limits, and the inability to distinguish incident-stopped instances from user-stopped ones required manual work at scale. Our goal is fast, predictable region recovery supported by both automation and rehearsed procedures.

Observability and traffic management must degrade gracefully with the region. Monitoring components had startup and data dependencies inside the affected region, which reduced our visibility during the acute phase and recovery. The regional health check used for global console traffic management reported the region healthy while it was not, briefly routing traffic back into it.

Post-incident action planPost-incident action plan

Facility resilience

• Complete a joint resilience review with the facility operator, covering redundancy of the facility-management and monitoring systems, the ability of the cooling plant to keep operating if those systems fail, electrical protection of the cooling equipment, and on-site response coverage.

• Introduce facility-level environmental alerting (cooling plant state, room temperature trend) that pages our on-call engineers independently of the building management system and before workload-level symptoms appear.

• Establish a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times.

• Define an emergency heat-load reduction procedure: explicit criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss.

Region-scale recovery

• Establish a formal, dependency-ordered region cold-start procedure with clearly defined ownership across each service layer.

• Implement a regular cadence of region-scale recovery drills, ensuring end-to-end validation across core platform components and customer workloads (VMs, storage attachments, and Managed Kubernetes clusters).

Recovery of customer resources

• Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions (completed).

• Improve the ways Compute users can configure recovery policies for their instances. Migrate all Managed Kubernetes control plane nodes to the improved recovery policy.

• Automate reconciliation of stale storage attachment state after abrupt host failures, so disks can be mounted and deleted without manual intervention.

Observability and traffic management

• Remove in-region dependencies from the monitoring stack startup and recovery paths, and extend cross-region monitoring so that we retain visibility into a region even when its local monitoring is degraded.

• Make regional health checks used for global traffic management reflect real service availability, so traffic fails over from an unhealthy region and does not return prematurely.