2026-08-27

A malformed domain name propagated to the public load balancers crash-looped them, making applications, the Console and the APIs unreachable for around 20 minutes.

On August 27, 2026, between 15:55 and 16:15 CEST, the public load balancers of Clever Cloud stopped serving traffic in every public region. Applications hosted on the platform, the Console, and our APIs were unreachable for up to 19 minutes, with traffic progressively recovering from 16:06. No data was lost, and applications themselves kept running throughout: the outage was confined to the routing layer.

The incident was limited to the public load balancers, which run Sōzu, in every region. Dedicated load balancers weren’t affected. Status updates were communicated through: https://www.clevercloudstatus.com/incidents/41b6e36e-ea51-4e88-b79b-ad1093c1d837

Timeline (CEST)

Analysis

At 15:55 CEST, a call to our self-service API added a domain name to an application, which triggered the propagation of the corresponding ACLs to our load balancers. The update contained an application hostname with a trailing slash (of the form example.com/): the faulty control-plane component published the domain name as-is, without stripping the path portion, when adding it to the load balancer configuration.

Sōzu, the open-source load balancer we develop and operate, interprets a / in a hostname as the opening of a regular-expression pattern. Inserting this malformed entry into its routing table failed an internal assertion, which crashed the worker processes.

Sōzu’s main process automatically restarts crashed workers and replays the current routing state to them. Since that state still contained the malformed entry, every new worker crashed within milliseconds, and the load balancers never re-bound their listening ports. Because routing configuration is propagated to all public regions, every public load balancer was affected within seconds. From the outside, connections were accepted by the layer in front and then failed to reach a backend, which surfaced as connection errors and 525 errors on sites behind Cloudflare.

Root cause

Two defects combined:

• A control-plane defect: a component published an application’s domain name including its path (example.com/) as the load balancer frontend hostname, instead of splitting host and path first
• A robustness defect in Sōzu: a malformed frontend hostname made the routing table insertion panic the worker instead of being rejected as an invalid configuration change

Two factors turned a bad configuration entry into a global outage:

• Routing configuration is propagated to all public regions at once, with no canary stage
• The worker auto-restart mechanism replays the faulty state, turning a single crash into a persistent crash loop

Detection and response

Our external end-to-end probes fired within seconds and paged the on-call teams at 15:55. Our process-level monitoring of the load balancers reported them as healthy: the main process was alive, only its workers were crash-looping.

The incident was published on our status page at 16:00. At around 16:05 we fully restarted the load balancer services in the Paris region. A full restart fetches a complete routing configuration from our API, which publishes correctly formed hostnames, so the restarted load balancers came back clean. Traffic in Paris recovered from 16:06, the remaining regions were restarted between 16:09 and 16:14, and all regions were nominal at 16:15. By 16:35 the root cause had been identified and reproduced locally, and fixes were opened the same day.

Actions

Done:

• The faulty control-plane component was taken out of production, fixed and redeployed. It now strips the path from domain names before publishing them to the load balancers
• Our load balancer monitoring now checks the actual listening ports, so a main process with dead workers can no longer report as healthy

In progress:

• Update Sōzu to reject a malformed frontend hostname as an invalid configuration change instead of panicking: sozu-proxy/sozu#1312

Longer term:

• Per-region routing configuration propagation with a canary region, to reduce the blast radius of faulty updates

Conclusion

A single malformed domain name stopped traffic in every public region for 19 minutes, because two layers that should have contained it did not: the control plane published the entry without validating it, and Sōzu crashed on it instead of rejecting it. We have fixed the component at fault, closed the monitoring blind spot that hid the crash loop, and are hardening Sōzu and our propagation model so that a bad configuration entry stays a local, recoverable event. We apologize to our customers for the disruption.