FoundationDB made this idea famous. Instead of relying only on unit tests and real-cluster end-to-end tests, it runs distributed workloads inside a simulated world where time, network faults, process failures, and scheduling decisions can be explored and reproduced from a seed. TigerBeetle pushed a complementary discipline: make invariants part of the code, assert the shape of the state continuously, and turn silent corruption into loud local failures.

At Clever Cloud, we have been applying that direction to Rust infrastructure software. Moonpool, developed by Pierre Zemb, gives Rust systems a deterministic simulation environment where time, networking, tasks, randomness, and storage can be virtualized behind provider traits. Magnetar is where we apply that approach to a client for Apache Pulsar. This is also the approach that we are taking in Sōzu.

Magnetar is a from-scratch Apache Pulsar client driver in Rust. It is built around a sans-io protocol core, a production Tokio engine, and a Moonpool engine for deterministic simulation. This first public release is meant for engineers who want to try Magnetar, evaluate the architecture, compare behavior with existing Pulsar clients, and help shape a stronger Rust-native Pulsar ecosystem.

The goal is not only to expose a Rust API around Pulsar. The goal is to build a client whose difficult behavior can be inspected, replayed, and made to fail before production gets the chance.

A Pulsar client is part of the distributed system

It is tempting to think about a client library as the easy side of a distributed system. The broker is where replication, storage, ownership, and coordination live. The client only connects, sends messages, receives messages, and acknowledges them.

That view does not survive contact with a real Apache Pulsar client.

A serious Pulsar client has to manage producers, consumers, readers, partitioned topics, multi-topic consumers, pattern subscriptions, transactions, schemas, authentication, proxy routing, broker lookup, cluster failover, reconnect, backoff, batching, chunking, ack grouping, unacked-message tracking, negative acknowledgements, retry-letter and dead-letter flows, observability, and a long list of Pulsar Improvement Proposals.

Many of the hard bugs are not about one command being encoded incorrectly. They are about time and ordering. A broker drops while a publish is in flight. A reconnect happens while a consumer has pending acknowledgements. A token refresh races with a session reset. A proxy lookup points to a new broker while the old connection is still draining. A topic migrates and the client must rebuild enough state to continue without pretending the old session still exists.

That means a Pulsar client is not just a convenience wrapper around a socket. It is part of the distributed system. It owns state that has to remain coherent while the world around it changes.

If that state is hidden inside runtime tasks, channel queues, implicit clocks, and timing accidents, then the client becomes hardest to reason about exactly when precision matters most. Magnetar starts from the opposite assumption: make the protocol state explicit, drive it through narrow boundaries, and keep the I/O outside.

From the Rust Pulsar ecosystem to Magnetar

Magnetar comes from experience with the Rust Pulsar ecosystem, including maintaining and using pulsar-rs. That work matters. It gave Rust users an Apache Pulsar client and created the practical context in which the next generation of driver work could be evaluated.

Magnetar is not an article-length criticism of what existed before it. It is a continuation of the same ecosystem from a different architectural starting point. After spending time with real Pulsar usage, the shape of the next step became clearer: a Rust Pulsar driver should make the protocol/runtime boundary sharper, make difficult behavior reproducible, track parity explicitly, and make testability a first-class design constraint.

That is why Magnetar does not begin with the public builder API. It begins with the protocol.

The user-facing API matters, and the first release already exposes a broad surface. But for a distributed client, the deeper question is what happens underneath that API when the broker moves, the connection drops, the clock advances, a timeout fires, a subscription is rebuilt, or the same trace is replayed under a different runtime.

The sans-io core

The heart of Magnetar is magnetar-proto. It is a synchronous protocol state machine. It has no sockets. It does not depend on Tokio. It does not spawn tasks. It does not use async. It does not read the host clock on the protocol hot path.

Its shape follows the same broad idea that made quinn-proto influential in the Rust networking world: feed bytes in, poll bytes out, poll semantic events, and ask the state machine when its next timer expires.

Conceptually, the interface looks like this:

> connection.handle_bytes(now, bytes);

connection.poll_transmit(&mut out);
connection.poll_event();
connection.poll_timeout();

That small shape is a large architectural choice.

handle_bytes lets an engine feed wire bytes into the state machine. poll_transmit drains outbound frames. poll_event yields semantic events that the runtime must react to, such as authentication challenges, lookup outcomes, checksum mismatches, reconnect signals, or protocol-level transitions. poll_timeout lets the runtime ask when to wake the state machine again.

The state machine does not decide where bytes come from. It does not know whether the network is a real TCP stream, a TLS stream, a byte pipe inside a simulator, or a scripted broker in a differential harness. It does not decide what "now" means. The engine passes time in.

The public traits in magnetar-proto follow the same rule. They are not runtime services hidden behind async callbacks. They are small protocol contracts at the points where the state machine needs a decision or a transformation: AuthProvider produces authentication bytes, Schema defines encoding, decoding, schema data, and broker-resolved schema hooks, MessageEncryptor and MessageDecryptor describe payload crypto over bytes and message metadata, ServiceUrlProvider exposes the current service URL for failover, and HealthProbe uses a poll-style contract instead of binding the protocol crate to an async runtime. The pattern is deliberate: the protocol core can ask for facts, bytes, or readiness, while the I/O, caching, refresh logic, network probes, and scheduler stay outside.

This separation is not just a clean layering exercise. It is what makes the protocol testable without a broker, without a socket, and without an async runtime. A protocol bug can be reduced to a fixture: feed these bytes at this instant, poll these outputs, assert this event stream, and replay the same sequence again.

For an Apache Pulsar client, that matters because the wire protocol is not only a happy path. It includes producer creation, consumer subscription, acks, nacks, transactions, broker errors, redirects, authentication challenges, topic migration, replicated-subscription markers, schema metadata, batching, chunking, and close paths. Each of these has state. Each state transition is easier to reason about when it is not mixed with socket reads, task scheduling, and runtime-specific wakeups.

No channels as an architectural constraint

Magnetar also makes an unusual Rust async choice: channel primitives are banned from the workspace.

The straightforward architecture for a network client is often producer future to channel, channel to driver task, driver task to channel, channel to consumer future. That can work, but it spreads state across queues and tasks. Backpressure becomes implicit. Close semantics become library-specific. Dropped futures can leave messages stranded in places that are hard to inspect. The debugging question becomes "where did this message go?"

Magnetar uses a different pattern. State lives inside the protocol state machine and the runtime-owned shared connection. User-facing futures register wakers in slabs keyed by operation identifiers. The driver owns the I/O loop, feeds bytes into the state machine, drains outbound frames, and wakes the matching futures as outcomes arrive.

This is not an aesthetic ban on channels. It is a way to keep ownership of state explicit. When the driver rebuilds producers after a reconnect, when a consumer waits for a message, when an ack outcome is delivered, or when a pending operation fails because a session is gone, the relevant state is in the state machine and can be inspected there.

That design has trade-offs. It puts more pressure on the correctness of the state machine. It requires careful lock ordering. It requires reviewers to understand waker registration and wakeup paths. But it also makes the behavior much easier to simulate, replay, and compare across runtimes.

Two engines, one state machine

Magnetar ships two engines that drive the same protocol core.

The default engine is magnetar-runtime-tokio. It is the production execution path: TCP, TLS through tokio-rustls, driver tasks, real time, real sockets, and the public PulsarClient surface most users will reach first.

The second engine is magnetar-runtime-moonpool. It drives the same magnetar-proto::Connection state machine through Moonpool providers. Instead of baking in the runtime, Moonpool exposes provider traits for networking, time, task spawning, randomness, and storage. Under a production-style provider bundle, the engine can run against real I/O. Under simulation providers, those same categories can be virtualized.

This is where deterministic simulation becomes practical. The client can be driven in a world where time advances under control, network delivery can be reordered or interrupted, tasks run under a deterministic scheduler, randomness is seeded, and failures can be replayed from the same seed.

Moonpool is not a mock. A mock usually replaces a component with simplified behavior. That is useful for many unit tests, but it is not the same thing as exercising a real client through hard interleavings.

Moonpool is also not a normal end-to-end test. An e2e test runs the real system, which is essential, but it usually runs with real time, real networking, and failures that are difficult to reproduce once the timing has moved on.

Moonpool sits in a different space. It keeps the client code close to the real execution path while virtualizing the providers that make distributed behavior hard to reproduce. The point is not to avoid e2e tests. The point is to add a regime where failures are not anecdotes. They become seeds.

Magnetar's Moonpool engine uses the same protocol state machine as the Tokio engine. The TLS path is also driven in a way that preserves the simulation boundary: instead of relying on tokio-rustls, the Moonpool engine drives rustls::ClientConnection over the byte pipe directly. That keeps TLS handshakes under the same deterministic control as the rest of the simulated network.

Differential testing: making the engines agree

Two engines are useful only if they do not silently become two different clients.

Magnetar therefore includes a differential harness. It runs a trace, such as connect, open producer, send, subscribe, receive, ack, seek, and close, against both the Tokio and Moonpool engines, then compares the user-visible event streams.

This matters because the Moonpool engine is used to find bugs that are difficult to expose with real time and real sockets. For that to be meaningful, the simulated engine must remain observationally aligned with the production engine at the user boundary.

If Tokio and Moonpool drift, the simulator becomes less trustworthy. If the differential harness keeps them aligned, then a seed failure in the simulated world is much more likely to point at a real property of the client, not a separate test-only implementation.

This is also why Magnetar's validation policy is strict about cross-runtime coverage. Behavioral changes are expected to exercise the sans-io layer, the Tokio runtime, the Moonpool runtime, differential equivalence, and the e2e surface when applicable. That is more work than adding one unit test. It is also the cost of making deterministic simulation a real engineering tool rather than a side experiment.

Java parity as a credibility contract

Simulation is not a substitute for feature breadth. It is what makes feature breadth less fragile.

Apache Pulsar has a mature Java client, and for many users that client defines what "a Pulsar client" means. Basic produce and consume are not enough. A credible client must cover the behaviors applications rely on: batching, compression, transactions, schemas, partitioned topics, multi-topic and pattern consumers, retries, dead-letter flows, authentication providers, proxy support, failover, admin operations, and the PIPs that shape modern Pulsar usage.

Magnetar tracks Java-client parity as a public contract. The README carries the parity matrix and the engine-by-engine status. The article does not need to reproduce that matrix, but the matrix itself is important. It gives users a way to evaluate the project as a Pulsar client, not as a narrow experiment around one happy path.

This is the reason Magnetar's first public release is broader than a minimal demo. It includes the protocol foundation, the public facade, runtime engines, admin support, authentication providers, message crypto, CLI work, observability hooks, and a growing list of PIP surfaces.

The exact reason for tracking parity is simple: users do not choose a client library only because its internal architecture is elegant. They choose it because it can represent the system they actually run. Architecture makes that surface maintainable. Parity makes it useful.

Beyond produce and consume

The phrase "Pulsar client" can hide a lot of surface area. From the outside, the first examples usually look small: build a client, create a producer, send a payload, subscribe with a consumer, receive a message, acknowledge it. Those examples are useful because they show the entry point. They are not enough to evaluate the client.

Real Pulsar applications depend on behaviors that only appear after the first demo works. A producer may need batching, compression, chunking, access modes, sequence ids, send timeouts, per-message properties, transactions, interceptors, and latency statistics. A consumer may need batch receive, cumulative acknowledgements, batch-index acknowledgements, nack backoff, ack timeout handling, retry topics, dead-letter topics, seek, pause/resume, subscription modes, and per-partition behavior. Applications with strict schemas need Avro, JSON, Protobuf, Protobuf-native, KeyValue, primitive schemas, and broker-side schema lookup semantics. Deployments with security requirements need token auth, mTLS, OAuth2, SASL, Kerberos/GSSAPI, Athenz, TLS provider choices, and a way to reason about crypto behavior without making every build drag every provider.

Those features are not independent checkboxes. They interact. Batching interacts with compression and send timeouts. Chunking interacts with sequence ids and redelivery. Transactions interact with acknowledgements. Reconnect interacts with producers, consumers, lookups, proxy routing, and failover. Observability has to describe the behavior without leaking secrets or flooding operators under churn.

That is why Magnetar's feature work and architecture work are tied together. The more complete the surface becomes, the more important it is that the protocol state is explicit and that the runtime boundary is narrow. A broad client built on accidental state becomes harder to maintain as it becomes more useful. A broad client built on replayable state gives maintainers a better chance to keep feature growth from turning into hidden behavior.

Validation as architecture

Magnetar's test strategy is intentionally layered because each layer catches a different class of bug.

The sans-io layer lets protocol behavior be tested without sockets or async tasks. The project testing documentation currently records more than 270 unit tests in magnetar-proto. Those tests exercise the state machine directly: bytes in, events out, transmit buffers out, state transitions checked.

Runtime integration tests exercise the glue that pure protocol tests cannot cover. The Tokio engine and Moonpool engine each have their own integration surfaces, and the project enforces runtime parity so one engine does not quietly fall behind the other.

The deterministic chaos pack targets the failure schedules that motivated the architecture in the first place: reconnect, failover, virtual-clock timers, TLS handshake ordering, in-flight publishes, broker migration, and adversarial network behavior. The Moonpool engine documentation describes a daily Moonpool seed sweep with 128 freshly rolled random seeds in parallel. The number is less important than what it represents: the client is being driven through schedules that a human would not write by hand.

Differential equivalence tests compare the two runtime worlds. They ask a direct question: if the same trace is run through Tokio and Moonpool, does the user see the same behavior?

End-to-end tests still matter. They keep Magnetar honest against a real Apache Pulsar broker. Simulation can make failures reproducible, but it does not replace compatibility with the system users actually deploy.

Taken together, these layers are not ceremony. They are part of the architecture. The point is not to collect large test numbers. The point is to put validation pressure exactly where distributed clients fail: protocol state, runtime boundaries, reconnect logic, timing, equivalence, and real broker behavior.

What reproducibility changes in practice

The practical value of deterministic simulation is not that it makes tests more sophisticated. It changes the debugging loop.

When a traditional end-to-end test fails because of timing, the first question is often whether the failure is reproducible at all. If it is not, engineers start adding sleeps, logs, larger timeouts, or retry loops. Some of those changes are useful. Many of them only move the failure somewhere else. The failure remains a story: "we saw it once in CI".

With a deterministic simulation, the failure should become an artifact. The artifact is a seed, a trace, and a state transition that can be replayed. The question changes from "can we make it happen again?" to "what invariant did this schedule violate?"

That difference matters for a client library. Client bugs often sit at the boundary between user code and the broker. They are easy to misattribute. A publish can fail because the broker closed the connection, because the client lost an outcome, because the reconnect path rebuilt the producer too late, because an ack was delivered to the wrong waiter, because a timeout fired against stale session state, or because the runtime task died after the protocol had already transitioned.

If all of that behavior is spread across tasks, channels, timers, and sockets, a failure report is hard to compress. If the protocol state machine is explicit, the runtime boundary is narrow, and the same sequence can be replayed under controlled providers, the report can become much smaller: this seed, this trace, this event order, this invariant.

That is why Magnetar treats reproducibility as part of the client design. The point is not only to catch more bugs. The point is to make the bugs that do appear cheaper to understand and safer to fix.

It also changes code review. A reviewer can ask whether a behavior is covered at the right layer. Is this a pure protocol transition? Then the sans-io test should feed bytes and assert events. Is it runtime glue? Then Tokio and Moonpool need equivalent coverage. Is it an observable behavior shared by both engines? Then the differential harness should compare the event streams. Is it broker compatibility? Then an e2e test should touch a real Pulsar broker.

Those questions are more precise than "did we add tests?" They connect the test to the failure model.

What to inspect first

For engineers reading Magnetar for the first time, the best entry point is not a single file. It is the relationship between the documents and the code.

The README gives the public surface: what the client exposes, which features are implemented, how the Java-client parity matrix is tracked, which PIPs are supported, and how the two engines are positioned. That is the user-facing contract.

The architecture document explains the shape underneath that contract. It shows the crate topology, the dependency direction, the sans-io boundary, the driver loop, the event model, the producer and consumer paths, the runtime engines, TLS sites, schemas, PIP coverage, and the validation strategy. That is the system map.

The ADR series explains why the system has this shape. ADR-0004 records the sans-io split. ADR-0003 records the no-channels rule. ADR-0011 records clock injection. ADR-0010 records the Java parity decision. ADR-0019 records how the production Tokio engine and the Moonpool engine relate to the parity goal. ADR-0024 records the cross-runtime testing policy.

This matters because Magnetar is intentionally not only a code drop. It is a set of engineering decisions with a public audit trail. If you disagree with the design, the ADRs give you the right surface to discuss. If you want to contribute a feature, the parity matrix and validation policy tell you what "done" means. If you want to evaluate the simulation work, the Moonpool engine documentation and the differential harness show how the same protocol core is driven in different worlds.

The first release should therefore be read in two ways. As a Pulsar user, look at the client surface and the parity matrix. As a Rust infrastructure engineer, look at the state-machine boundary and the simulation strategy. The interesting part of Magnetar is that both views are meant to reinforce each other.

What the first public release gives you

This first public release is a technical entry point. It is for engineers who want to try Magnetar, read the architecture, inspect the ADRs, compare behavior with existing Pulsar clients, and help shape the Rust-native Pulsar ecosystem.

The high-level client API starts with the usual shape:

let client = PulsarClient::builder()
    .service_url("pulsar://localhost:6650")
    .build()
    .await?;

let producer = client
    .producer("persistent://public/default/orders")
    .create()
    .await?;

That API is important because the client must be usable. But the release is also an invitation to inspect the lower layers. Read the README for the feature matrix. Read the architecture document for the sans-io design. Read the ADRs for the decisions behind no channels, swappable engines, clock injection, Java parity, Moonpool parity, TLS provider choices, and cross-runtime validation. Read the Moonpool engine documentation if you are interested in how deterministic simulation can be applied to a real network client.

The important point for this article is the direction: Magnetar is public so engineers can evaluate both the client surface and the engineering model behind it.

The inaugural technical entry point is available via v1.0.0, while the current state of the driver can be tracked in the v1.2.0 release. 

Why this matters beyond Magnetar

Magnetar is an Apache Pulsar client. It is also a concrete example of how we want to build more Rust infrastructure software.

Distributed systems fail through state transitions. Some of those transitions are obvious and easy to test. Many are not. They happen when time advances in an unlucky place, when a retry overlaps with a close path, when a broker drops a connection after accepting part of the session, when a callback is registered just after a notification, or when a reconnect path rebuilds one handle but not another.

You can try to chase those bugs after they happen. Sometimes that is unavoidable. But the architecture can make the chase easier or harder.

A client with a pure protocol core, explicit events, injected time, replaceable I/O, deterministic seeds, and cross-runtime comparison gives engineers better tools. It lets them reduce a failure to a trace. It lets them replay that trace. It lets them ask whether a simulated failure corresponds to the production runtime. It lets them add an invariant where a silent state drift used to hide.

That is the direction Magnetar takes. Make the protocol explicit. Make the runtime replaceable. Make failures reproducible. Make correctness something the code has to demonstrate continuously.

Thank you to Pierre Zemb for Moonpool and for pushing this deterministic-simulation direction in Rust, and to the contributors and maintainers in the Rust Pulsar ecosystem whose work made the next step possible.

Magnetar is open source. The code, architecture notes, ADRs, and documentation live at github.com/CleverCloud/magnetar.

References

Magnetar

• Magnetar repository. Public Rust Apache Pulsar client repository. https://github.com/CleverCloud/magnetar
• Magnetar README. Public feature surface, Java client parity matrix, supported PIPs, engine coverage, and status. https://github.com/CleverCloud/magnetar/blob/main/README.md
• Magnetar architecture. Sans-io rationale, crate topology, driver loop, protocol state machine, and tests. https://github.com/CleverCloud/magnetar/blob/main/ARCHITECTURE.md
• Magnetar documentation index. Moonpool engine, testing, CLI, observability, logging, PIP features. https://github.com/CleverCloud/magnetar/blob/main/docs/README.md
• Moonpool engine documentation. Deterministic-simulation engine, chaos pack, differential harness, provider model. https://github.com/CleverCloud/magnetar/blob/main/docs/moonpool-engine.md
• Testing documentation. Test categories and validation commands. https://github.com/CleverCloud/magnetar/blob/main/docs/testing.md
• ADR-0004. Sans-io magnetar-proto and swappable I/O engines. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0004-sans-io-protocol-core.md
• ADR-0010. Full Java-client parity scope. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0010-v0-1-full-java-parity.md
• ADR-0024. Cross-runtime test and coverage policy. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0024-cross-runtime-test-and-coverage-policy.md

Ecosystem

• pulsar-rs. Existing Rust Apache Pulsar client project. https://github.com/streamnative/pulsar-rs
• Apache Pulsar. Distributed messaging and streaming platform. https://pulsar.apache.org/
• Apache Pulsar Java client. Reference client surface for many Pulsar users. https://pulsar.apache.org/docs/

Deterministic simulation

• FoundationDB testing. Deterministic simulation and fault injection lineage. https://apple.github.io/foundationdb/testing.html
• TigerBeetle TigerStyle. Assertion-first engineering style. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md
• TigerBeetle VOPR. Simulation testing approach. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/internals/vopr.md
• Moonpool repository. Deterministic simulation for Rust systems, developed by Pierre Zemb. https://github.com/PierreZ/moonpool
• moonpool-sim. Simulation engine crate for Moonpool. https://crates.io/crates/moonpool-sim

It is a function distinct from containerization itself, which packages an application and its dependencies into a standard format. The differences between Docker and Kubernetes rest precisely on that distinction: Docker builds and runs containers, Kubernetes orchestrates them. Today Kubernetes is the reference orchestrator, but orchestration as a concept exists independently of the tool that implements it.

Why orchestration exists

Running three containers on a single server does not call for orchestration. You start them, you watch them by eye, you restart one by hand if something crashes. The nature of the problem changes the moment an application becomes a distributed system: several services, several machines, variable traffic, frequent deployments. The operational questions that appear then are concrete. Which machine should this new container run on? What happens if a node goes down in the middle of the night? How do you move from version 1.4 to 1.5 with no visible interruption? How does one internal service find another when IP addresses change on every restart?

Each of these questions has a possible manual answer, one you can even script. Orchestration is about automating them coherently, under a single model.

The five core functions of orchestration

Kubernetes handles these questions through adeclarative model built on reconciliation loops: you describe the desired state of the system, and the orchestrator constantly compares that desired state to the actual state to bring them together. The five functions that follow are instances of this general mechanism.

Scheduling: placing containers on the right machines

Scheduling answers one question: when a new container needs to start, which machine in the cluster should run it? On a cluster of a few nodes with identical profiles, the decision is trivial. On a real cluster, it is not: some machines have GPUs and others do not, some are already loaded, some applications must be isolated while others must instead be co-located for latency reasons.

The kube-scheduler component makes this choice in two steps: filtering, which rules out the nodes that cannot host the pod (insufficient capacity, affinity constraints, taints and tolerations), and scoring, which ranks the remaining nodes against several criteria. The pod is then assigned to the chosen node through a binding operation with the API server. The quality of the decision depends on the information given to the scheduler. It is the resource requests declared on the pods that drive placement: the scheduler looks for a node whose available capacity covers those requests. The limits, on the other hand, play no part in placement; they cap a container’s consumption once it is running.

Scaling: matching capacity to real traffic

Scaling automates a decision that is simple to state: how many copies of a service should run at a given moment? Too many copies cost money. Too few cause incidents at the first spike.

Scaling is a function common to every orchestrator; the mechanisms vary by implementation. In Kubernetes, it rests on three distinct building blocks. The Horizontal Pod Autoscaler (HPA), part of core Kubernetes, adjusts the number of pods in a deployment based on a metric. To do so it relies on the Metrics Server, a component you install separately, because metric collection is not provided by default. The Vertical Pod Autoscaler (VPA), also shipped as a separate project, adjusts the resources allocated to each pod. The Cluster Autoscaler, finally, adds or removes whole nodes according to aggregate needs. These mechanisms assume reliable metrics: an HPA driven by CPU, when the service is actually constrained by disk I/O, will not produce the expected result.

Self-healing: replacing what fails

Past a certain number of machines, failures stop being exceptional events and become background noise. A disk fails, a kernel panic happens, a container leaks memory and ends up OOM-killed. Without orchestration, every failure triggers human intervention, more or less urgent. With orchestration, these events are absorbed without intervention.

Within Kubernetes, the mechanism rests on two elements. First, probes (liveness, readiness, startup) that let it know whether a container is actually working, and not merely whether it has started. Second, the principle that any object managed by a controller (Deployment, StatefulSet, DaemonSet) is constantly compared against the desired state: if a pod disappears, the controller requests a new one. The accuracy of the probes determines the quality of self-healing: probes that are too strict cause needless restarts, probes that are too lax let broken containers through.

Service discovery and internal load balancing

In a moving fleet of containers, where each pod has an ephemeral IP address, two services that need to communicate cannot rely on static IP configurations. Service discovery solves this through a layer of indirection. In Kubernetes, a Service object groups a set of pods (selected by labels) under a stable DNS name and a virtual IP. CoreDNS, the cluster’s internal DNS server, resolves these names. Traffic sent to the Service IP is distributed across the ready pods via kube-proxy. This principle is not specific to Kubernetes: other orchestrators meet the same need differently. On Clever Cloud, for instance, services find each other through configuration injection between applications and through Network Groups, an encrypted private network with internal DNS resolution.

Several types of Services exist (ClusterIP for internal traffic, NodePort and LoadBalancer for external traffic), alongside the notion of Ingress for application-level HTTP routing. This network layer, simple in appearance, holds a substantial share of the operational complexity of Kubernetes: CNI plugin choice, network policies, east-west traffic observability.

Rolling updates and rollbacks

Shipping a new version to production without interrupting the service is a risky operation when done by hand. Kubernetes treats it as a state transition: you change the Deployment manifest to point to the new image version, and the controller applies the configured rollout strategy.

The RollingUpdate strategy (the default) progressively replaces old pods with new ones, guaranteeing that a minimum number stays available at all times. The Recreate strategy stops everything and then restarts, useful for incompatible schema migrations. In case of failure, the rollout does not revert on its own to the previous state: it stops, the new pods that fail their probes do not replace the old ones, and it is an operator who triggers the return to the previous version with kubectl rollout undo. For this to work, the versions must be compatible with each other during the transition, which implies discipline on API contracts and database migrations.

Beyond the five core functions

Kubernetes orchestration does not stop at these five functions. In practice, a production Kubernetes cluster also manages application configuration (ConfigMaps), secrets (with or without encryption at rest), persistent storage through PersistentVolumeClaims, authorization (RBAC), network isolation (NetworkPolicies), observability (logs, metrics, traces), and certificate lifecycle management. Each of these is itself an operational responsibility that adds to the base.

Without an orchestrator vs with Kubernetes: a function-by-function comparison

The table does not rank the two columns. It shows that Kubernetes provides a unified model for these five functions, where an approach without an orchestrator handles them separately with different tools. For a system that does not need this unification, the left-hand column remains perfectly valid.

Common practices worth knowing

Choose the tool based on the need, not by default. Not every application needs scheduling, automatic scaling or service discovery at the level Kubernetes offers. A PaaS, a VM deployment, or a systemd binary meet many needs with a different operational model.

Think about observability from the start. A Kubernetes cluster without centralized logs, aggregated metrics and traces quickly becomes hard to operate beyond a handful of services. Observability is an integral part of orchestration, not an optional add-on.

Define requests and limits before enabling autoscaling. The HPA and the Cluster Autoscaler rely on the pods’ declared resources to decide on scaling. Without declarations, their decisions rest on partial information.

Get probe configuration right. Liveness and readiness probes govern both self-healing and traffic routing. A rough configuration degrades both functions at once.

When Kubernetes orchestration becomes relevant

No single one of the five functions, taken in isolation, justifies bringing Kubernetes into a system. A deployment script, a properly configured load balancer, a monitoring tool with automatic restart can each cover one of them individually. What justifies adopting an orchestrator is the point where the five functions all become necessary at once, on an infrastructure distributed enough to make ad hoc solutions expensive to maintain.

Concretely, that corresponds to architectures with a dozen services or more, deployed across several machines, with frequent deployment cycles, significant load variations and resilience requirements that no longer tolerate manual intervention. For contexts more constrained in resources (edge, IoT, development machines, small clusters), a lightweight Kubernetes distribution such as K3s can be a good fit. A PaaS also meets these needs, as it does many other application profiles (industrialized deployment, automatic scaling, resilience, updates), with an operational model distinct from Kubernetes.

When Kubernetes does become relevant, the next question rarely concerns installation and more often day-to-day operations: control plane updates, etcd management, certificate rotation, observability, backups. That observation is what explains the growing adoption of managed Kubernetes services. At Clever Cloud, Clever Kubernetes Engine addresses this need with Materia etcd, a serverless implementation of the etcd API built on FoundationDB that takes over from standard etcd, a known bottleneck at scale, on sovereign infrastructure spread across our three Paris datacenters.

Container orchestration in summary

Container orchestration automates several categories of operational decisions, chief among them five main classes: where to place a container, when to start more of them, what to do when one of them fails, how they find one another, and how to move from one version to the next without interruption. Kubernetes brings these functions together under a unified model, which makes it the reference tool for distributed systems that need them all at once. The useful question, faced with a project, is not whether you should do orchestration, but which of these functions are actually needed, and at what level of automation.

FAQ - Kubernetes orchestration

What is the difference between containerization and orchestration?

Containerization packages an application and its dependencies into a standardized format that runs identically on any compatible host. Orchestration automates the management of a fleet of containers in production: placement, scaling, self-healing, service discovery, updates. Docker is the reference tool for the former, Kubernetes for the latter.

Do you necessarily need Kubernetes to orchestrate containers?

No. Kubernetes is the most widely adopted orchestrator, but others exist: HashiCorp’s Nomad, which orchestrates containers, VMs and binaries in a single cluster, or Docker Swarm, still maintained but with declining adoption. Apache Mesos, long cited as an alternative, was retired to the Apache Attic in October 2025 and is no longer in active development. Beyond that, a PaaS like Clever Cloud covers the full set of these orchestration functions (placement, automatic scaling, self-healing, rolling updates, service discovery) with its own control plane, independent of Kubernetes. It is that control plane that orchestrates both the applications running in VMs and the containers of the platform’s Docker runtime.

Is container orchestration necessary for CI/CD?

No, but the two are often used together. Modern CI/CD pipelines produce container images as their final artifact, and the orchestrator (Kubernetes or another) takes over to deploy and operate those images. CI/CD and orchestration are two complementary stages of the application lifecycle.

Does K3s allow orchestration in constrained environments?

Yes. K3s is a CNCF-certified Kubernetes distribution designed for resource-limited environments (edge, IoT, development machines, small clusters). It keeps the standard Kubernetes APIs while reducing the resource footprint and deployment complexity.

Does managed Kubernetes change the orchestration functions available?

No. Managed Kubernetes provides the same set of orchestration functions as a self-hosted cluster, since it is the same Kubernetes. The difference lies in how operational responsibilities are split: the provider takes on control plane management, updates, and the underlying monitoring. Users then focus on their application workloads.

Which orchestration function is the hardest to master?

Networking is frequently cited as the most complex. Service discovery, internal load balancing, NetworkPolicies, Ingress, CNI plugin, east-west traffic observability: the Kubernetes network layer concentrates a significant share of advanced operational topics and remains a frequent source of production incidents.

This release is a milestone: the underlying machine is now ready for the product features we have been waiting to build on top of it. Rather than a list of patches, we've grouped the work into six themes; for each, we say two things — what shipped, and what it makes possible, for the people running their applications on the platform and for the operators who run Sōzu on their own infrastructure.

1. A from-scratch HTTP/2 multiplexer

Sōzu's HTTP/1 stack had already been rewritten around kawa, our pivot format: a single internal representation of the HTTP message, independent of its protocol version — the same idea as HAProxy's HTX. The HTTP/2 multiplexer is its natural extension: it adds session management and stream multiplexing on top of kawa.

It supports the full protocol matrix (H1↔H1, H1↔H2, H2↔H1, H2↔H2), with shared stream state, HPACK compression via loona-hpack, backend H2 connection pooling, RFC 9218 Extensible Priorities for stream prioritisation, and per-listener ALPN negotiation so each TLS connection lands on the right code path. Around 181 end-to-end tests and two cargo-fuzz targets keep the parser and the HPACK decoder honest — including regression guards that pin large-response integrity byte-for-byte across an H1-backend → H2-frontend path, the exact boundary where edge-triggered epoll readiness bugs hide.

For Clever Cloud customers, the practical impact is simple: every application served by the platform now speaks HTTP/2 by default, on the frontend — no opt-in, no code change, no configuration. Enabling HTTP/2 all the way to the backend stays a per-cluster choice — a cluster, in Sōzu's model, is one of your applications; turned on end-to-end, it unlocks gRPC across the whole chain. Page loads finish in fewer TCP connections; browsers can coalesce requests across hostnames that share a certificate (we honour RFC 7540 §9.1.1 SAN coalescing, so when the certificate, authority and connection conditions line up, Firefox and Chrome can reuse a single connection across your cdn.example.com and assets.example.com instead of opening parallel ones). It is, in the best sense, a quiet platform upgrade.

It is also the structural prerequisite for what comes after — HTTP/3 over QUIC and stronger streaming foundations at the edge. The mux rewrite is the part of the roadmap that had to land first.

2. Security as a baseline, not a toggle

The second half of the H2 rewrite is the part nobody asks for until it's missing: flood and DoS hardening. Sōzu 2.0 ships built-in mitigations for CVE-2023-44487 (Rapid Reset), CVE-2024-27316 (CONTINUATION flood), CVE-2025-8671 (MadeYouReset), and the PING / SETTINGS / empty-DATA flood family from CVE-2019-9512/CVE-2019-9515/CVE-2019-9518. Each mitigation surfaces a dedicated counter — twelve metrics under h2.flood.violation.* — so a SIEM can window the trip rate without parsing logs. Seventeen HPACK rejection reasons are surfaced the same way; that is the first operator-visible signal for request-smuggling probes against the H2 stack.

The rest of the security work fans out across the proxy: a per-(cluster, source-IP) connection cap with 429 Too Many Requests; opt-in eviction of the oldest sessions when the accept queue saturates; hardening across the command channel, the HTTP/1 parser, the pattern-trie router (closing a routing-bypass through unanchored regex) and the wildcard matcher; audit-log sanitisation against Trojan-Source and SIEM column-smuggling; and a TLS certificate hot-rotation that never drops the working certificate on failure, even if the new one is malformed. Four dependency advisories cleared in the same window.

Every privileged mutation now lands in a structured audit log: each control-plane action is recorded as a Command(verb=…, actor_uid=…, actor_user=…, result=…) line — who did what, from where, and whether it succeeded. Two dedicated sinks ship it out: audit_logs_target for the human-readable stream, and audit_logs_json_target for one stable-schema JSON object per line, so the trail tails straight into a SIEM (Wazuh, Elastic, Loki, Splunk) without a bespoke parser — in a shape designed for PCI-DSS 10.5.

The product framing is straightforward: our job is to operate the platform and shield your applications as soon as we can. When the next H/2 vulnerability in this class lands at 9pm on a Friday, it does not have to translate into a patch-and-redeploy weekend across thousands of applications — attacks of this class are largely absorbed or mitigated at the proxy, surfaced as a counter on a dashboard, while applications keep serving traffic. Trust-by-default is not a marketing claim; it is the cumulative effect of dozens of small, defensive fixes shipped at the layer where the security boundary actually lives.

3. Visibility on every layer

The largest single user-facing addition in 2.0 is sozu top — an operator TUI (behind the tui Cargo feature) that gives you a live btop/htop-style view across seven panes: Overview, Clusters, Backends, Listeners, H2, Certificates, Events. Colour-blind-safe palette, customisable themes — the essentials fit in a terminal, no external dashboard required.

Underneath the TUI, the metric surface itself was rewritten. The mux exposes per-frame-type counters and a breakdown by RFC 9113 error code, TLS handshake telemetry, per-status HTTP counters, and new worker lifecycle gauges. The access log gains TLS and forwarding fields (version, cipher, SNI, ALPN, XFF chain), x_request_id propagation end-to-end, and client and server RTT — enough to follow a request from one hop to the next.

A new command, SetMetricDetail, lets an operator raise metric cardinality on demand via a time-bounded lease that expires: production stays low-cardinality by default, and deep inspection becomes a one-off decision rather than a config rewrite.

For Clever Cloud, this is the foundation of the next chapter of customer-facing observability — per-application latency percentiles, per-cluster availability, TLS handshake breakdowns, request IDs you can follow across hops. The metrics now exist at the proxy. The next step is to surface them in the Clever Cloud console where they belong, alongside the build and deploy views — so you do not need an external APM to understand the traffic your application actually sees.

4. Traffic policies that you can finally toggle

Sōzu 2.0 reshapes the entire frontend-policy surface (#1231): typed HSTS (RFC 6797) configurable per listener and per frontend; URL rewrite (host, path, port) with regex capture propagation from the routing trie into rewrite templates; request- and response-header rewriting per frontend — add, set, or delete any header (an empty value deletes it, HAProxy del-header parity), with listener-scoped X-Real-IP injection and anti-spoof elision of client-supplied values on top; HTTP 301 / 302 / 308 redirects through a typed RedirectPolicy enum; and HTTP Basic authentication per frontend, with SHA-256-hashed credentials and a constant-time compare via the subtle crate (the credential boundary is hardened against timing side-channels — the audit caught it at review time).

This is the part of the release where the product opportunity is most visible. Today, doing a clean domain migration on a managed platform typically requires either a backend that knows how to redirect or a separately deployed redirect service; putting a staging URL behind a password typically requires an authentication add-on. In Sōzu 2.0, these become knobs at the proxy. The plumbing exists; what remains is to expose those knobs in the Clever Cloud console — as a checkbox on a domain, or a one-click toggle for preview environments. That is the no-code traffic-control surface we want to build next.

5. Operations as the customer benefit

A rewrite this deep only ships safely if the fleet keeps moving underneath it. Day-to-day operations got a lot of attention in 2.0.

sozu listener {http,https,tcp} update is a new field-masked patch verb that tunes non-bind-only listener settings on a running proxy without cycling sockets — H/2 flood thresholds, SNI binding, ALPN preference, idle timeouts, HSTS, custom answers. CVE mitigations can now be tightened under attack without cycling listener sockets. Active backend health checks run inside the existing mio event loop (no async runtime, no extra threads) with HTTP/1.1 and HTTP/2 probes, jittered intervals, and a fail-open path that routes through Normal-status backends whose retry policy allows it. systemd integration (closes #228) now does the right thing: Type=notify units, READY=1 only after the initial workers spawn and the saved state replays, STOPPING=1 on graceful shutdown, MAINPID=<new> across hot upgrades.

Releases themselves changed shape. Pushing a tag now produces ten pre-built tarballs (three Linux targets crossed with up to four crypto providers), signed keyless via sigstore (cosign + GitHub OIDC), with SLSA build provenance and a SOURCE.txt corresponding-source pointer satisfying AGPL §6 / LGPL §4 — closing the gap noted in #1089. ACME fullchain.pem files now load cleanly even when the client emitted the leaf at the start (Certbot, lego, acme.sh); a six-year-old worker-auto-restart binary race (#515) is fixed by pinning the original inode through /proc/self/fd. The LoadState IPC verb stays forward-compatible with sozu-command-lib 1.1.1 clients, so the ecosystem of integrations does not break on the upgrade.

At Clever Cloud's scale, the cost of an operational incident isn't a ticket — it is compounding latency for thousands of applications, and the engineers' time we owe to building the next thing. Sōzu 2.0 is the proxy that is quiet to operate, and that quietness is what customers experience as uptime. It's also what makes Sōzu credible as something you can run on your own infrastructure: signed binaries, hot reloads, ACME quirks ironed out, systemd integration done properly.

6. Crypto that's ready for tomorrow

The TLS stack got a quieter but equally important rework (#1191). Sōzu now supports four pluggable crypto providers for rustls — crypto-ring (the default), crypto-aws-lc-rs, crypto-openssl, and fips (which implies aws-lc-rs in FIPS mode). All four are exercised by CI, and the precedence chain fips > ring > aws-lc-rs > openssl resolves the active provider deterministically when several features are enabled together.

What we are most happy about, though, is the default groups_list: X25519MLKEM768 is now the first-preference key-exchange group where the provider supports it. That is the post-quantum hybrid being standardised through the IETF (draft-ietf-tls-ecdhe-mlkem) and already registered in the IANA TLS registry; what it means in practice is that an X25519-only client and a PQ-capable client both negotiate the most robust mutually supported exchange, with no operator action.

When the post-quantum migration becomes a regulatory requirement — on a timeline the industry is still debating — the groundwork on Clever Cloud will already be in place: every TLS 1.3 client that offers the hybrid negotiates it today, with no operator action. We made the choice quietly, and we made it the default.

From reverse proxy to programmable edge

Sōzu has always been an infrastructure load balancer: well beyond HTTP, it also balances raw TCP, with a zero-copy forwarding path via splice(2) on Linux for TCP listeners. With 2.0 it moves closer to an API gateway and becomes the substrate of a programmable edge — and "programmable" is the operative word. Each capability described above (HTTP/2 by default, anti-abuse controls, observability, traffic policies, crypto agility, operational quietness) is a knob. The next year of Clever Cloud's roadmap is about exposing those knobs as product features: through the console, through the API, through the workflows you already use to deploy your applications.

This is what we mean when we say Sōzu 2.0 is a first step of something greater. Two-point-zero is not the destination — it is the platform we now build on top of. Managed HTTP/2 everywhere is the first building block. The next is a console where you toggle HSTS, password-protect a preview environment, or redirect a migrated domain. The destination — a fully programmable edge with policy primitives you can compose — is what we are building toward over the coming release cycle.

Two more pieces are already on the bench. First, a UDP load-balancing layer — in the spirit of IPVS, but with Sōzu's hot-reload model — paired with TCP health checks that probe the liveness of the UDP backends; this cements the infrastructure-load-balancer role further. Second, HTTPS-reachable backends, which firm up upstream connectivity and open the road to the API gateway.

Sōzu is open source under AGPL-3.0 (the command library is LGPL-3.0). The 2.0 release binaries are signed via sigstore and ship SLSA provenance; if you operate your own edge, this release is yours to use, audit, and extend. The code, the issue tracker, and the conversations live at github.com/sozu-proxy/sozu.

Thank you to the contributors who made this release happen — and to everyone running Sōzu in production, whose feedback shapes where it goes next.

References

Standards and specifications

• RFC 6797 — HTTP Strict Transport Security (HSTS). https://datatracker.ietf.org/doc/html/rfc6797
• RFC 7540 — Hypertext Transfer Protocol Version 2 (HTTP/2). Now obsoleted by RFC 9113, but §9.1.1 on connection coalescing remains the citation reused by RFC 9113. https://datatracker.ietf.org/doc/html/rfc7540
• RFC 9113 — HTTP/2 (current). §5 streams, §6 frames, §6.8 GOAWAY, §7 error codes, §8.1 HTTP semantics. https://datatracker.ietf.org/doc/html/rfc9113
• RFC 9218 — Extensible Prioritization Scheme for HTTP. https://datatracker.ietf.org/doc/html/rfc9218
• draft-ietf-tls-ecdhe-mlkem — Hybrid key exchange in TLS 1.3: X25519MLKEM768 (IETF Internet-Draft, IANA-registered; the post-quantum hybrid Sōzu prefers by default). https://datatracker.ietf.org/doc/html/draft-ietf-tls-ecdhe-mlkem

CVEs

• CVE-2019-9512 — HTTP/2 Ping Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9512
• CVE-2019-9515 — HTTP/2 Settings Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9515
• CVE-2019-9518 — HTTP/2 Empty Frames Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9518
• CVE-2021-42574 — Trojan Source (bidirectional override). https://nvd.nist.gov/vuln/detail/CVE-2021-42574
• CVE-2023-44487 — HTTP/2 Rapid Reset. https://nvd.nist.gov/vuln/detail/CVE-2023-44487
• CVE-2024-27316 — HTTP/2 CONTINUATION Flood. https://nvd.nist.gov/vuln/detail/CVE-2024-27316
• CVE-2025-8671 — MadeYouReset (HTTP/2). https://nvd.nist.gov/vuln/detail/CVE-2025-8671

Further reading

• Cloudflare Learning Center — What is HSTS?. Accessible primer on HSTS, the max-age / includeSubDomains / preload semantics, and the HSTS preload list policy. https://www.cloudflare.com/learning/ssl/what-is-hsts/
• Cloudflare — HTTP/2 Rapid Reset: deconstructing the record-breaking attack. The canonical write-up of the October 2023 coordinated disclosure, including attack mechanics and the 398 M rps peak captured live. https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/
• HAProxy Technologies — HAProxy is Not Affected by the HTTP/2 Rapid Reset Attack (CVE-2023-44487). HAProxy's structural argument for why their stream lifecycle naturally absorbs Rapid Reset; a useful contrast against Sōzu's flood-counter approach. https://www.haproxy.com/blog/haproxy-is-not-affected-by-the-http-2-rapid-reset-attack-cve-2023-44487
• Cloudflare blog — Post-quantum series. Multi-year coverage by Bas Westerbaan and colleagues on PQ key exchange and hybrid groups; lineage of the X25519MLKEM768 hybrid Sōzu now negotiates by default. https://blog.cloudflare.com/tag/post-quantum/

In short: K3s is a CNCF-certified Kubernetes distribution optimized for constrained environments (edge, IoT, labs). K8s refers to the original Kubernetes project, designed for large-scale production clusters. The choice depends on your available resources, deployment context, and operational workload.

K8s: Standard Kubernetes for Enterprise Environments

K8s is the abbreviation of Kubernetes, with the “8” representing the eight letters between the “K” and the “s”. It is the original open-source project maintained by the Cloud Native Computing Foundation (CNCF) and initially developed by Google.

Kubernetes is designed for large-scale production environments: multi-node clusters, high availability, integration with public clouds (AWS, GCP, Azure), or on-premises datacenters. Its control plane includes several separate components: API server, scheduler, controller manager, etcd, deployed independently, which provides maximum flexibility but requires significant operational expertise.

Typical requirements for a production control plane node: at least 2 vCPUs and 2 GB of RAM for Kubernetes components alone, excluding etcd and application workloads.

K3s: A CNCF-Certified Kubernetes Distribution, Not a Fork

K3s is a certified Kubernetes distribution, not an unofficial lightweight version or a fork. Created by Rancher Labs, it was donated to the CNCF in June 2020 and passes the same Sonobuoy conformance tests as all certified distributions. Any valid Kubernetes manifest works on K3s.

Its main goal is to drastically reduce the resources required to run Kubernetes in constrained environments (edge, IoT, CI/CD, labs) without giving up API compatibility.

What K3s Changes Compared to K8s

• A single binary under 70 MB (supporting x86, ARM64, ARMv7, and S390X), including the containerd runtime, Flannel CNI, a Traefik ingress controller, and a Klipper load balancer.
• SQLite as the default datastore in single-node mode instead of etcd. In high-availability configurations (minimum three server nodes), K3s can use embedded etcd or an external datastore (MySQL, PostgreSQL, external etcd).
• Alpha and beta components removed to reduce the attack surface and memory footprint.

Important point: K3s does not remove etcd; it makes it optional. In single-node mode, SQLite is sufficient. In high availability, embedded or external etcd is supported - although the latter is not officially supported by the K3s team.

Resource Footprint

K3s can run with as little as 512 MB of RAM on an agent node. According to the official documentation (updated May 2026), a server node (control plane) requires 2 GB of RAM and 2 CPU cores, excluding application workloads. A load-tested profile is available in the K3s resource profiling guide. It is worth noting that tests on hardware with 1 GB of total RAM showed instability across K3s, k0s, and MicroK8s when deploying real application workloads (even a lightweight Kubernetes cluster still consumes non-negligible control-plane resources).

Key Technical Differences

Datastore

Runtime and packaging

K8s no longer provides a default runtime since the removal of dockershim (v1.24, 2022). Starting with Kubernetes 1.24, you must install a CRI-compatible runtime (containerd or CRI-O). K3s embeds containerd directly into its binary.

Control Plane Architecture

In K8s, control plane components (API server, scheduler, controller manager) are separate processes. In K3s, they are merged into a single binary, which reduces overhead but limits some advanced isolation configurations.

Scalability

K3s is suitable for moderately sized clusters. In high-availability configurations (3 server nodes, 4 vCPU / 8 GB RAM), the official documentation indicates a capacity of around 1,200 agents. For very large clusters (several thousand nodes), K8s remains the reference.

K3s vs K8s Comparison Table

Use Cases: Which One Should You Choose?

Choose K3s if:

• You deploy on constrained hardware (Raspberry Pi, industrial appliances, edge servers with limited RAM).
• You manage remote IoT or edge clusters, potentially in disconnected environments.
• You need a development or CI cluster that starts quickly, including on modest hardware.
• You want an operational cluster with minimal initial configuration.

Choose K8s (or an enterprise distribution) if:

• You orchestrate hundreds or thousands of production nodes.
• Your workloads require advanced cloud-native integrations (storage, load balancers, IAM) provided by hyperscalers.
• You need alpha or beta API components unavailable in K3s.
• Your organization has a dedicated SRE team operating clusters.

Hybrid Use Cases: K3s and K8s Together

K3s and K8s are not mutually exclusive. Several hybrid architectures are documented in production:

Fleet Management with Rancher

Edge K3s clusters are managed from a central control plane running on K8s or RKE2. The Home Depot (large American retailer with more than 2,300 stores) manages its sites using K3s supervised through Rancher.

K3s for Dev and Staging, K8s for Production

API compatibility guarantees that manifests and Helm charts tested on K3s work in production on an enterprise cluster. This parity reduces surprises when promoting environments.

CI/CD

Test pipelines run on K3s (low cost, fast startup) while production environments use managed K8s clusters.

Managed Kubernetes: A Third Path

Neither K3s nor K8s solves the question of daily operations: updates, certificates, control plane monitoring, or failure management. This is precisely what managed Kubernetes solutions cover.

Hyperscalers (EKS, GKE, AKS) provide this management within their own clouds. But managed solutions also exist outside these ecosystems, especially for organizations that want to retain control over their data and choose sovereign Kubernetes, or even French-operated Kubernetes.

Clever Kubernetes Engine (CKE) is Clever Cloud’s managed Kubernetes service, designed for teams already using Kubernetes and wanting to delegate control plane management (updates, high availability, monitoring) without being constrained to a single hyperscaler. CKE explicitly targets teams that are not fully covered by the traditional  PaaS model (multi-runtime use cases, non-twelve-factor workloads, or the need for granular resource control).

FAQ

Is K3s a Kubernetes fork?

No. K3s is a CNCF-certified Kubernetes distribution. It passes Sonobuoy conformance tests and supports the same APIs as K8s. It is not maintained separately from Kubernetes: it follows upstream Kubernetes releases.

Can K3s Be Used in Production?

Yes, with some nuances. K3s is documented for production workloads in constrained environments (edge, IoT). For large-scale clusters or critical workloads with high SLA requirements, K8s (or an enterprise distribution such as RKE2) is more appropriate.

Does K3s Support Helm?

Yes. K3s includes an integrated Helm controller and is compatible with any valid Helm chart.

What Is the Difference Between K3s and K3d?

K3d is a tool that runs K3s inside Docker containers. It further simplifies the creation of local K3s clusters for development, but it is not intended for production use.

Does K3s Run on ARM?

Yes. ARM64 and ARMv7 are natively supported, which explains its popularity on Raspberry Pi devices and industrial appliances.

Managed Kubernetes vs Self-Hosted K3s: Which One Should You Choose?

Self-hosted K3s gives you full control but makes you responsible for operations (updates, security, high availability). Managed Kubernetes delegates this responsibility to an operator, at the cost of dependency on that provider. The choice depends on your operational resources and control requirements.

Several recent Linux kernel vulnerabilities have required a swift response from infrastructure operators.

Among them, Copy Fail and Dirty Frag drew attention because they involve local privilege escalation scenarios. Copy Fail is tracked as CVE-2026-31431.

Dirty Frag covers two distinct vulnerabilities, CVE-2026-43284 and CVE-2026-43500, tied to Linux kernel components.

At Clever Cloud, we treated these vulnerabilities as critical infrastructure matters. Our goal was twofold: quickly shrink the exposure window, then sustainably improve our kernel selection and deployment process.

This article reviews our approach, the decisions made, and the changes brought to our operations pipeline

Why these vulnerabilities called for a fast response

Copy Fail and Dirty Frag belong to the family of local privilege escalation vulnerabilities. In this type of scenario, an attacker must already be able to execute code locally, but can then attempt to gain higher privileges on the affected machine.

Dirty Frag rests on two Linux kernel flaws.

They notably affect modules related to ESP, used by IPsec, and to RxRPC. On a cloud platform, this type of vulnerability calls for a rapid analysis. The risk is not limited to a single isolated machine.

Scenarios tied to shared environments, containerized workloads, and isolation mechanisms must also be assessed.

What we verified

We analyzed the potential impact of these vulnerabilities on our environments. This step is not just about reading security advisories. It also involves verifying whether a theoretical scenario can become relevant in our operating context.

In the case of Copy Fail, the flaw came under embargo together with its patch. We published a new system image with the patch applied in the days that followed.

Our customers' applications were redeployed shortly after.

In the case of Dirty Frag, our internal analyses confirmed that these vulnerabilities had to be taken seriously. ESP modules are enabled in our kernels to support some specific customer needs. Fortunately, RxRPC-related modules are not present in our environment, as they serve no purpose for our usage.

We do not detail the technical steps of the exploitation here, since the purpose of this article is to inform our customers, not to publish a reproducible procedure.

This validation confirmed the operational decision: handle the matter immediately, reduce the exposed surface, then force the necessary redeployments.

Our operational response

Rolling out immediate measures

We first applied quick measures on the affected kernels. In the case of Dirty Frag, the publicly recommended measures focus in particular on the kernel components related to ESP and RxRPC.

On Clever Cloud's side, the goal was clear: reduce the identified exposed surfaces and shrink the exposure window without waiting for a standard maintenance cycle.

Redeploying the affected workloads

A kernel update only matters if the affected systems actually restart on a patched environment. We therefore launched a progressive redeployment of applications, then handled the cases that blocked this redeployment.

This phase matters. On a managed platform, the fix is not limited to producing an image or compiling a kernel. The execution chain must also actually use the expected version.

Improving the process along the way

We also took advantage of this sequence to replace a temporary mechanism with a cleaner integration into our orchestration pipeline.

Concretely, the kernel choice is now passed more explicitly through our internal pipeline, all the way to Supernova, our hypervisor agent. This evolution replaces the stiffer workaround put in place in the heat of the moment.

That is the central point of this intervention: fix fast, then make the fix more reliable for future operations.

What this changes for Clever Cloud customers

For customers, the expected effect is simple: reduce exposure without any manual action on their part whenever the platform can handle the redeployment.

Clever Cloud runs an architecture that relies in particular on isolation through virtualization. This approach is documented on our security pages and in our technical content on running containers inside virtual machines. It does not eliminate every risk, but it limits certain lateral movement scenarios compared to models where multiple workloads share the same execution environment directly.

We avoid, however, presenting this isolation as an absolute guarantee. A kernel vulnerability must always be taken seriously.

That is why we combined mitigation, redeployment, and improvement of our operations pipeline.

What we take away

This sequence confirms three principles.

First, a kernel vulnerability must be analyzed in its actual operating context. A public alert is not enough. We need to understand whether the conditions required for exploitation can exist on the platform.

Second, reaction speed matters. The Copy Fail and Dirty Frag vulnerabilities were disclosed publicly within a few days of each other, with analyses published by several players in the Linux and cloud ecosystem.

Finally, a useful security response must not only fix the problem of the moment. It must also improve the system that will handle the next incident.

That is what we did here: handled the vulnerabilities, shrank the exposure window, and strengthened our kernel management process.

Q&A

What is a local kernel vulnerability?

A local kernel vulnerability is a flaw that already requires execution capability on the affected machine. It can then allow gaining higher privileges, such as root, if the kernel is vulnerable.

Why do these flaws concern cloud platforms?

Cloud platforms run many workloads with isolation mechanisms. A kernel flaw can become critical if it allows crossing certain boundaries between processes, containers, or execution environments.

Are Dirty Frag and Copy Fail the same vulnerability?

No. Copy Fail is tracked as CVE-2026-31431. Dirty Frag covers CVE-2026-43284 and CVE-2026-43500. These vulnerabilities are close in impact, but they are distinct.

What action is required from Clever Cloud customers?

No general action is required from customers for environments handled by the platform. The automation brought by Clever Cloud allowed everything to be updated without action needed. Specific cases are tracked individually.

How Kubernetes Works: a Declarative Orchestrator

Most descriptions of Kubernetes list its components (pods, deployments, services) without explaining the central mechanism. The fundamental concept lies elsewhere: Kubernetes is first and foremost an orchestrator, and its core engine relies on a reconciliation loop.

You don't tell Kubernetes what to do. You tell it what you want your system to look like. This distinction, declarative versus imperative, changes everything.

In practice, you describe the desired state in manifest files, typically in YAML format: "I want 3 replicas of this image, exposed on port 80, with these environment variables." You submit this manifest to the Kubernetes API via kubectl. From that point on, Kubernetes continuously compares the actual state of the cluster (what is actually running) to the desired state (what you declared), and acts to bring them into alignment. If a node dies, its pods are rescheduled elsewhere. If you change from 3 to 10 replicas in the manifest, Kubernetes starts 7 more. If a container crashes, it gets restarted.

This reconciliation loop is the heart of everything Kubernetes does: self-healing, scaling, rolling updates, and rollbacks. To dive deeper into this mechanism and the other capabilities it enables, learn more about container orchestration.

Architecture in Brief: Control Plane, Nodes, Pods

A Kubernetes cluster is divided into two parts. The control plane is the brain: it makes global decisions, accepts API requests, schedules workloads, and monitors the state of the cluster. It relies on a few key components, including the API server (kube-apiserver), the scheduler (kube-scheduler), and the controller manager (kube-controller-manager), along with a distributed data store that holds the entire cluster state, traditionally etcd.

Nodes are the machines that actually run the workloads. On each node, an agent called kubelet receives instructions from the control plane and launches containers through a container runtime (containerd, CRI-O, etc.). The smallest deployable unit is not an individual container but a pod: one or more containers that share a network and storage. Higher-level objects (Deployment, Service, Ingress, ConfigMap, Secret) describe how pods should be managed, exposed, and configured.

This architecture is also the source of a common confusion with Docker, whose role is actually complementary to Kubernetes rather than competitive.

Why K8S Became the Orchestration Standard

According to the CNCF Annual Cloud Native Survey 2025, published in January 2026, 98% of surveyed organizations have adopted cloud native techniques, and 82% of container users deploy Kubernetes in production, up from 66% in 2023. The dominance is massive. But explaining it solely through technical qualities would be incomplete; it is also a story of ecosystem dynamics and aligned interests.

On the technical side, three properties explain adoption. First, the declarative model described above: it makes deployments reproducible, versionable in Git, and resilient to failures. Second, portability: the same manifest works on a development machine (Minikube, kind, k3d), on an on-premise cluster, and on any cloud. Third, extensibility: the Kubernetes API accepts Custom Resource Definitions (CRDs) and custom controllers (Operators), turning it into a platform for building platforms. This triggered a massive ecosystem dynamic.

On the strategic side, the story is less often told. By 2014, Google had more than a decade of experience managing containers at scale with its internal systems Borg and Omega, whose design principles were shared publicly through academic research papers (Omega in 2013, Borg in 2015). Rather than open-sourcing Borg itself, which remained tightly coupled to Google's proprietary infrastructure, the team created Kubernetes as a new project inspired by that experience, with a distinct implementation designed from the outset for external adoption. The project was released as open source in June 2014 and donated to the Cloud Native Computing Foundation in 2015. This neutrality, a project hosted by a Linux foundation rather than a cloud provider, proved decisive. No competitor could afford not to adopt it without being marginalized from the emerging cloud-native ecosystem. AWS, which initially pushed its own proprietary solution (ECS), announced EKS in late 2017 and launched it in general availability in June 2018. By then, the standard was sealed.

On the ecosystem side, the network effect did the rest: Helm for packaging applications, Prometheus for monitoring, Istio and Linkerd for service mesh, ArgoCD and Flux for GitOps, Trivy and Falco for security. Each additional tool reinforces the value of the standard. On the talent side, Kubernetes skills have become massively in-demand across DevOps and SRE roles, creating a virtuous cycle: more trained engineers, more adopting companies, more engineers getting trained.

In the CNCF 2025 report, the community now describes Kubernetes as "boring," using the term as the highest praise: a mature, predictable tool whose APIs no longer break with every release. That is exactly what you want from infrastructure that has become standard.

When Kubernetes Adds Value, and When Other Approaches Are a Better Fit

The fact that K8S is the standard doesn't mean it's the right answer to every problem. Choosing Kubernetes, a PaaS, or a combination of both depends on the technical and organizational context; at Clever Cloud, many teams use both in parallel for different workloads.

Kubernetes delivers real value in several contexts:

• strong portability requirements: multi-cloud, hybrid, or on-premise combined with cloud, where the Kubernetes manifest becomes a common denominator;
• distributed architectures that apply 12-factor app principles and the "cattle" approach (interchangeable, stateless instances) with sophisticated orchestration needs;
• strategic alignment with the CNCF ecosystem (Helm, Operators, ArgoCD, Istio, etc.) or client/partner prerequisites that impose the standard;
• need for a shared platform across large teams, with a dedicated platform engineering team or the willingness to outsource that responsibility to a managed service.

Conversely, in other contexts, a PaaS like Clever Cloud delivers the same outcomes as Kubernetes (industrialized deployments, autoscaling, resilience) without the operational complexity of the orchestrator. This is particularly true for standard application architectures (web + backend + database) where the effort of configuring and operating Kubernetes doesn't translate into tangible added value.

And for intermediate workloads (IoT, edge, development environments, small clusters), the differences between K3S and K8S are worth weighing before deciding: the lightweight distribution is often a better fit. The de facto standard is not a moral obligation. It is a powerful and costly tool to operate, and its use should be chosen based on the problems it solves, often as a complement to other approaches rather than a replacement.

The Limits of the Standard: Operational Debt

Running Kubernetes in production is not trivial. That's one of the reasons why many companies that adopt K8S opt for a managed service rather than a self-managed installation.

The sources of complexity are numerous: configuring access control correctly (RBAC), choosing and operating a network plugin (CNI), wiring up persistent storage (CSI), setting up observability, managing certificates, performing minor and major upgrades without downtime, hardening security, managing control plane backups. Each of these topics is a discipline in itself. The CNCF 2025 report shows that challenges have actually shifted from purely technical to organizational: 47% of organizations now cite "cultural changes with the development team" as their top obstacle, ahead of raw technical complexity.

At the heart of this debt sits a less-discussed component: etcd. It is the distributed key-value database that stores the complete state of the cluster. etcd is solid for moderately sized clusters, but becomes a bottleneck at scale. It's no coincidence that Google announced in late 2024 the replacement of etcd with Spanner for its managed GKE offering, retaining only the API compatibility layer. AWS, for its part, has built a "new generation" etcd architecture to handle scale. K3S, designed for lightweight environments, has pushed the logic further by offering several alternatives to etcd, including SQLite as the default. When the central component needs to be re-engineered to handle production use at scale, it's a telling sign.

This realization is what led us, at Clever Cloud, to rethink this component. Our Clever Kubernetes Engine replaces standard etcd with Materia etcd, our reimplementation of the etcd protocol built on top of Materia KV and FoundationDB, replicated across three Paris datacenters. This approach is also part of what makes Clever Cloud unique: a multi-tenant control plane that scales horizontally without degrading performance, benefits from FoundationDB’s continuous failure simulation model, and frees teams from managing thousands of fragile etcd instances.

But whether you choose CKE or another service, the principle remains the same: if you want Kubernetes in production without building a dedicated platform engineering team, a managed Kubernetes is almost always the right decision.

In Summary

Kubernetes became the standard for good technical reasons, but also thanks to an alignment of interests that drove the industry to converge around a neutral project governed by the CNCF. The core mechanism, the reconciliation loop and the declarative model, explains its robustness. The ecosystem that has built up around it explains its staying power.

That doesn't mean it should be adopted for everything. For many contexts, a PaaS or another approach is a better fit, and in practice, the two often coexist within the same architecture, each where it delivers the most value. For serious distributed architectures, it remains the tool of reference, provided you account for the operational debt it introduces and choose between running it yourself or relying on a managed service.

This is precisely the promise that Clever Kubernetes Engine, our managed Kubernetes, seeks to deliver: standard Kubernetes, operated in France on sovereign infrastructure, with a control plane redesigned to eliminate the friction of etcd at scale. And for teams that don't need Kubernetes, our PaaS remains the most direct path to production.

FAQ

Are K8S and Kubernetes the same thing?

Yes. K8S is an abbreviation: the letter K, followed by 8 (representing the eight letters in "ubernete"), followed by S. Both refer to the same container orchestration system.

What is the difference between Docker and Kubernetes?

Docker is a containerization engine: it packages an application with its dependencies into an image that runs as a container. Kubernetes is an orchestrator: it deploys, monitors, and scales those containers across a fleet of servers. This is one of the most commonly misunderstood distinctions in the ecosystem, even though the two tools serve different and complementary purposes.

Is Kubernetes free?

The software is open source and free under the Apache 2.0 license. But the infrastructure it runs on, the engineering time to maintain it, and the complementary tools (monitoring, backups, security) have a real cost. That's why many companies opt for a managed Kubernetes offering.

Do you always need Kubernetes for production deployments?

No. Kubernetes provides sophisticated orchestration, particularly suited to distributed architectures requiring portability, CNCF ecosystem alignment, or advanced orchestration. For many other contexts, a PaaS delivers the same outcomes (industrialized deployment, autoscaling, resilience) without the operational complexity. And in practice, the two approaches often coexist within the same architecture.

What is the difference between K3S and K8S?

K3S is a CNCF-certified Kubernetes distribution (not a fork), designed to be lightweight and suited for resource-constrained environments: edge, IoT, development machines, small clusters. It replaces some components with lighter alternatives and ships as a single binary. The differences between K3S and K8S come down to several specific architectural choices worth evaluating before making a decision.

How do I get started with Kubernetes?

The fastest way is to spin up a local cluster with Minikube, kind, or k3d, then deploy a simple application via a YAML manifest. For production, the reasonable choice for most teams is a managed Kubernetes offering.

This represents a major milestone that culminates three years of preparation and is part of our broader strategy to maintain complete control over our Paris region's network infrastructure.

Why We Made This Change

In Clever Cloud's early years, we delegated network responsibility to partners. This approach made sense: it allowed us to accelerate development, focus on cloud services, and avoid investing in expertise we hadn't yet mastered. But as our infrastructure grew, the limitations of this dependency became clear. We had no control over strategic decisions — how traffic was routed across the Internet, which paths our packets took, or how quickly we could respond to failures. Every modification, every incident required the involvement of a third party.

We decided to take this responsibility back. In doing so, we gained several concrete advantages. We optimize costs through direct management of our transit and peering relationships. We define our own routing policy instead of following an intermediary's constraints. We resolve incidents ourselves, without waiting for external providers. And we achieve complete control of our network stack — the same way we progressively took control of our servers and datacenters over the past few years.

But this transition isn't just about our operational independence. It brings immediate, tangible benefits for you. The most critical is resilience. Previously, all traffic was routed through a single provider. Any incident on their side impacted every service we offered. We now maintain four upstream providers across three datacenters in the Paris area. When one link fails — and it has happened over the past year — traffic automatically shifts to available alternatives without customer-impacting interruption. We can even withstand the simultaneous loss of multiple transit links.

Beyond redundancy, we gain control over routing itself. We now decide how your traffic reaches its destination. This allows us to optimize paths for lower latency and better performance, and to adjust those decisions based on your specific needs and our network topology. We respond to congestion, to changing conditions, and to your requirements in real time.

Finally, there is the question of operational responsibility. Network issues no longer require us to wait for an external provider to acknowledge and resolve them. Public network failures fall directly under our responsibility — we detect them, analyze them, and fix them ourselves. This directly reduces the time between problem and resolution, which means less downtime and better reliability for our customers.

Operating Your Own Network on the Internet

To operate as an independent network on the Internet, organizations must work with a Regional Internet Registry (RIR). RIRs are responsible for allocating and managing IP addresses and AS numbers within specific geographical regions.

There are five RIRs worldwide:

• RIPE NCC — Europe, Central Asia, and the Middle East
• ARIN — North America (United States, Canada, and the Caribbean)
• LACNIC — Latin America and the Caribbean
• APNIC — Asia-Pacific region
• AFRINIC — Africa

For Clever Cloud, since our infrastructure is primarily in Europe, we work with RIPE NCC (Réseaux Internet Publics Européens).

Allocated Address Space

As a member of a RIR, organizations receive allocations of both IPv4 and IPv6 address space. For RIPE NCC members, this typically includes a /24 block of IPv4 addresses (depending on the availability of such a block) and a /29 block of IPv6 addresses. These allocations are managed under your membership and can be used to operate your network globally.

Creating Our Autonomous System

The groundwork for this transition began several years ago. In 2019, we created our RIPE NCC account to become a LIR (Local Internet Registry). This gave us access to a /24 IPv4 block (91.208.207.0/24) and a /29 IPv6 block (2a0f:d0c0::/29).

Then, in 2022, we registered our Autonomous System Number (ASN) with the Regional Internet Registry for our region. Our AS number is AS213394. Here is the aut-num object:

> whois AS213394

aut-num:        AS213394
as-name:        CleverCloud
org:            ORG-CCS42-RIPE
import:         from AS29075 accept ANY
import:         from AS3257 accept ANY
import:         from AS3356 accept ANY
import:         from AS43424 accept ANY
export:         to AS29075 announce AS213394:AS-CLVRCLD
export:         to AS3257 announce AS213394:AS-CLVRCLD
export:         to AS3356 announce AS213394:AS-CLVRCLD
export:         to AS43424 announce AS213394:AS-CLVRCLD
admin-c:        QA171-RIPE
tech-c:         QA171-RIPE
status:         ASSIGNED
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         mnt-fr-clvrcldnet-1
created:        2022-11-28T08:24:23Z
last-modified:  2025-02-25T16:36:15Z
source:         RIPE

An Autonomous System Number (ASN) is a unique identifier for networks on the Internet. It's required to announce routes via BGP — the protocol that makes inter-network routing possible. Creating an AS early on allowed us to plan for this eventual transition and prepare the necessary infrastructure in advance.

Now that we have an ASN, we can start announcing our prefixes to other networks on the Internet using the BGP protocol.

The Role of the RIPE Database

The RIPE NCC maintains a public database of routing objects (like the aut-num object above). Among these objects are route objects, which specify which AS is authorized to announce a particular IP prefix. In practice, these entries are primarily used by network operators and transit providers to build routing policy and filters (IRR-based filtering) to accept or deny announcements from their peers. This is one way to try to prevent BGP hijacks. By applying those filters to the routes you receive from your peers, you can limit the propagation of a prefix that originates from the wrong ASN.

Let's say that Org A owns 192.0.2.0/24 and announces it to Transit X. Transit X applies a filter on routes learned from Org A to only accept the IP prefixes that Org A has in its RIR database. This way, if Org A starts announcing a prefix it doesn't own (let's say our public prefix, 91.208.207.0/24), then Transit X is supposed to reject that route. This helps prevent the bad route from being propagated and traffic from being forwarded to the wrong entity.

However, not all networks implement IRR filtering. Better mechanisms like ROA (Route Origin Authorization) exist to address this gap.

The BGP Protocol: How the Internet Routes Traffic

To understand how we announce our prefixes on the Internet, it's essential to understand BGP — the Border Gateway Protocol. BGP is the de facto standard routing protocol of the Internet. It allows networks (Autonomous Systems) to exchange information about which IP prefixes they own and how to reach them.

BGP works in both directions. When we announce to our peers and transit providers "we own 91.208.207.0/24", this announcement travels through the Internet from network to network. Each network that forwards our announcement prepends its own AS number to the AS_PATH — a list showing the sequence of networks a packet traverses to reach us. For example, OVHcloud (AS16276) sees the path [AS29075, AS213394]: traffic goes through one of our transit providers (AS29075), then reaches us (AS213394). Each network that forwards the announcement updates it this way, building a complete path. Here's an example using the OVHcloud Looking Glass service:

> show route for 91.208.207.0/24 all

91.208.207.0/24    via 172.18.16.0 on eno1 [lil1_rbx1_bagg1_8k 2025-12-25] * (100/0) [AS213394i]
    Type: BGP unicast univ
    BGP.origin: IGP
    BGP.as_path: 29075 213394
    BGP.next_hop: 172.18.16.0
    BGP.med: 161
    BGP.local_pref: 40
    BGP.community: (0,0) (29075,18000) (65535,65281)
    BGP.23 [t]: 00 00 b8 6e
                   via 172.18.16.64 on eno1 [lil1_rbx8_bagg1_8k 2025-12-25] (100/0) [AS213394i]

At the same time, we receive announcements from other networks about their prefixes and the paths to reach them. This builds the opposite view: when we need to send traffic outbound, we know which path to take to reach any given destination. Here's an example with one of OVHcloud prefixes:

> /routing/route/print detail where dst-address=5.39.0.0/17 and active

Ab   afi=ip4 contribution=active dst-address=5.39.0.0/17 routing-table=main pref-src=185.133.116.2 gateway=213.242.111.201 immediate-gw=213.242.111.201%sfp28-6 distance=20 scope=40 target-scope=10 belongs-to="bgp-IP-213.242.111.201"

      bgp.as-path="3356,16276" bgp.communities=3356:2,3356:2066,3356:22,16276:40001,3356:100,65002:7018,3356:123,3356:901,65002:701,65000:64990,65000:64995,65000:64996,3356:502 .med=0 .atomic-aggregate=no .origin=igp

Here the network path OVHcloud uses to reach us is different from the one we use to reach them.

The Migration Process

Our IP prefixes were entirely managed by our historical provider. While we legally owned the addresses, we delegated the technical responsibility of announcing them to the Internet to this single provider. This meant:

• Our provider's AS (AS43424) was listed as the origin of our prefixes in the Internet routing tables
• All traffic destined for our services or outgoing to the Internet had to flow through their infrastructure

Clever Cloud Services
   |
   | all inbound/outbound traffic
   v
Historical Provider (AS43424)
   |
   | originates: 91.208.207.0/24 (origin AS43424)
   v
Internet

We now want our ASN to be the origin of the announcements. To migrate safely, we planned a three-step migration. The requirements were simple: we could not accept any customer-impacting interruption.

Migrating a prefix between ASNs

To migrate a prefix from one AS to another, we needed to modify its route object in the RIPE database. The procedure was straightforward but required careful timing.

First, we created a second route object in the RIPE database for our 91.208.207.0/24 prefix. Now both ASNs were registered as authorized to announce the same prefix — both our historical provider's AS and our own AS (213394).

These routing objects are publicly queryable via the whois command or through the RIPE web interface. For example, running whois -h whois.ripe.net -T route 91.208.207.0/24 returns both registered objects:

❯ whois -h whois.ripe.net -T route 91.208.207.0/24
% Information related to '91.208.207.0/24AS213394'

route:          91.208.207.0/24
mnt-by:         mnt-fr-clvrcldnet-1
descr:          CleverCloud subnet
origin:         AS213394
created:        2025-01-15T10:29:14Z
last-modified:  2025-01-15T10:29:14Z
source:         RIPE

% Information related to '91.208.207.0/24AS43424'

route:          91.208.207.0/24
mnt-by:         mnt-fr-clvrcldnet-1
mnt-by:         MAGICRETAIL-MNT
descr:          CleverCloud subnet
origin:         AS43424
created:        2020-02-13T09:06:33Z
last-modified:  2020-02-13T09:06:48Z
source:         RIPE

Once this second route object was registered and propagated across the Internet (i.e., network operators pulled an up-to-date version of the RIPE database to build their routing filters), our new transit providers could see that our ASN was authorized to announce this prefix. At that point, we could begin announcing the prefix through our own infrastructure.

If we didn't create that route object, our route announcement might have been rejected and we could have been flagged as BGP hijackers.

Announcing Through Our Historical Provider

Once the second route object was propagated, we performed the first step during the night of January 16, 2025: we began announcing the prefix ourselves via BGP to our historical provider. This was still using the same transit path, but now with Clever Cloud AS213394 originating the announcements instead of our historical provider. Our historical provider continued to relay the prefix, but now received it from us rather than announcing it directly.

Clever Cloud Services (AS213394)
   |
   | originates: 91.208.207.0/24 (origin AS213394)
   v
Historical Provider (AS43424)
   |
   | re-announces: 91.208.207.0/24 (origin AS213394)
   v
Internet

This first phase served as validation — if any issues arose, we could quickly revert without impacting other transit paths.

Announcing Through Our Own Transits

A few days later, during the night of January 21, 2025, we took the final step: we began announcing the prefix through our own dedicated transit connections.

Clever Cloud Services (AS213394)
   |
   | originates: 91.208.207.0/24 (origin AS213394)
   |
   +---+---+---+
   |   |   |   |
   v   v   v   v
  T1  T2  T3  HP
(Transit providers + historical provider)
   |   |   |   |
   +---+---+---+
   |
   v
Internet

We now announce our prefixes directly to four upstream providers (three transit providers, T1/T2/T3, plus our historical provider HP). Traffic flows across all paths, and we have full control over routing decisions and redundancy.

Throughout both phases, we observed no customer-impacting interruption. The BGP protocol's built-in redundancy and the gradual nature of the transition ensured that traffic flowed smoothly regardless of which path was preferred at any given moment.

Complete Internet Routing Visibility

As part of Phase 2, our three primary transit providers began sending us a "full view" of the Internet's routing table. This is the complete set of all publicly announced IPv4 and IPv6 prefixes — roughly ~1 million IPv4 routes and ~220,000 IPv6 routes.

A full view gives us unprecedented visibility into how the Internet is structured and allows us to make sophisticated routing decisions. Rather than relying on a single provider's perspective, we now see all available paths to reach any destination on the Internet.

With this information, we are able to:

• Choose optimal paths for our outbound traffic based on our network topology and preferences
• Implement traffic engineering to direct flows through specific transit providers
• Respond dynamically to network conditions and congestion
• Balance load across our four transit connections based on real-time routing data

This fine-grained control over our routing policy is a direct result of operating our own AS and managing our own announcements — exactly the kind of operational independence we sought when we began this transition.

One Year Later

Nearly a year into operating our own network announcements, the transition has proven successful. We have experienced no major incidents, and our infrastructure has proven resilient. When minor issues have occurred — such as packet loss through a specific transit provider or the temporary loss of a transit link — traffic has automatically rebalanced across our remaining connections. We have been able to detect and respond to these issues directly, without waiting for a third-party provider to take action. Our customers experienced no customer-impacting interruption. This ability to own our problems and resolve them quickly is perhaps the greatest benefit we've gained.

What's Next

This transition is far from finished. We have several roadmap items ahead of us:

• Increased network capacity to handle growing traffic demands
• BGP peering with other networks to optimize traffic locally without paying for transit
• ROA (Route Origin Authorization) deployment to cryptographically sign our route announcements and prevent unauthorized parties from hijacking our prefixes
• RPKI (Resource Public Key Infrastructure) validation to ensure the legitimacy of announcements we receive from other networks and protect against prefix hijacking attacks
• IPv6 expansion, both inbound (accepting IPv6 traffic) and outbound (sending IPv6 traffic) — a transition we will roll out in phases

Conclusion

In early 2025, Clever Cloud completed its transition to fully independent network operations. We now announce our own IP prefixes through four upstream providers, giving us full authority over how traffic flows in and out of our infrastructure.

For our customers, this translates to better reliability and faster problem resolution in our Paris region. When network issues occur, we handle them directly — and our multi-provider redundancy ensures traffic keeps flowing even when incidents occur.

This milestone is just the beginning. We're already working on BGP peering to optimize local traffic, ROA signing and RPKI validation to strengthen routing security, and IPv6 expansion to fully embrace dual-stack connectivity. We're building a network as robust and self-sufficient as the rest of our infrastructure — and we're excited about what comes next.

So we built our own orchestrator. It runs on micro-VMs and gives us a clean kernel boundary between workloads, with no shared kernel between tenants. That is what runs tens of thousands of applications in production on our PaaS today, and for most projects it is still the shortest path from a commit to production.

When Docker arrived, some workloads fit better as a container image than as one of our native runtimes. So we added a Docker runtime to the platform, where it made sense. Not to replace our approach, but to broaden it.

Then Kubernetes established itself as the de facto standard for container orchestration. Its ecosystem (Helm, operators, GitOps, and so on) became unavoidable for many teams.

We could have kept saying that, in many cases, Kubernetes is not the best path to production. That is still true. But it was no longer enough. That is why we are launching CKE, our Clever Kubernetes Engine, available in public beta starting today.

As many know, we resisted for years the idea of offering Kubernetes as just another checkbox in the Console. Not because Kubernetes is useless, but because it has too often become a default answer to problems the PaaS solves more simply: deploying an application, scaling it, monitoring it, isolating it, connecting it to a database, managing its environment variables, its logs and its lifecycle. And we still believe that.

But Kubernetes has become the common language of a large part of the cloud-native ecosystem. Helm, operators, GitOps, already-containerized workloads, internal platforms and existing toolchains are part of the daily reality of many teams. And our reason for being is to make life easier for technical teams.

The question was no longer "should we do Kubernetes?", but "can we do Kubernetes without giving up what makes Clever Cloud?"

Not a quickly repackaged Kubernetes

The fastest way to ship managed Kubernetes is to wrap an upstream distribution behind an admin console, add a provisioning API, and bill the cluster by the hour. It is a valid strategy if you want to ship fast. It is not the one we wanted.

For CKE, we had three requirements that cannot be solved by simply bolting Kubernetes on top of the rest of the platform.

The first is sovereignty. CKE is operated in Europe, on our infrastructure and that of our partners, and can also be deployed on the on-premises infrastructure of customers who need it. No dependency on US hyperscalers, no grey areas around data jurisdiction, no vague promises about where the infrastructure actually runs.

The second is integration with the rest of Clever Cloud. We did not want to create a Kubernetes silo sitting alongside the PaaS, the managed databases, the object storage and the private network. We wanted a Kubernetes that fits inside the same platform, with the same Console, the same tooling, the same billing, the same governance rules and the same managed services.

The third is operational predictability. Kubernetes is a complex distributed system, and its behaviour under load depends heavily on its foundations. We did not want to operate a product whose underlying layer would remain a black box or a default-accepted limitation. That is what led us to work on something few providers touch: Kubernetes' internal consistency layer.

Under the hood: Materia etcd on FoundationDB

In Kubernetes, etcd is the component that stores cluster state: manifests, resources, secrets, node state. It is the source of truth for the entire orchestrator.

It is also one of the most sensitive components in the system.

etcd works very well within the scope it was designed for. But its limits at scale are well known: store size, latencies under heavy write load, behaviour during network partitions, backup and restore operations, the need for compaction and fine-grained monitoring. When Kubernetes becomes a critical foundation, etcd becomes an operational concern in its own right.

We did not want to build CKE on top of a brick we would then treat as a fragile black box.

So we took the problem back to its root: keep the contract Kubernetes expects, but replace the internal consistency layer with an implementation backed by FoundationDB. We call it Materia etcd.

FoundationDB gives us distributed ACID transactions, a robust consistency model, and a deterministic simulation testing approach that fits very well with how we build critical infrastructure.

For end users, this work is invisible. That is precisely the point. Under the hood, this foundation lets us build CKE with the auto-scaling, auto-healing and operational predictability we expect from a Clever Cloud service.

We will come back to Materia etcd in more detail soon, because the topic deserves a dedicated article.

Standard Kubernetes on the developer side

All this work on the underlying layer has a simple goal: on the user side, CKE has to be standard Kubernetes.

Your manifests work without modification. Your Helm charts install normally. Your Argo CD or your Flux plugs in like on any other cluster. Your Kubernetes operators run without surprises.

CKE does not try to reinvent the developer interface of Kubernetes. That would be counterproductive. If you already have a Kubernetes deployment chain, the goal is for it to run on CKE with as little friction as possible.

The difference happens elsewhere: in how this Kubernetes integrates with the rest of the Clever Cloud platform.

Kubernetes when you need it, the PaaS when it is enough

CKE does not replace our PaaS. It complements it.

For many applications, the PaaS remains the best choice: less operations, less configuration, less YAML, less maintenance surface. If your application fits naturally in a Clever Cloud runtime, the PaaS is often still the simplest and most robust path.

But there are cases where Kubernetes is the right tool.

You may need to install :

• An operator;
• Reuse existing manifests;
• Standardize a GitOps chain;
• Deploy a workload already distributed as a Helm chart;
• Run an internal platform built around the Kubernetes API;
• Or simply meet the habits of a team that already works with Kubernetes day to day.

In those cases, the problem is not Kubernetes itself. The problem is Kubernetes isolated from the rest of your system.

That is where CKE changes things. You can keep a Node.js API on the PaaS, a static frontend on Cellar, a managed PostgreSQL database, and run on CKE only the component, the operator or the workload that really needs Kubernetes.

All within the same Clever Cloud environment.

Native integration with the Clever Cloud ecosystem

CKE was designed to integrate with the services you already use on Clever Cloud.

• Cellar, our S3-compatible object storage, can be used from your Kubernetes workloads.
• Managed databases (PostgreSQL, MySQL, MongoDB, Redis, Materia) attach to your cluster the same way they do to any other Clever Cloud application.
• IAM as a Service lets you manage authentication and permissions on the cluster and on the teams that access it.
• Network Groups lets you connect your CKE cluster to your existing Clever Cloud PaaS applications, in the same private network.

The Network Groups integration is probably the one that changes day-to-day work the most.

Kubernetes is often introduced into organizations as a new island: new console, new network, new secrets, new access rules, new billing, a new way to connect services together.

With CKE, the goal is the opposite. Kubernetes becomes one more brick in the Clever Cloud architecture, not a parallel world.

So you can build a hybrid architecture without workarounds: part on the PaaS, part on CKE, managed databases, object storage, private networking, and shared governance.

Enabling CKE and deploying a first application

For this demo, we will stick to the bare minimum: enable the feature, create a cluster, fetch a kubeconfig, add a node group, and deploy a first application with a LoadBalancer service.

On the prerequisites side, you will need Clever Tools version 4.3 or later, and kubectl installed locally.

Since the public beta opened on April 27, the Kubernetes feature can be enabled by any Clever Cloud customer, with a single command:

clever features enable k8s

You can then create a cluster. Give it a name, point to your organization, and the --watch option lets you follow the deployment progress:

clever k8s create my-cluster --org <your-org-id> --watch

Creation takes about a minute. At any time, you can list the clusters in your organization with clever k8s list.

Once the cluster is ready, fetch its kubeconfig and write it directly as your default local configuration:

clever k8s get-kubeconfig my-cluster --org <your-org-id> > ~/.kube/config

From there, it is standard Kubernetes. kubectl talks to the cluster, and your usual tooling follows. You can start by checking that everything is in place:

kubectl get nodes

At this stage, the list is empty: the cluster is created but has no compute capacity. This is a good moment to introduce the first CKE-specific API resource: the NodeGroup. A node group is a set of Kubernetes nodes with the same profile (same flavor, same region), managed as a unit. You describe it like any other Kubernetes resource, in a YAML file:

apiVersion: api.clever-cloud.com/v1
kind: NodeGroup
metadata:
  name: example-nodegroup
spec:
  flavor: M
  nodeCount: 2

And you apply it with kubectl:

kubectl create -f example-nodegroup.yaml

Sixty to ninety seconds later, the nodes join the cluster:

kubectl get nodegroups
NAME                DESIREDNODECOUNT   CURRENTNODECOUNT   FLAVOR   STATUS   AGE
example-nodegroup   2                  2                  M        Synced   2m

kubectl get nodes
NAME                      STATUS   ROLES    AGE   VERSION
example-nodegroup-node0   Ready       2m    v1.35.0
example-nodegroup-node1   Ready       2m    v1.35.0

To resize the node group, you stay within the Kubernetes API:

kubectl scale nodegroup example-nodegroup --replicas=4

With a cluster that now has compute capacity, you can deploy a first application. To keep things simple, an nginx exposed through a LoadBalancer service, which will automatically provision a load balancer on the Clever Cloud side:

kubectl create deployment nginx --image=nginx:alpine --replicas=2 
kubectl expose deployment/nginx --type=LoadBalancer --port 80 
kubectl get service nginx

A few seconds later, the service exposes a public address. This is exactly the responsive behaviour we mentioned about the private testing phase: provisioning a node group, scaling it or exposing a service requires no waiting and no manual configuration.

From here, it is Kubernetes like anywhere else. You can add persistent storage through the Clever Cloud CSI, install the Clever Kubernetes Operator to provision a managed database from your manifests, or plug in your usual GitOps chain. The cluster supports Kubernetes versions 1.34, 1.35 and 1.36, with 1.35 as the default. Everything is detailed in the CKE documentation.

What we saw during private testing and at Devoxx, and what is next

CKE is now in public beta, but some of our customers have had private access for several months. That phase was very useful to expose the product to real-world usage before opening it more widely.

The feedback has been very positive. The cluster behaves as expected, and the operations that come up most often in a Kubernetes team's daily life, such as adding a node or setting up a load balancer, are fast and predictable. That is exactly the behaviour we were aiming for when we invested so much time into the internal consistency layer.

At Devoxx France, we presented CKE on the Clever Cloud booth for three days. The conversations confirmed two things we were already observing.

First, teams are not just looking for a Kubernetes cluster. They are looking for a Kubernetes that integrates cleanly with their platform, their network, their databases, their security constraints and their existing practices.

Second, and probably the most striking feedback, developers are not looking to put everything on Kubernetes. Many want to keep their classic applications on our PaaS, where it is the most efficient, and deploy on Kubernetes only the components that really warrant it, because of their complexity, their distributed architecture, or the constraints of the ecosystem they fit into.

This is exactly the kind of hybrid architecture CKE is designed to enable.

It is also why, ahead of the launch, we built the Clever Kubernetes Operator. It allows a workload running in a Kubernetes cluster, whether hosted with us or elsewhere, to provision and consume our managed services directly from the Kubernetes API: PostgreSQL, MySQL, MongoDB, Redis, Cellar or Materia.

For your teams, it is a kubectl apply that creates a managed database. For CKE, it is the natural tool to bridge Kubernetes workloads and the rest of the Clever Cloud platform.

The beta is open to all Clever Cloud customers. You can enable it right now, deploy your first workloads, and tell us what is missing, what surprises you or what you would like to see come next.

The discussion is open on our GitHub community, and the documentation is available here.

CKE does not replace our PaaS. It complements it.

For many applications, the PaaS remains the simplest path from a commit to production. But when you need Kubernetes, for an operator, a Helm chart, a GitOps chain, an already-standardized workload or an internal platform, you can now do it inside the Clever Cloud environment, with our infrastructure choices, our network, our managed services and our sovereignty requirements.

That is the kind of Kubernetes we wanted to build.

In this context, the question is no longer whether to migrate, but how to structure what already exists, maintain control over day-to-day operations, and retain ownership of environments and data.

On Tuesday 28 April at 11:30 AM, Clever Cloud and Cycloid are hosting a webinar to address these challenges.

Two complementary perspectives on a shared challenge

We are offering a cross-functional discussion to provide concrete insight into these topics.

Cycloid covers the structuring and governance of environments through a unified internal developer platform and portal. Clever Cloud addresses managed operations, operational maintenance (MCO) and operational sovereignty.

Join Florent Perreux (Chief Sales Officer @ Clever Cloud), Steven Le Roux (CTO @ Clever Cloud), Alexandre Blin (Business Director SEMEA @ Cycloid) and Olivier de Turckheim (Solution Architect @ Cycloid) to understand how to align governance and operations without adding complexity.

Agenda

🔹 Why cloud modernisation goes beyond migration;plus à une migration;

🔹 Structuring and governance challenges in multi-cloud environments;

🔹 Operational challenges related to operations and maintenance;

🔹 How to align governance and operations without multiplying complexity;

🔹 A live Q&A session to ask your questions directly.

Why attend?

✅ Understand the root causes of complexity in cloud projects;

✅ Identify the levers to structure your usage and reduce vendor dependency;

✅ Gain a clearer understanding of operational and sovereignty challenges;

✅ Benefit from concrete real-world experience;

✅ Engage directly with experts.

Register now

🗓️ Tuesday 28 April 2026

⏰ 11:30 AM – 12:00 PM

💻 Online webinar on Livestorm

Facing these challenges? Join us on 28 April!

Nantes, France — Clever Cloud, a European cloud provider, announces the public beta launch of Clever Kubernetes Engine (CKE) on April 27, 2026, in the afternoon. The product will be previewed at Devoxx starting April 22, where attendees will be able to test it directly at the Clever Cloud booth.

Built for production and scalability

Kubernetes has become the standard for container orchestration. Clever Cloud spent two years developing a managed, sovereign version designed to integrate naturally into the Clever Cloud ecosystem and operate with the same simplicity as the platform's other services. CKE is built to handle real production workloads, with high scalability and predictable behavior at scale.

To achieve this, Clever Cloud developed Materia etcd, a reimplementation of Kubernetes' internal consistency layer built on FoundationDB. This foundational work — invisible to the end user — is what guarantees CKE's robustness and stability in production, notably through native auto-scaling and auto-healing capabilities.

CKE integrates natively with existing Clever Cloud services — Cellar object storage (S3-compatible), managed databases, IAM as a Service, Network Groups — and remains accessible via standard industry tools: kubectl, Helm, or GitOps.

Sovereign by design

Hosted and operated in Europe by Clever Cloud on its public cloud, CKE offers organizations a concrete alternative to the Kubernetes offerings of major American platforms, without compromising on performance or legal data control. CKE can also be deployed on the customer's own on-premises infrastructure.

Access details

The public beta is open to all Clever Cloud customers from April 27, 2026, via feature activation performed by the user. Notably, some customers have already had access through a private test for over six months, allowing the product to be refined before this general release.

"Clever Cloud has long developed alternatives to Kubernetes. Our customers expressed a clear need for this technology, and we decided to build it alongside them, with the level of technical excellence and sovereignty that defines our platform." — Quentin Adam, CEO of Clever Cloud

Learn more