FoundationDB made this idea famous. Instead of relying only on unit tests and real-cluster end-to-end tests, it runs distributed workloads inside a simulated world where time, network faults, process failures, and scheduling decisions can be explored and reproduced from a seed. TigerBeetle pushed a complementary discipline: make invariants part of the code, assert the shape of the state continuously, and turn silent corruption into loud local failures.

At Clever Cloud, we have been applying that direction to Rust infrastructure software. Moonpool, developed by Pierre Zemb, gives Rust systems a deterministic simulation environment where time, networking, tasks, randomness, and storage can be virtualized behind provider traits. Magnetar is where we apply that approach to a client for Apache Pulsar. This is also the approach that we are taking in Sōzu.

Magnetar is a from-scratch Apache Pulsar client driver in Rust. It is built around a sans-io protocol core, a production Tokio engine, and a Moonpool engine for deterministic simulation. This first public release is meant for engineers who want to try Magnetar, evaluate the architecture, compare behavior with existing Pulsar clients, and help shape a stronger Rust-native Pulsar ecosystem.

The goal is not only to expose a Rust API around Pulsar. The goal is to build a client whose difficult behavior can be inspected, replayed, and made to fail before production gets the chance.

A Pulsar client is part of the distributed system

It is tempting to think about a client library as the easy side of a distributed system. The broker is where replication, storage, ownership, and coordination live. The client only connects, sends messages, receives messages, and acknowledges them.

That view does not survive contact with a real Apache Pulsar client.

A serious Pulsar client has to manage producers, consumers, readers, partitioned topics, multi-topic consumers, pattern subscriptions, transactions, schemas, authentication, proxy routing, broker lookup, cluster failover, reconnect, backoff, batching, chunking, ack grouping, unacked-message tracking, negative acknowledgements, retry-letter and dead-letter flows, observability, and a long list of Pulsar Improvement Proposals.

Many of the hard bugs are not about one command being encoded incorrectly. They are about time and ordering. A broker drops while a publish is in flight. A reconnect happens while a consumer has pending acknowledgements. A token refresh races with a session reset. A proxy lookup points to a new broker while the old connection is still draining. A topic migrates and the client must rebuild enough state to continue without pretending the old session still exists.

That means a Pulsar client is not just a convenience wrapper around a socket. It is part of the distributed system. It owns state that has to remain coherent while the world around it changes.

If that state is hidden inside runtime tasks, channel queues, implicit clocks, and timing accidents, then the client becomes hardest to reason about exactly when precision matters most. Magnetar starts from the opposite assumption: make the protocol state explicit, drive it through narrow boundaries, and keep the I/O outside.

From the Rust Pulsar ecosystem to Magnetar

Magnetar comes from experience with the Rust Pulsar ecosystem, including maintaining and using pulsar-rs. That work matters. It gave Rust users an Apache Pulsar client and created the practical context in which the next generation of driver work could be evaluated.

Magnetar is not an article-length criticism of what existed before it. It is a continuation of the same ecosystem from a different architectural starting point. After spending time with real Pulsar usage, the shape of the next step became clearer: a Rust Pulsar driver should make the protocol/runtime boundary sharper, make difficult behavior reproducible, track parity explicitly, and make testability a first-class design constraint.

That is why Magnetar does not begin with the public builder API. It begins with the protocol.

The user-facing API matters, and the first release already exposes a broad surface. But for a distributed client, the deeper question is what happens underneath that API when the broker moves, the connection drops, the clock advances, a timeout fires, a subscription is rebuilt, or the same trace is replayed under a different runtime.

The sans-io core

The heart of Magnetar is magnetar-proto. It is a synchronous protocol state machine. It has no sockets. It does not depend on Tokio. It does not spawn tasks. It does not use async. It does not read the host clock on the protocol hot path.

Its shape follows the same broad idea that made quinn-proto influential in the Rust networking world: feed bytes in, poll bytes out, poll semantic events, and ask the state machine when its next timer expires.

Conceptually, the interface looks like this:

> connection.handle_bytes(now, bytes);

connection.poll_transmit(&mut out);
connection.poll_event();
connection.poll_timeout();

That small shape is a large architectural choice.

handle_bytes lets an engine feed wire bytes into the state machine. poll_transmit drains outbound frames. poll_event yields semantic events that the runtime must react to, such as authentication challenges, lookup outcomes, checksum mismatches, reconnect signals, or protocol-level transitions. poll_timeout lets the runtime ask when to wake the state machine again.

The state machine does not decide where bytes come from. It does not know whether the network is a real TCP stream, a TLS stream, a byte pipe inside a simulator, or a scripted broker in a differential harness. It does not decide what "now" means. The engine passes time in.

The public traits in magnetar-proto follow the same rule. They are not runtime services hidden behind async callbacks. They are small protocol contracts at the points where the state machine needs a decision or a transformation: AuthProvider produces authentication bytes, Schema defines encoding, decoding, schema data, and broker-resolved schema hooks, MessageEncryptor and MessageDecryptor describe payload crypto over bytes and message metadata, ServiceUrlProvider exposes the current service URL for failover, and HealthProbe uses a poll-style contract instead of binding the protocol crate to an async runtime. The pattern is deliberate: the protocol core can ask for facts, bytes, or readiness, while the I/O, caching, refresh logic, network probes, and scheduler stay outside.

This separation is not just a clean layering exercise. It is what makes the protocol testable without a broker, without a socket, and without an async runtime. A protocol bug can be reduced to a fixture: feed these bytes at this instant, poll these outputs, assert this event stream, and replay the same sequence again.

For an Apache Pulsar client, that matters because the wire protocol is not only a happy path. It includes producer creation, consumer subscription, acks, nacks, transactions, broker errors, redirects, authentication challenges, topic migration, replicated-subscription markers, schema metadata, batching, chunking, and close paths. Each of these has state. Each state transition is easier to reason about when it is not mixed with socket reads, task scheduling, and runtime-specific wakeups.

No channels as an architectural constraint

Magnetar also makes an unusual Rust async choice: channel primitives are banned from the workspace.

The straightforward architecture for a network client is often producer future to channel, channel to driver task, driver task to channel, channel to consumer future. That can work, but it spreads state across queues and tasks. Backpressure becomes implicit. Close semantics become library-specific. Dropped futures can leave messages stranded in places that are hard to inspect. The debugging question becomes "where did this message go?"

Magnetar uses a different pattern. State lives inside the protocol state machine and the runtime-owned shared connection. User-facing futures register wakers in slabs keyed by operation identifiers. The driver owns the I/O loop, feeds bytes into the state machine, drains outbound frames, and wakes the matching futures as outcomes arrive.

This is not an aesthetic ban on channels. It is a way to keep ownership of state explicit. When the driver rebuilds producers after a reconnect, when a consumer waits for a message, when an ack outcome is delivered, or when a pending operation fails because a session is gone, the relevant state is in the state machine and can be inspected there.

That design has trade-offs. It puts more pressure on the correctness of the state machine. It requires careful lock ordering. It requires reviewers to understand waker registration and wakeup paths. But it also makes the behavior much easier to simulate, replay, and compare across runtimes.

Two engines, one state machine

Magnetar ships two engines that drive the same protocol core.

The default engine is magnetar-runtime-tokio. It is the production execution path: TCP, TLS through tokio-rustls, driver tasks, real time, real sockets, and the public PulsarClient surface most users will reach first.

The second engine is magnetar-runtime-moonpool. It drives the same magnetar-proto::Connection state machine through Moonpool providers. Instead of baking in the runtime, Moonpool exposes provider traits for networking, time, task spawning, randomness, and storage. Under a production-style provider bundle, the engine can run against real I/O. Under simulation providers, those same categories can be virtualized.

This is where deterministic simulation becomes practical. The client can be driven in a world where time advances under control, network delivery can be reordered or interrupted, tasks run under a deterministic scheduler, randomness is seeded, and failures can be replayed from the same seed.

Moonpool is not a mock. A mock usually replaces a component with simplified behavior. That is useful for many unit tests, but it is not the same thing as exercising a real client through hard interleavings.

Moonpool is also not a normal end-to-end test. An e2e test runs the real system, which is essential, but it usually runs with real time, real networking, and failures that are difficult to reproduce once the timing has moved on.

Moonpool sits in a different space. It keeps the client code close to the real execution path while virtualizing the providers that make distributed behavior hard to reproduce. The point is not to avoid e2e tests. The point is to add a regime where failures are not anecdotes. They become seeds.

Magnetar's Moonpool engine uses the same protocol state machine as the Tokio engine. The TLS path is also driven in a way that preserves the simulation boundary: instead of relying on tokio-rustls, the Moonpool engine drives rustls::ClientConnection over the byte pipe directly. That keeps TLS handshakes under the same deterministic control as the rest of the simulated network.

Differential testing: making the engines agree

Two engines are useful only if they do not silently become two different clients.

Magnetar therefore includes a differential harness. It runs a trace, such as connect, open producer, send, subscribe, receive, ack, seek, and close, against both the Tokio and Moonpool engines, then compares the user-visible event streams.

This matters because the Moonpool engine is used to find bugs that are difficult to expose with real time and real sockets. For that to be meaningful, the simulated engine must remain observationally aligned with the production engine at the user boundary.

If Tokio and Moonpool drift, the simulator becomes less trustworthy. If the differential harness keeps them aligned, then a seed failure in the simulated world is much more likely to point at a real property of the client, not a separate test-only implementation.

This is also why Magnetar's validation policy is strict about cross-runtime coverage. Behavioral changes are expected to exercise the sans-io layer, the Tokio runtime, the Moonpool runtime, differential equivalence, and the e2e surface when applicable. That is more work than adding one unit test. It is also the cost of making deterministic simulation a real engineering tool rather than a side experiment.

Java parity as a credibility contract

Simulation is not a substitute for feature breadth. It is what makes feature breadth less fragile.

Apache Pulsar has a mature Java client, and for many users that client defines what "a Pulsar client" means. Basic produce and consume are not enough. A credible client must cover the behaviors applications rely on: batching, compression, transactions, schemas, partitioned topics, multi-topic and pattern consumers, retries, dead-letter flows, authentication providers, proxy support, failover, admin operations, and the PIPs that shape modern Pulsar usage.

Magnetar tracks Java-client parity as a public contract. The README carries the parity matrix and the engine-by-engine status. The article does not need to reproduce that matrix, but the matrix itself is important. It gives users a way to evaluate the project as a Pulsar client, not as a narrow experiment around one happy path.

This is the reason Magnetar's first public release is broader than a minimal demo. It includes the protocol foundation, the public facade, runtime engines, admin support, authentication providers, message crypto, CLI work, observability hooks, and a growing list of PIP surfaces.

The exact reason for tracking parity is simple: users do not choose a client library only because its internal architecture is elegant. They choose it because it can represent the system they actually run. Architecture makes that surface maintainable. Parity makes it useful.

Beyond produce and consume

The phrase "Pulsar client" can hide a lot of surface area. From the outside, the first examples usually look small: build a client, create a producer, send a payload, subscribe with a consumer, receive a message, acknowledge it. Those examples are useful because they show the entry point. They are not enough to evaluate the client.

Real Pulsar applications depend on behaviors that only appear after the first demo works. A producer may need batching, compression, chunking, access modes, sequence ids, send timeouts, per-message properties, transactions, interceptors, and latency statistics. A consumer may need batch receive, cumulative acknowledgements, batch-index acknowledgements, nack backoff, ack timeout handling, retry topics, dead-letter topics, seek, pause/resume, subscription modes, and per-partition behavior. Applications with strict schemas need Avro, JSON, Protobuf, Protobuf-native, KeyValue, primitive schemas, and broker-side schema lookup semantics. Deployments with security requirements need token auth, mTLS, OAuth2, SASL, Kerberos/GSSAPI, Athenz, TLS provider choices, and a way to reason about crypto behavior without making every build drag every provider.

Those features are not independent checkboxes. They interact. Batching interacts with compression and send timeouts. Chunking interacts with sequence ids and redelivery. Transactions interact with acknowledgements. Reconnect interacts with producers, consumers, lookups, proxy routing, and failover. Observability has to describe the behavior without leaking secrets or flooding operators under churn.

That is why Magnetar's feature work and architecture work are tied together. The more complete the surface becomes, the more important it is that the protocol state is explicit and that the runtime boundary is narrow. A broad client built on accidental state becomes harder to maintain as it becomes more useful. A broad client built on replayable state gives maintainers a better chance to keep feature growth from turning into hidden behavior.

Validation as architecture

Magnetar's test strategy is intentionally layered because each layer catches a different class of bug.

The sans-io layer lets protocol behavior be tested without sockets or async tasks. The project testing documentation currently records more than 270 unit tests in magnetar-proto. Those tests exercise the state machine directly: bytes in, events out, transmit buffers out, state transitions checked.

Runtime integration tests exercise the glue that pure protocol tests cannot cover. The Tokio engine and Moonpool engine each have their own integration surfaces, and the project enforces runtime parity so one engine does not quietly fall behind the other.

The deterministic chaos pack targets the failure schedules that motivated the architecture in the first place: reconnect, failover, virtual-clock timers, TLS handshake ordering, in-flight publishes, broker migration, and adversarial network behavior. The Moonpool engine documentation describes a daily Moonpool seed sweep with 128 freshly rolled random seeds in parallel. The number is less important than what it represents: the client is being driven through schedules that a human would not write by hand.

Differential equivalence tests compare the two runtime worlds. They ask a direct question: if the same trace is run through Tokio and Moonpool, does the user see the same behavior?

End-to-end tests still matter. They keep Magnetar honest against a real Apache Pulsar broker. Simulation can make failures reproducible, but it does not replace compatibility with the system users actually deploy.

Taken together, these layers are not ceremony. They are part of the architecture. The point is not to collect large test numbers. The point is to put validation pressure exactly where distributed clients fail: protocol state, runtime boundaries, reconnect logic, timing, equivalence, and real broker behavior.

What reproducibility changes in practice

The practical value of deterministic simulation is not that it makes tests more sophisticated. It changes the debugging loop.

When a traditional end-to-end test fails because of timing, the first question is often whether the failure is reproducible at all. If it is not, engineers start adding sleeps, logs, larger timeouts, or retry loops. Some of those changes are useful. Many of them only move the failure somewhere else. The failure remains a story: "we saw it once in CI".

With a deterministic simulation, the failure should become an artifact. The artifact is a seed, a trace, and a state transition that can be replayed. The question changes from "can we make it happen again?" to "what invariant did this schedule violate?"

That difference matters for a client library. Client bugs often sit at the boundary between user code and the broker. They are easy to misattribute. A publish can fail because the broker closed the connection, because the client lost an outcome, because the reconnect path rebuilt the producer too late, because an ack was delivered to the wrong waiter, because a timeout fired against stale session state, or because the runtime task died after the protocol had already transitioned.

If all of that behavior is spread across tasks, channels, timers, and sockets, a failure report is hard to compress. If the protocol state machine is explicit, the runtime boundary is narrow, and the same sequence can be replayed under controlled providers, the report can become much smaller: this seed, this trace, this event order, this invariant.

That is why Magnetar treats reproducibility as part of the client design. The point is not only to catch more bugs. The point is to make the bugs that do appear cheaper to understand and safer to fix.

It also changes code review. A reviewer can ask whether a behavior is covered at the right layer. Is this a pure protocol transition? Then the sans-io test should feed bytes and assert events. Is it runtime glue? Then Tokio and Moonpool need equivalent coverage. Is it an observable behavior shared by both engines? Then the differential harness should compare the event streams. Is it broker compatibility? Then an e2e test should touch a real Pulsar broker.

Those questions are more precise than "did we add tests?" They connect the test to the failure model.

What to inspect first

For engineers reading Magnetar for the first time, the best entry point is not a single file. It is the relationship between the documents and the code.

The README gives the public surface: what the client exposes, which features are implemented, how the Java-client parity matrix is tracked, which PIPs are supported, and how the two engines are positioned. That is the user-facing contract.

The architecture document explains the shape underneath that contract. It shows the crate topology, the dependency direction, the sans-io boundary, the driver loop, the event model, the producer and consumer paths, the runtime engines, TLS sites, schemas, PIP coverage, and the validation strategy. That is the system map.

The ADR series explains why the system has this shape. ADR-0004 records the sans-io split. ADR-0003 records the no-channels rule. ADR-0011 records clock injection. ADR-0010 records the Java parity decision. ADR-0019 records how the production Tokio engine and the Moonpool engine relate to the parity goal. ADR-0024 records the cross-runtime testing policy.

This matters because Magnetar is intentionally not only a code drop. It is a set of engineering decisions with a public audit trail. If you disagree with the design, the ADRs give you the right surface to discuss. If you want to contribute a feature, the parity matrix and validation policy tell you what "done" means. If you want to evaluate the simulation work, the Moonpool engine documentation and the differential harness show how the same protocol core is driven in different worlds.

The first release should therefore be read in two ways. As a Pulsar user, look at the client surface and the parity matrix. As a Rust infrastructure engineer, look at the state-machine boundary and the simulation strategy. The interesting part of Magnetar is that both views are meant to reinforce each other.

What the first public release gives you

This first public release is a technical entry point. It is for engineers who want to try Magnetar, read the architecture, inspect the ADRs, compare behavior with existing Pulsar clients, and help shape the Rust-native Pulsar ecosystem.

The high-level client API starts with the usual shape:

let client = PulsarClient::builder()
    .service_url("pulsar://localhost:6650")
    .build()
    .await?;

let producer = client
    .producer("persistent://public/default/orders")
    .create()
    .await?;

That API is important because the client must be usable. But the release is also an invitation to inspect the lower layers. Read the README for the feature matrix. Read the architecture document for the sans-io design. Read the ADRs for the decisions behind no channels, swappable engines, clock injection, Java parity, Moonpool parity, TLS provider choices, and cross-runtime validation. Read the Moonpool engine documentation if you are interested in how deterministic simulation can be applied to a real network client.

The important point for this article is the direction: Magnetar is public so engineers can evaluate both the client surface and the engineering model behind it.

The inaugural technical entry point is available via v1.0.0, while the current state of the driver can be tracked in the v1.2.0 release. 

Why this matters beyond Magnetar

Magnetar is an Apache Pulsar client. It is also a concrete example of how we want to build more Rust infrastructure software.

Distributed systems fail through state transitions. Some of those transitions are obvious and easy to test. Many are not. They happen when time advances in an unlucky place, when a retry overlaps with a close path, when a broker drops a connection after accepting part of the session, when a callback is registered just after a notification, or when a reconnect path rebuilds one handle but not another.

You can try to chase those bugs after they happen. Sometimes that is unavoidable. But the architecture can make the chase easier or harder.

A client with a pure protocol core, explicit events, injected time, replaceable I/O, deterministic seeds, and cross-runtime comparison gives engineers better tools. It lets them reduce a failure to a trace. It lets them replay that trace. It lets them ask whether a simulated failure corresponds to the production runtime. It lets them add an invariant where a silent state drift used to hide.

That is the direction Magnetar takes. Make the protocol explicit. Make the runtime replaceable. Make failures reproducible. Make correctness something the code has to demonstrate continuously.

Thank you to Pierre Zemb for Moonpool and for pushing this deterministic-simulation direction in Rust, and to the contributors and maintainers in the Rust Pulsar ecosystem whose work made the next step possible.

Magnetar is open source. The code, architecture notes, ADRs, and documentation live at github.com/CleverCloud/magnetar.

References

Magnetar

• Magnetar repository. Public Rust Apache Pulsar client repository. https://github.com/CleverCloud/magnetar
• Magnetar README. Public feature surface, Java client parity matrix, supported PIPs, engine coverage, and status. https://github.com/CleverCloud/magnetar/blob/main/README.md
• Magnetar architecture. Sans-io rationale, crate topology, driver loop, protocol state machine, and tests. https://github.com/CleverCloud/magnetar/blob/main/ARCHITECTURE.md
• Magnetar documentation index. Moonpool engine, testing, CLI, observability, logging, PIP features. https://github.com/CleverCloud/magnetar/blob/main/docs/README.md
• Moonpool engine documentation. Deterministic-simulation engine, chaos pack, differential harness, provider model. https://github.com/CleverCloud/magnetar/blob/main/docs/moonpool-engine.md
• Testing documentation. Test categories and validation commands. https://github.com/CleverCloud/magnetar/blob/main/docs/testing.md
• ADR-0004. Sans-io magnetar-proto and swappable I/O engines. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0004-sans-io-protocol-core.md
• ADR-0010. Full Java-client parity scope. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0010-v0-1-full-java-parity.md
• ADR-0024. Cross-runtime test and coverage policy. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0024-cross-runtime-test-and-coverage-policy.md

Ecosystem

• pulsar-rs. Existing Rust Apache Pulsar client project. https://github.com/streamnative/pulsar-rs
• Apache Pulsar. Distributed messaging and streaming platform. https://pulsar.apache.org/
• Apache Pulsar Java client. Reference client surface for many Pulsar users. https://pulsar.apache.org/docs/

Deterministic simulation

• FoundationDB testing. Deterministic simulation and fault injection lineage. https://apple.github.io/foundationdb/testing.html
• TigerBeetle TigerStyle. Assertion-first engineering style. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md
• TigerBeetle VOPR. Simulation testing approach. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/internals/vopr.md
• Moonpool repository. Deterministic simulation for Rust systems, developed by Pierre Zemb. https://github.com/PierreZ/moonpool
• moonpool-sim. Simulation engine crate for Moonpool. https://crates.io/crates/moonpool-sim

We lead the project and are its principal contributor (github.com/sozu-proxy/sozu). With Sōzu 2.0, we explained how the proxy moved from "reverse proxy" toward "programmable edge": a new HTTP/2 multiplexer, stronger security defaults, a larger observability surface, traffic policies, operational hardening, and crypto agility.

Sōzu 2.0.2 came a few days later with a smaller but revealing release. It added defence-in-depth against the HTTP/2 bomb class and fixed the timestamp foundation needed to reconstruct trustworthy OpenTelemetry spans from access logs. That release showed how we want to operate the edge: when a protocol-level attack class appears, the proxy should absorb as much of it as possible; when observability is subtly wrong, the foundation should be fixed before product features are built on top.

Sōzu 2.1.0 is the next step in that same story. One of the pieces we described as "on the bench" for 2.0 has landed: first-class UDP load balancer support.

This is not just one more protocol checkbox. Sōzu already handled HTTP, HTTPS, and TCP. With 2.1.0, UDP becomes part of the same model: the same hot-reconfigurable control plane, the same operator-facing posture, the same metrics discipline, and the same open-source release process. That widens Sōzu from a web edge into a broader infrastructure load balancer, able to front datagram services such as DNS, syslog, NTP, and generic UDP workloads without forcing operators to put a second load balancer in front of it.

For Clever Cloud, this matters because platform networking is broader than HTTP. Managed Kubernetes and PaaS both need a clean story for direct transport traffic. For operators running Sōzu themselves, it means one component can cover more of the L4/L7 surface while preserving the design properties that made Sōzu useful in the first place.

Why UDP still matters

Most user-facing traffic on the web goes through HTTP, and it is tempting to treat everything else as a special case. Real platforms do not get that luxury.

DNS is UDP by default. Syslog often runs over UDP. NTP is UDP. Many internal and infrastructure protocols use datagrams because they are simple, latency-sensitive, or request/response shaped enough that a full TCP connection is unnecessary. Kubernetes platforms and PaaS products also need to expose direct transport services cleanly, not only web applications that terminate at HTTP.

Before Sōzu 2.1.0, an operator wanting a single edge for HTTP, TCP, and UDP usually had to compose multiple systems. Sōzu could handle the HTTP and TCP side, while a separate component handled UDP. That split carries operational cost: two models of health checking, two ways to observe traffic, two release cycles, two failure surfaces, two sets of configuration, and two places where routing state can drift.

The point of first-class UDP in Sōzu is convergence. A UDP listener now sits beside tcp, http, and https listeners. UDP frontends and backends are part of the same command, configuration, and state machinery. Operators get UDP-specific metrics at the proxy instead of treating datagram traffic as something happening off to the side.

The customer-facing benefit is not that every application suddenly needs UDP. It is that the platform can converge more traffic classes under one operational model: hot reconfiguration, health awareness, metrics, release discipline, and a single edge component that can evolve toward product features rather than remaining a pile of protocol-specific exceptions.

What is available in Sōzu 2.1.0

Sōzu 2.1.0 adds a new protocol = "udp" listener type alongside the existing HTTP, HTTPS, and TCP listeners. It is opt-in. Existing configurations do not become UDP-aware by accident, and existing sozu-command-lib consumers can move from 2.0.2 to 2.1.0 without rewriting their integrations.

The central idea is to model UDP traffic as virtual flows. UDP itself has no connection, no stream, and no accept loop. A datagram arrives on one socket. There is no kernel-created per-client socket like there is with TCP. Sōzu therefore reconstructs the useful part of a connection in userland: a flow keyed by the client source address, optionally including the source port when that is the right affinity model for the workload.

Once Sōzu has a flow, it can make the same class of decisions a load balancer has to make for any transport:

• which backend owns this client;
• how long this flow should stay alive;
• what happens when a backend becomes unhealthy;
• how overload should be shed;
• what metrics and logs should be emitted.

The release adds two flow-affine algorithms for UDP. The recommended default is HRW, also known as rendezvous hashing. HRW has a property that matters a lot for hot-reconfigurable systems: when the backend set changes, it minimizes movement without requiring a precomputed lookup table. Sōzu also ships Maglev as an opt-in algorithm for larger backend sets or higher packet rates where an O(1) table lookup is more attractive.

Sōzu can also send PROXY protocol v2 information to UDP backends. This carries the real client address to the backend, so the backend does not have to treat the proxy as the only visible peer. By default, Sōzu sends it on the first datagram of a flow, with an opt-in mode to send it on every datagram when the backend needs that context repeatedly.

Health checking is part of the UDP design rather than an afterthought. A UDP backend can be checked through a companion TCP probe, which is the common practical liveness signal for services where bare UDP does not provide a reliable "connected" state. Sōzu can also use an application-level UDP probe when that is meaningful. The checks support rise/fall hysteresis and fail-open semantics, so a transient or global health-check failure does not automatically black-hole traffic.

Finally, UDP gets its own metric surface: datagrams in and out, bytes in and out, active flows, created, evicted, or shed flows, dropped datagrams by reason, backend health, and flow duration. This follows the same principle as the 2.0 observability work: a feature is not production-grade until operators can see what it is doing.

UDP is easy to add badly

The hard part of UDP load balancing is not receiving a datagram and sending it somewhere else. The hard part is deciding what state exists around that datagram.

TCP gives a proxy a connection lifecycle. HTTP gives it requests, responses, headers, and status codes. UDP gives it packets. Packets can arrive out of order. They can disappear. A client can stop sending without closing anything, because there is nothing to close. A backend can become unhealthy while a virtual flow still points to it. A reconfiguration can arrive while flows are active. A listener can reach its flow cap. A timeout can fire just after the flow was refreshed. These are not exotic cases. They are the normal shape of datagram traffic under load.

That is why the Sōzu 2.1.0 UDP work is as much about testability as it is about features.

The UDP core is structured around a sans-io split. The pure core owns the flow table, admission decisions, timers, teardown policy, and load-balancing requests. The impure shell owns sockets, the buffer pool, health checks, syscalls, and metrics emission. The core has no socket, no wall-clock read on the datapath, no random source hidden inside it. Time and seeds are injected.

That shape matters because it makes the core deterministically simulatable. The same seed drives the same sequence. The same virtual clock jumps to the same instants. The same reconfiguration storm happens at the same step. A failure can print a seed and be replayed exactly.

Sōzu 2.1.0 ships a FoundationDB/VOPR-style deterministic simulation harness for the UDP core. The default sweep runs 256 seeds, each driving a randomized adversarial workload through client datagrams, backend replies, stale backend resolutions, reconfiguration bursts, flow-cap changes, clock jumps, drains, and mass teardown. After each action, the harness drains the core outputs and checks model invariants.

On top of that, the release includes an assertion-density sweep inspired by TigerBeetle's TigerStyle. Across the proxy, command plane, and supervisor, Sōzu now carries roughly 1100 debug_assert! checks and multiple invariant sweeps. These assertions are compiled out of release builds, but they are live in tests, fuzzing, end-to-end runs, simulations, and developer builds. They turn a silent state drift into a loud, local failure.

Those numbers are useful because they are not volume metrics. Lines of code do not prove reliability. A 256-seed simulation sweep and dense invariants do tell you something: the code is being driven through adversarial state transitions, and the data structures are continuously asked to prove that they still make sense.

For a proxy, that distinction matters. A wrong counter is not just a counter. It may mean a flow is leaked. A stale timer is not just a timer. It may mean a backend keeps receiving traffic after the state says otherwise. A silent mismatch between a routing table and a slab is not just an internal inconsistency. It is the kind of thing that becomes a production symptom several layers away from the bug.

Deterministic simulation as a direction

The UDP simulator is also a signal about where we want Sōzu engineering to go next.

FoundationDB made deterministic simulation famous in infrastructure software: run the system inside a simulated world, inject failures, explore thousands of schedules, and make every failure reproducible from a seed. TigerBeetle pushes a complementary discipline: state your invariants directly in the code, assert both the positive and the negative space, and make illegal state impossible to ignore during testing.

Sōzu 2.1.0 applies those ideas to the UDP core. The next architectural effort is to make more of Sōzu fit this shape: more sans-io components, more injected clocks and deterministic boundaries, more layers that can be simulated across the relevant parts of the OSI stack before they are exposed as production behavior.

This is also why Moonpool matters to Clever Cloud. Moonpool, developed by PierreZ, is the deterministic-simulation engine we use at Clever Cloud for this direction in Rust. Version 0.7.0 has now landed publicly on crates.io, including moonpool and moonpool-sim. Sōzu's UDP harness is currently an in-tree synchronous simulator shaped around the pure UDP core, but the broader direction is the same: deterministic workloads, virtual time, reproducible seeds, and networked systems that can be made to fail before production gets the chance.

The primary benefit is reliability. We want fewer edge-case bugs in protocol and reconfiguration code. The second benefit is velocity. When more of the proxy is structured as pure state machines with explicit I/O boundaries, new protocol and load-balancing behavior becomes safer to evolve. The third benefit is operability. Hot reload, overload, and failure behavior become easier to reason about because the code has been exercised under those exact shapes.

That is the next layer under the programmable edge. The knobs exposed to users are only worth exposing if the machinery beneath them is deterministic enough to trust.

What this unlocks

For managed Kubernetes, UDP support gives Sōzu a stronger basis for direct transport services. A platform can offer web traffic, TCP services, and UDP services under a more coherent load-balancing model, rather than splitting responsibilities between unrelated components.

For PaaS, it opens the way to product features around datagram workloads without starting from a separate operational stack. DNS-like services, syslog-like flows, NTP-like probes, and generic UDP applications can be reasoned about through the same proxy vocabulary: listeners, clusters, backends, health, metrics, and hot reconfiguration.

For self-hosted Sōzu operators, it means the project covers more of the infrastructure edge. If you already use Sōzu because you want an open-source, hot-reconfigurable proxy with signed releases and a Rust codebase you can audit, 2.1.0 expands the class of traffic you can bring under that umbrella.

There is an observability consequence too. UDP traffic now has native proxy metrics. You can ask how many flows are active, how many were shed, why datagrams were dropped, whether backends are healthy, and how long flows live. This is what turns a protocol feature into something operators can run.

Reliability also comes from the release around UDP

The headline is UDP, but 2.1.0 also continues the operational work from 2.0.

Hot reconfiguration is one of Sōzu's core promises, so state replay correctness matters. This release fixes a case where a listener whose configuration changed while it remained active could be replayed as remove plus add without re-emitting activation. That kind of bug is exactly why the state model has to be tested as a lifecycle, not only as isolated commands.

The release also fixes an arithmetic-underflow panic in the pattern trie for hostnames whose leftmost segment is a regex. It fixes the release pipeline by pinning the cosign binary to the 2.x line, after a tooling change broke the signature artifact flow during the 2.0.2 release.

These are not the parts of the release that users will see first. They are still part of the product. At Clever Cloud's scale, operational quietness is a customer benefit. A proxy that reconfigures safely, signs its artifacts predictably, and turns edge cases into tests is a proxy that lets applications keep serving traffic while the platform moves underneath them.

What comes next

UDP support will continue to mature with real-world use. There is room for performance work, batching, and broader operational feedback. But the larger direction is architectural: make more of Sōzu deterministic, make more of it sans-io where that makes sense, and make more failure modes reproducible before they can become incidents.

That is the lesson from Sōzu 2.1.0. The programmable edge is not only a set of user-facing toggles. It is also a way of building infrastructure software: protocol by protocol, state machine by state machine, with enough observability to operate it and enough simulation to trust it.

Sōzu is open source under AGPL-3.0, with its command library under LGPL-3.0. The code, release notes, issues, and design discussions live at github.com/sozu-proxy/sozu. Thank you to the contributors and operators who keep pushing the project toward a broader, safer, more programmable edge.

References

Sōzu

• Sōzu 2.1.0 release notes. First-class UDP load balancing, deterministic simulation, assertion-density work, CI and documentation updates. https://github.com/sozu-proxy/sozu/releases/tag/2.1.0
• Sōzu UDP RFC and master plan. Public design issue for first-class UDP load balancing. https://github.com/sozu-proxy/sozu/issues/1273
• Sōzu UDP implementation PR. Implementation of first-class UDP listeners and load balancing. https://github.com/sozu-proxy/sozu/pull/1274
• Long-standing UDP load-balancing issue. Original public tracking issue opened in 2020. https://github.com/sozu-proxy/sozu/issues/654
• Sōzu 2.0: turning a reverse proxy into a programmable edge. Milestone release article. https://www.clever.cloud/blog/engineering/2026/05/29/sozu-2-0-reverse-proxy-programmable-edge/
• Sōzu configuration documentation. UDP listeners, UDP clusters, health checks, PROXY protocol, and metrics. https://github.com/sozu-proxy/sozu/blob/2.1.0/doc/configure.md
• Sōzu testing documentation. Testing doctrine: unit, e2e, fuzz, deterministic simulation, and regression guards. https://github.com/sozu-proxy/sozu/blob/2.1.0/doc/testing.md
• Sōzu UDP simulation documentation. 256-seed default sweep and replay knobs. https://github.com/sozu-proxy/sozu/blob/2.1.0/doc/udp_simulation.md

Protocols and algorithms

• PROXY protocol v2 specification. Client-address propagation across proxies. https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
• Rendezvous hashing / HRW. Flow-affine selection with minimal disruption on backend-set changes. https://en.wikipedia.org/wiki/Rendezvous_hashing
• Maglev: A Fast and Reliable Software Network Load Balancer. Google paper describing Maglev consistent hashing. https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/

Testing

• FoundationDB testing. Deterministic simulation and fault injection lineage. https://apple.github.io/foundationdb/testing.html
• TigerBeetle TigerStyle. Assertion-first engineering style. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md
• TigerBeetle VOPR. Simulation testing approach. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/internals/vopr.md
• Moonpool. Deterministic simulation testing for distributed systems in Rust. https://crates.io/crates/moonpool
• moonpool-sim. Simulation engine for the Moonpool framework. https://crates.io/crates/moonpool-sim
• Moonpool repository. Developed by PierreZ. https://github.com/PierreZ/moonpoo

This release is a milestone: the underlying machine is now ready for the product features we have been waiting to build on top of it. Rather than a list of patches, we've grouped the work into six themes; for each, we say two things — what shipped, and what it makes possible, for the people running their applications on the platform and for the operators who run Sōzu on their own infrastructure.

1. A from-scratch HTTP/2 multiplexer

Sōzu's HTTP/1 stack had already been rewritten around kawa, our pivot format: a single internal representation of the HTTP message, independent of its protocol version — the same idea as HAProxy's HTX. The HTTP/2 multiplexer is its natural extension: it adds session management and stream multiplexing on top of kawa.

It supports the full protocol matrix (H1↔H1, H1↔H2, H2↔H1, H2↔H2), with shared stream state, HPACK compression via loona-hpack, backend H2 connection pooling, RFC 9218 Extensible Priorities for stream prioritisation, and per-listener ALPN negotiation so each TLS connection lands on the right code path. Around 181 end-to-end tests and two cargo-fuzz targets keep the parser and the HPACK decoder honest — including regression guards that pin large-response integrity byte-for-byte across an H1-backend → H2-frontend path, the exact boundary where edge-triggered epoll readiness bugs hide.

For Clever Cloud customers, the practical impact is simple: every application served by the platform now speaks HTTP/2 by default, on the frontend — no opt-in, no code change, no configuration. Enabling HTTP/2 all the way to the backend stays a per-cluster choice — a cluster, in Sōzu's model, is one of your applications; turned on end-to-end, it unlocks gRPC across the whole chain. Page loads finish in fewer TCP connections; browsers can coalesce requests across hostnames that share a certificate (we honour RFC 7540 §9.1.1 SAN coalescing, so when the certificate, authority and connection conditions line up, Firefox and Chrome can reuse a single connection across your cdn.example.com and assets.example.com instead of opening parallel ones). It is, in the best sense, a quiet platform upgrade.

It is also the structural prerequisite for what comes after — HTTP/3 over QUIC and stronger streaming foundations at the edge. The mux rewrite is the part of the roadmap that had to land first.

2. Security as a baseline, not a toggle

The second half of the H2 rewrite is the part nobody asks for until it's missing: flood and DoS hardening. Sōzu 2.0 ships built-in mitigations for CVE-2023-44487 (Rapid Reset), CVE-2024-27316 (CONTINUATION flood), CVE-2025-8671 (MadeYouReset), and the PING / SETTINGS / empty-DATA flood family from CVE-2019-9512/CVE-2019-9515/CVE-2019-9518. Each mitigation surfaces a dedicated counter — twelve metrics under h2.flood.violation.* — so a SIEM can window the trip rate without parsing logs. Seventeen HPACK rejection reasons are surfaced the same way; that is the first operator-visible signal for request-smuggling probes against the H2 stack.

The rest of the security work fans out across the proxy: a per-(cluster, source-IP) connection cap with 429 Too Many Requests; opt-in eviction of the oldest sessions when the accept queue saturates; hardening across the command channel, the HTTP/1 parser, the pattern-trie router (closing a routing-bypass through unanchored regex) and the wildcard matcher; audit-log sanitisation against Trojan-Source and SIEM column-smuggling; and a TLS certificate hot-rotation that never drops the working certificate on failure, even if the new one is malformed. Four dependency advisories cleared in the same window.

Every privileged mutation now lands in a structured audit log: each control-plane action is recorded as a Command(verb=…, actor_uid=…, actor_user=…, result=…) line — who did what, from where, and whether it succeeded. Two dedicated sinks ship it out: audit_logs_target for the human-readable stream, and audit_logs_json_target for one stable-schema JSON object per line, so the trail tails straight into a SIEM (Wazuh, Elastic, Loki, Splunk) without a bespoke parser — in a shape designed for PCI-DSS 10.5.

The product framing is straightforward: our job is to operate the platform and shield your applications as soon as we can. When the next H/2 vulnerability in this class lands at 9pm on a Friday, it does not have to translate into a patch-and-redeploy weekend across thousands of applications — attacks of this class are largely absorbed or mitigated at the proxy, surfaced as a counter on a dashboard, while applications keep serving traffic. Trust-by-default is not a marketing claim; it is the cumulative effect of dozens of small, defensive fixes shipped at the layer where the security boundary actually lives.

3. Visibility on every layer

The largest single user-facing addition in 2.0 is sozu top — an operator TUI (behind the tui Cargo feature) that gives you a live btop/htop-style view across seven panes: Overview, Clusters, Backends, Listeners, H2, Certificates, Events. Colour-blind-safe palette, customisable themes — the essentials fit in a terminal, no external dashboard required.

Underneath the TUI, the metric surface itself was rewritten. The mux exposes per-frame-type counters and a breakdown by RFC 9113 error code, TLS handshake telemetry, per-status HTTP counters, and new worker lifecycle gauges. The access log gains TLS and forwarding fields (version, cipher, SNI, ALPN, XFF chain), x_request_id propagation end-to-end, and client and server RTT — enough to follow a request from one hop to the next.

A new command, SetMetricDetail, lets an operator raise metric cardinality on demand via a time-bounded lease that expires: production stays low-cardinality by default, and deep inspection becomes a one-off decision rather than a config rewrite.

For Clever Cloud, this is the foundation of the next chapter of customer-facing observability — per-application latency percentiles, per-cluster availability, TLS handshake breakdowns, request IDs you can follow across hops. The metrics now exist at the proxy. The next step is to surface them in the Clever Cloud console where they belong, alongside the build and deploy views — so you do not need an external APM to understand the traffic your application actually sees.

4. Traffic policies that you can finally toggle

Sōzu 2.0 reshapes the entire frontend-policy surface (#1231): typed HSTS (RFC 6797) configurable per listener and per frontend; URL rewrite (host, path, port) with regex capture propagation from the routing trie into rewrite templates; request- and response-header rewriting per frontend — add, set, or delete any header (an empty value deletes it, HAProxy del-header parity), with listener-scoped X-Real-IP injection and anti-spoof elision of client-supplied values on top; HTTP 301 / 302 / 308 redirects through a typed RedirectPolicy enum; and HTTP Basic authentication per frontend, with SHA-256-hashed credentials and a constant-time compare via the subtle crate (the credential boundary is hardened against timing side-channels — the audit caught it at review time).

This is the part of the release where the product opportunity is most visible. Today, doing a clean domain migration on a managed platform typically requires either a backend that knows how to redirect or a separately deployed redirect service; putting a staging URL behind a password typically requires an authentication add-on. In Sōzu 2.0, these become knobs at the proxy. The plumbing exists; what remains is to expose those knobs in the Clever Cloud console — as a checkbox on a domain, or a one-click toggle for preview environments. That is the no-code traffic-control surface we want to build next.

5. Operations as the customer benefit

A rewrite this deep only ships safely if the fleet keeps moving underneath it. Day-to-day operations got a lot of attention in 2.0.

sozu listener {http,https,tcp} update is a new field-masked patch verb that tunes non-bind-only listener settings on a running proxy without cycling sockets — H/2 flood thresholds, SNI binding, ALPN preference, idle timeouts, HSTS, custom answers. CVE mitigations can now be tightened under attack without cycling listener sockets. Active backend health checks run inside the existing mio event loop (no async runtime, no extra threads) with HTTP/1.1 and HTTP/2 probes, jittered intervals, and a fail-open path that routes through Normal-status backends whose retry policy allows it. systemd integration (closes #228) now does the right thing: Type=notify units, READY=1 only after the initial workers spawn and the saved state replays, STOPPING=1 on graceful shutdown, MAINPID=<new> across hot upgrades.

Releases themselves changed shape. Pushing a tag now produces ten pre-built tarballs (three Linux targets crossed with up to four crypto providers), signed keyless via sigstore (cosign + GitHub OIDC), with SLSA build provenance and a SOURCE.txt corresponding-source pointer satisfying AGPL §6 / LGPL §4 — closing the gap noted in #1089. ACME fullchain.pem files now load cleanly even when the client emitted the leaf at the start (Certbot, lego, acme.sh); a six-year-old worker-auto-restart binary race (#515) is fixed by pinning the original inode through /proc/self/fd. The LoadState IPC verb stays forward-compatible with sozu-command-lib 1.1.1 clients, so the ecosystem of integrations does not break on the upgrade.

At Clever Cloud's scale, the cost of an operational incident isn't a ticket — it is compounding latency for thousands of applications, and the engineers' time we owe to building the next thing. Sōzu 2.0 is the proxy that is quiet to operate, and that quietness is what customers experience as uptime. It's also what makes Sōzu credible as something you can run on your own infrastructure: signed binaries, hot reloads, ACME quirks ironed out, systemd integration done properly.

6. Crypto that's ready for tomorrow

The TLS stack got a quieter but equally important rework (#1191). Sōzu now supports four pluggable crypto providers for rustls — crypto-ring (the default), crypto-aws-lc-rs, crypto-openssl, and fips (which implies aws-lc-rs in FIPS mode). All four are exercised by CI, and the precedence chain fips > ring > aws-lc-rs > openssl resolves the active provider deterministically when several features are enabled together.

What we are most happy about, though, is the default groups_list: X25519MLKEM768 is now the first-preference key-exchange group where the provider supports it. That is the post-quantum hybrid being standardised through the IETF (draft-ietf-tls-ecdhe-mlkem) and already registered in the IANA TLS registry; what it means in practice is that an X25519-only client and a PQ-capable client both negotiate the most robust mutually supported exchange, with no operator action.

When the post-quantum migration becomes a regulatory requirement — on a timeline the industry is still debating — the groundwork on Clever Cloud will already be in place: every TLS 1.3 client that offers the hybrid negotiates it today, with no operator action. We made the choice quietly, and we made it the default.

From reverse proxy to programmable edge

Sōzu has always been an infrastructure load balancer: well beyond HTTP, it also balances raw TCP, with a zero-copy forwarding path via splice(2) on Linux for TCP listeners. With 2.0 it moves closer to an API gateway and becomes the substrate of a programmable edge — and "programmable" is the operative word. Each capability described above (HTTP/2 by default, anti-abuse controls, observability, traffic policies, crypto agility, operational quietness) is a knob. The next year of Clever Cloud's roadmap is about exposing those knobs as product features: through the console, through the API, through the workflows you already use to deploy your applications.

This is what we mean when we say Sōzu 2.0 is a first step of something greater. Two-point-zero is not the destination — it is the platform we now build on top of. Managed HTTP/2 everywhere is the first building block. The next is a console where you toggle HSTS, password-protect a preview environment, or redirect a migrated domain. The destination — a fully programmable edge with policy primitives you can compose — is what we are building toward over the coming release cycle.

Two more pieces are already on the bench. First, a UDP load-balancing layer — in the spirit of IPVS, but with Sōzu's hot-reload model — paired with TCP health checks that probe the liveness of the UDP backends; this cements the infrastructure-load-balancer role further. Second, HTTPS-reachable backends, which firm up upstream connectivity and open the road to the API gateway.

Sōzu is open source under AGPL-3.0 (the command library is LGPL-3.0). The 2.0 release binaries are signed via sigstore and ship SLSA provenance; if you operate your own edge, this release is yours to use, audit, and extend. The code, the issue tracker, and the conversations live at github.com/sozu-proxy/sozu.

Thank you to the contributors who made this release happen — and to everyone running Sōzu in production, whose feedback shapes where it goes next.

References

Standards and specifications

• RFC 6797 — HTTP Strict Transport Security (HSTS). https://datatracker.ietf.org/doc/html/rfc6797
• RFC 7540 — Hypertext Transfer Protocol Version 2 (HTTP/2). Now obsoleted by RFC 9113, but §9.1.1 on connection coalescing remains the citation reused by RFC 9113. https://datatracker.ietf.org/doc/html/rfc7540
• RFC 9113 — HTTP/2 (current). §5 streams, §6 frames, §6.8 GOAWAY, §7 error codes, §8.1 HTTP semantics. https://datatracker.ietf.org/doc/html/rfc9113
• RFC 9218 — Extensible Prioritization Scheme for HTTP. https://datatracker.ietf.org/doc/html/rfc9218
• draft-ietf-tls-ecdhe-mlkem — Hybrid key exchange in TLS 1.3: X25519MLKEM768 (IETF Internet-Draft, IANA-registered; the post-quantum hybrid Sōzu prefers by default). https://datatracker.ietf.org/doc/html/draft-ietf-tls-ecdhe-mlkem

CVEs

• CVE-2019-9512 — HTTP/2 Ping Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9512
• CVE-2019-9515 — HTTP/2 Settings Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9515
• CVE-2019-9518 — HTTP/2 Empty Frames Flood. https://nvd.nist.gov/vuln/detail/CVE-2019-9518
• CVE-2021-42574 — Trojan Source (bidirectional override). https://nvd.nist.gov/vuln/detail/CVE-2021-42574
• CVE-2023-44487 — HTTP/2 Rapid Reset. https://nvd.nist.gov/vuln/detail/CVE-2023-44487
• CVE-2024-27316 — HTTP/2 CONTINUATION Flood. https://nvd.nist.gov/vuln/detail/CVE-2024-27316
• CVE-2025-8671 — MadeYouReset (HTTP/2). https://nvd.nist.gov/vuln/detail/CVE-2025-8671

Further reading

• Cloudflare Learning Center — What is HSTS?. Accessible primer on HSTS, the max-age / includeSubDomains / preload semantics, and the HSTS preload list policy. https://www.cloudflare.com/learning/ssl/what-is-hsts/
• Cloudflare — HTTP/2 Rapid Reset: deconstructing the record-breaking attack. The canonical write-up of the October 2023 coordinated disclosure, including attack mechanics and the 398 M rps peak captured live. https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/
• HAProxy Technologies — HAProxy is Not Affected by the HTTP/2 Rapid Reset Attack (CVE-2023-44487). HAProxy's structural argument for why their stream lifecycle naturally absorbs Rapid Reset; a useful contrast against Sōzu's flood-counter approach. https://www.haproxy.com/blog/haproxy-is-not-affected-by-the-http-2-rapid-reset-attack-cve-2023-44487
• Cloudflare blog — Post-quantum series. Multi-year coverage by Bas Westerbaan and colleagues on PQ key exchange and hybrid groups; lineage of the X25519MLKEM768 hybrid Sōzu now negotiates by default. https://blog.cloudflare.com/tag/post-quantum/

When you access a website or an online application, you most often do so in a "secure" way. This is for example the well-known green padlock that symbolizes HTTPS connections in your browser, which has become a standard these years thanks to initiatives like Let's Encrypt. 

This means that the data transferred to the server is encrypted, and that even if they are intercepted, they cannot be read by a third party. This protection has been provided by the TLS (Transport Layer Security) protocol for almost 20 years, whether it’s a personal site, an online shop or an access to your bank's services.

Over time, this critical technical brick on the Internet has evolved to strengthen the level of security it offers. In August 2018, its version 1.3 (the latest) was released. Meanwhile, versions 1.0 and 1.1 were considered to no longer offer a sufficient level of protection. They have been deprecated by the IETF (Internet Engineering Task Force) since March 2021 and have therefore been gradually removed from recent browsers such as Firefox, Chrome and its derivatives or Safari.

At Clever Cloud, we have seen our customers adopt TLS 1.2 and 1.3 gradually. On our load balancers, based on our in-house and open source reverse proxy Sōzu, the latest version accounts for over 90% of the requests processed each day. TLS 1.2 for just under 9%. TLS 1.0 and 1.1 for only a few tens of thousands of requests per day, less than 0.1% of our traffic.

While we have maintained these versions for compatibility reasons, this will no longer be the case as of June 30. We will of course inform the customers affected by this choice, and encourage them to switch to more recent versions, which will have advantages for them in terms of security, performance and SEO.

Several reminders will be sent between now and the final shutdown of TLS 1.0 and 1.1. If you have any questions on this subject, please contact our support team through the Console.

Hello 🖖, fellow humans and robots! Our engineering team is proud to announce a brand new Rust software development kit (sdk), also known as "clevercloud-sdk" on crates.io.

Some of you may have noticed that there are new repositories on our GitHub organization. Those repositories are named "clevercloud-sdk-rust" and "oauth10a-rust". Their goal is to provide a convenient way to interact with the Clever Cloud's API with interesting features that we will highlights in this blog post.

Built to be asynchronous

The software development kit and the oauth 1.0a client are mainly built on top of two crates that are well-known in the community. Those are hyper, and the asynchronous runtime that power it, named tokio.

The main idea behind using the hyper crate and by extension the tokio asynchronous runtime is to leverage the current ecosystem that gravitates around these crates and efforts to integrate a new kernel API called io_uring. Once this new kernel API will be integrated into the tokio runtime, this will lead to significant performance, enhancement according to the tokio's announcement blog post. We can’t wait for this enhancement that will give us a way to iterate faster! 🚀

Observability in mind

These crates are developed with the idea to be fully observable. It has been achieved through others well-known crates as log and tracing, or using community initiatives like the prometheus crate. Integrations of those crates are taking part of the compiler feature flags system provided by the cargo project.

Standard logging facade

First, you need to know that the standard logging facade is enabled by default. It will output useful information of internals behaviors of above crates, using the standard logger if it is defined. It will help you to understand how calls to the Clever Cloud's API are achieved with some debug information, if you need so.

if you want to disable it, you have to disable the default sets of enabled features. This could be achieved using the following syntax for the clevercloud-sdk crate in your Cargo.toml file.

clevercloud-sdk = { version = "^0.10.0", default-features = false }

Metrics-oriented

At Clever Cloud, we think that metrics are important. They help to understand systems by correlating things that doesn't seem to be related in the first place.

This is why we have launched Clever Cloud Metrics rather soon, some years ago. We recently added more features to it by providing more ways to query Clever Cloud Metrics, thanks to Erlenmeyer which is a time series query language proxy. You can find more details of how you could use it through this blog post.

Lastly, we have announced the Tardis product which empowers you to send us your metrics and we take care of the rest as we always do and more recently we have integrated pre-built Grafana's dashboards to help you to visualize your applications and add-ons state, you can learn more about it here.

Coming back to our crates. You can enable metrics collection using the prometheus crate, by enabling the metrics flags at crate-level in your Cargo.toml file. It could be achieved using the following syntax.

clevercloud-sdk = { version = "^0.10.0", features = ["metrics"] }

Once this feature is enabled, there isn't much more to do. You will need to expose metrics through an HTTP server or send them using the prometheus push gateway. This part is described in the prometheus crate's documentation or in its examples.

Tracing capabilities

Last but not least, a pillar of observability is tracing. According to the OpenTelemetry initiative and documentation, tracing is explained as such:

Traces track the progression of a single request, called a trace, as it is handled by services that make up an application. The request may be initiated by a user or an application. Distributed tracing is a form of tracing that traverses process, network and security boundaries. Each unit of work in a trace is called a span; a trace is a tree of spans. Spans are objects that represent the work being done by individual services or components involved in a request as it flows through a system. A span contains a span context, which is a set of globally unique identifiers that represent the unique request that each span is a part of. A span provides Request, Error and Duration (RED) metrics that can be used to debug availability as well as performance issues...

OpenTelemetry documentation

If you want to take a closer look to tracing, even if it is not in Rust, you can take a look at the OpenTelemetry GitHub organization. There are plenty of libraries, SDKs and documentation to help you to build your tracing solution or integrate with an already existing one. Besides, if you are using the rust language and maybe the tokio runtime, you can have a look at those blog posts which explain how tracing works in the above asynchronous runtime and how to visualize them using the console or using the OpenTelemetry connector which allow to send traces to more sinks, the list is available on the rust OpenTelemetry GitHub repository.

So how do we concretely enable the tracing feature using the clevercloud-sdk ? As mentioned above all features are driven using a feature flag, so to get tracing capabilities, you need to enable the trace flag.

clevercloud-sdk = { version = "^0.10.0", features = ["trace"] }

Once the feature flag has been enabled, you will need to collect and send traces using the above connector and sinks.

How does it look?

Let us look at a concrete example to illustrate what the clevercloud-sdk looks like. But before going through the usage of the clevercloud-sdk, I will introduce you to the dependencies and requirements you will need. The first thing you need to achieve is to declare crates in the Cargo.toml file.

[dependencies]
tokio = { version = "^1.17.0", features = ["full"] }
clevercloud-sdk = { version = "^0.10.0", features = ["metrics", "tokio", "trace", "jsonschemas"] }

We are now able to write a proper piece of software using the clevercloud-sdk. Here is an example of how to use the SDK:

...
use clevercloud_sdk::{
    oauth10a::{
        proxy::{self, ProxyConnectorBuilder},
        Credentials,
    },
    v2::myself,
    Client,
};
...

// See the full code at:
// - https://github.com/CleverCloud/clevercloud-sdk-rust/blob/main/examples/cleverctl
// - https://github.com/CleverCloud/clevercloud-sdk-rust/blob/main/examples/cleverctl/src/cmd/myself.rs
pub async fn get(config: Arc<Configuration>, output: &Output) -> Result<(), Error> {
    let credentials: Credentials = config.credentials.to_owned().into();
    let connector = ProxyConnectorBuilder::try_from_env().map_err(Error::ProxyConnector)?;
    let client = Client::builder()
        .with_credentials(credentials)
        .build(connector);

    let user = myself::get(&client).await.map_err(Error::Get)?;

    println!(
        "{}",
        output
            .format(&user)
            .map_err(|err| Error::FormatOutput(Box::new(err)))?
    );

    Ok(())
}

Another example of the sdk which interact with add-ons:

...
use clevercloud_sdk::{
    oauth10a::{
        proxy::{self, ProxyConnectorBuilder},
        Credentials,
    },
    v2::addon,
    Client,
};
...

// See the full code at:
// - https://github.com/CleverCloud/clevercloud-sdk-rust/blob/main/examples/cleverctl
// - https://github.com/CleverCloud/clevercloud-sdk-rust/blob/main/examples/cleverctl/src/cmd/addon/mod.rs 
pub async fn list(
    config: Arc<Configuration>,
    output: &Output,
    organisation_id: &str,
) -> Result<(), Error> {
    let credentials: Credentials = config.credentials.to_owned().into();
    let connector = ProxyConnectorBuilder::try_from_env().map_err(Error::ProxyConnector)?;
    let client = Client::builder()
        .with_credentials(credentials)
        .build(connector);

    let addons = addon::list(&client, organisation_id)
        .await
        .map_err(|err| Error::List(organisation_id.to_owned(), err))?;

    println!(
        "{}",
        output
            .format(&addons)
            .map_err(|err| Error::FormatOutput(Box::new(err)))?
    );
    Ok(())
}

You can find the entire source code of the command line interface example in the clevercloud-sdk project into the examples repository or you can follow this link.

What's next?

Let's have a look at the incoming features that are to be integrated into these crates.

Biscuit and oauth2 bearer token

At Clever Cloud, we are working on a new authentication and authorization token called Biscuit. More details about it in the introduction blog post.

This new authentication and authorization token will work with the OAuth 2.0 standard. One job of the SDK and the client under the hood is to ease the transition from OAuth 1.0a to OAuth 2.0 using Biscuit.

New features!

The Software Development Kit will grow to support the entire API. The next feature is the integration of the event bus in order to subscribe to events that happen on Clever Cloud's platform. You will be able to get notifications about a deployment of an application and so on...

Besides, those crates are used to build another software that you have discovered in another blog post. It introduce a Kubernetes operator that you may use on OpenShift, that exposes Clever Cloud's add-ons as Custom Resources.