FoundationDB made this idea famous. Instead of relying only on unit tests and real-cluster end-to-end tests, it runs distributed workloads inside a simulated world where time, network faults, process failures, and scheduling decisions can be explored and reproduced from a seed. TigerBeetle pushed a complementary discipline: make invariants part of the code, assert the shape of the state continuously, and turn silent corruption into loud local failures.
At Clever Cloud, we have been applying that direction to Rust infrastructure software. Moonpool, developed by Pierre Zemb, gives Rust systems a deterministic simulation environment where time, networking, tasks, randomness, and storage can be virtualized behind provider traits. Magnetar is where we apply that approach to a client for Apache Pulsar. This is also the approach that we are taking in Sōzu.
Magnetar is a from-scratch Apache Pulsar client driver in Rust. It is built around a sans-io protocol core, a production Tokio engine, and a Moonpool engine for deterministic simulation. This first public release is meant for engineers who want to try Magnetar, evaluate the architecture, compare behavior with existing Pulsar clients, and help shape a stronger Rust-native Pulsar ecosystem.
The goal is not only to expose a Rust API around Pulsar. The goal is to build a client whose difficult behavior can be inspected, replayed, and made to fail before production gets the chance.
A Pulsar client is part of the distributed system
It is tempting to think about a client library as the easy side of a distributed system. The broker is where replication, storage, ownership, and coordination live. The client only connects, sends messages, receives messages, and acknowledges them.
That view does not survive contact with a real Apache Pulsar client.
A serious Pulsar client has to manage producers, consumers, readers, partitioned topics, multi-topic consumers, pattern subscriptions, transactions, schemas, authentication, proxy routing, broker lookup, cluster failover, reconnect, backoff, batching, chunking, ack grouping, unacked-message tracking, negative acknowledgements, retry-letter and dead-letter flows, observability, and a long list of Pulsar Improvement Proposals.
Many of the hard bugs are not about one command being encoded incorrectly. They are about time and ordering. A broker drops while a publish is in flight. A reconnect happens while a consumer has pending acknowledgements. A token refresh races with a session reset. A proxy lookup points to a new broker while the old connection is still draining. A topic migrates and the client must rebuild enough state to continue without pretending the old session still exists.
That means a Pulsar client is not just a convenience wrapper around a socket. It is part of the distributed system. It owns state that has to remain coherent while the world around it changes.
If that state is hidden inside runtime tasks, channel queues, implicit clocks, and timing accidents, then the client becomes hardest to reason about exactly when precision matters most. Magnetar starts from the opposite assumption: make the protocol state explicit, drive it through narrow boundaries, and keep the I/O outside.
From the Rust Pulsar ecosystem to Magnetar
Magnetar comes from experience with the Rust Pulsar ecosystem, including maintaining and using pulsar-rs. That work matters. It gave Rust users an Apache Pulsar client and created the practical context in which the next generation of driver work could be evaluated.
Magnetar is not an article-length criticism of what existed before it. It is a continuation of the same ecosystem from a different architectural starting point. After spending time with real Pulsar usage, the shape of the next step became clearer: a Rust Pulsar driver should make the protocol/runtime boundary sharper, make difficult behavior reproducible, track parity explicitly, and make testability a first-class design constraint.
That is why Magnetar does not begin with the public builder API. It begins with the protocol.
The user-facing API matters, and the first release already exposes a broad surface. But for a distributed client, the deeper question is what happens underneath that API when the broker moves, the connection drops, the clock advances, a timeout fires, a subscription is rebuilt, or the same trace is replayed under a different runtime.
The sans-io core
The heart of Magnetar is magnetar-proto. It is a synchronous protocol state machine. It has no sockets. It does not depend on Tokio. It does not spawn tasks. It does not use async. It does not read the host clock on the protocol hot path.
Its shape follows the same broad idea that made quinn-proto influential in the Rust networking world: feed bytes in, poll bytes out, poll semantic events, and ask the state machine when its next timer expires.
Conceptually, the interface looks like this:
> connection.handle_bytes(now, bytes);
connection.poll_transmit(&mut out);
connection.poll_event();
connection.poll_timeout();That small shape is a large architectural choice.
handle_bytes lets an engine feed wire bytes into the state machine. poll_transmit drains outbound frames. poll_event yields semantic events that the runtime must react to, such as authentication challenges, lookup outcomes, checksum mismatches, reconnect signals, or protocol-level transitions. poll_timeout lets the runtime ask when to wake the state machine again.
The state machine does not decide where bytes come from. It does not know whether the network is a real TCP stream, a TLS stream, a byte pipe inside a simulator, or a scripted broker in a differential harness. It does not decide what “now” means. The engine passes time in.
The public traits in magnetar-proto follow the same rule. They are not runtime services hidden behind async callbacks. They are small protocol contracts at the points where the state machine needs a decision or a transformation: AuthProvider produces authentication bytes, Schema defines encoding, decoding, schema data, and broker-resolved schema hooks, MessageEncryptor and MessageDecryptor describe payload crypto over bytes and message metadata, ServiceUrlProvider exposes the current service URL for failover, and HealthProbe uses a poll-style contract instead of binding the protocol crate to an async runtime. The pattern is deliberate: the protocol core can ask for facts, bytes, or readiness, while the I/O, caching, refresh logic, network probes, and scheduler stay outside.
This separation is not just a clean layering exercise. It is what makes the protocol testable without a broker, without a socket, and without an async runtime. A protocol bug can be reduced to a fixture: feed these bytes at this instant, poll these outputs, assert this event stream, and replay the same sequence again.
For an Apache Pulsar client, that matters because the wire protocol is not only a happy path. It includes producer creation, consumer subscription, acks, nacks, transactions, broker errors, redirects, authentication challenges, topic migration, replicated-subscription markers, schema metadata, batching, chunking, and close paths. Each of these has state. Each state transition is easier to reason about when it is not mixed with socket reads, task scheduling, and runtime-specific wakeups.
No channels as an architectural constraint
Magnetar also makes an unusual Rust async choice: channel primitives are banned from the workspace.
The straightforward architecture for a network client is often producer future to channel, channel to driver task, driver task to channel, channel to consumer future. That can work, but it spreads state across queues and tasks. Backpressure becomes implicit. Close semantics become library-specific. Dropped futures can leave messages stranded in places that are hard to inspect. The debugging question becomes “where did this message go?”
Magnetar uses a different pattern. State lives inside the protocol state machine and the runtime-owned shared connection. User-facing futures register wakers in slabs keyed by operation identifiers. The driver owns the I/O loop, feeds bytes into the state machine, drains outbound frames, and wakes the matching futures as outcomes arrive.
This is not an aesthetic ban on channels. It is a way to keep ownership of state explicit. When the driver rebuilds producers after a reconnect, when a consumer waits for a message, when an ack outcome is delivered, or when a pending operation fails because a session is gone, the relevant state is in the state machine and can be inspected there.
That design has trade-offs. It puts more pressure on the correctness of the state machine. It requires careful lock ordering. It requires reviewers to understand waker registration and wakeup paths. But it also makes the behavior much easier to simulate, replay, and compare across runtimes.
Two engines, one state machine
Magnetar ships two engines that drive the same protocol core.
The default engine is magnetar-runtime-tokio. It is the production execution path: TCP, TLS through tokio-rustls, driver tasks, real time, real sockets, and the public PulsarClient surface most users will reach first.
The second engine is magnetar-runtime-moonpool. It drives the same magnetar-proto::Connection state machine through Moonpool providers. Instead of baking in the runtime, Moonpool exposes provider traits for networking, time, task spawning, randomness, and storage. Under a production-style provider bundle, the engine can run against real I/O. Under simulation providers, those same categories can be virtualized.
This is where deterministic simulation becomes practical. The client can be driven in a world where time advances under control, network delivery can be reordered or interrupted, tasks run under a deterministic scheduler, randomness is seeded, and failures can be replayed from the same seed.
Moonpool is not a mock. A mock usually replaces a component with simplified behavior. That is useful for many unit tests, but it is not the same thing as exercising a real client through hard interleavings.
Moonpool is also not a normal end-to-end test. An e2e test runs the real system, which is essential, but it usually runs with real time, real networking, and failures that are difficult to reproduce once the timing has moved on.
Moonpool sits in a different space. It keeps the client code close to the real execution path while virtualizing the providers that make distributed behavior hard to reproduce. The point is not to avoid e2e tests. The point is to add a regime where failures are not anecdotes. They become seeds.
Magnetar’s Moonpool engine uses the same protocol state machine as the Tokio engine. The TLS path is also driven in a way that preserves the simulation boundary: instead of relying on tokio-rustls, the Moonpool engine drives rustls::ClientConnection over the byte pipe directly. That keeps TLS handshakes under the same deterministic control as the rest of the simulated network.
Differential testing: making the engines agree
Two engines are useful only if they do not silently become two different clients.
Magnetar therefore includes a differential harness. It runs a trace, such as connect, open producer, send, subscribe, receive, ack, seek, and close, against both the Tokio and Moonpool engines, then compares the user-visible event streams.
This matters because the Moonpool engine is used to find bugs that are difficult to expose with real time and real sockets. For that to be meaningful, the simulated engine must remain observationally aligned with the production engine at the user boundary.
If Tokio and Moonpool drift, the simulator becomes less trustworthy. If the differential harness keeps them aligned, then a seed failure in the simulated world is much more likely to point at a real property of the client, not a separate test-only implementation.
This is also why Magnetar’s validation policy is strict about cross-runtime coverage. Behavioral changes are expected to exercise the sans-io layer, the Tokio runtime, the Moonpool runtime, differential equivalence, and the e2e surface when applicable. That is more work than adding one unit test. It is also the cost of making deterministic simulation a real engineering tool rather than a side experiment.
Java parity as a credibility contract
Simulation is not a substitute for feature breadth. It is what makes feature breadth less fragile.
Apache Pulsar has a mature Java client, and for many users that client defines what “a Pulsar client” means. Basic produce and consume are not enough. A credible client must cover the behaviors applications rely on: batching, compression, transactions, schemas, partitioned topics, multi-topic and pattern consumers, retries, dead-letter flows, authentication providers, proxy support, failover, admin operations, and the PIPs that shape modern Pulsar usage.
Magnetar tracks Java-client parity as a public contract. The README carries the parity matrix and the engine-by-engine status. The article does not need to reproduce that matrix, but the matrix itself is important. It gives users a way to evaluate the project as a Pulsar client, not as a narrow experiment around one happy path.
This is the reason Magnetar’s first public release is broader than a minimal demo. It includes the protocol foundation, the public facade, runtime engines, admin support, authentication providers, message crypto, CLI work, observability hooks, and a growing list of PIP surfaces.
The exact reason for tracking parity is simple: users do not choose a client library only because its internal architecture is elegant. They choose it because it can represent the system they actually run. Architecture makes that surface maintainable. Parity makes it useful.
Beyond produce and consume
The phrase “Pulsar client” can hide a lot of surface area. From the outside, the first examples usually look small: build a client, create a producer, send a payload, subscribe with a consumer, receive a message, acknowledge it. Those examples are useful because they show the entry point. They are not enough to evaluate the client.
Real Pulsar applications depend on behaviors that only appear after the first demo works. A producer may need batching, compression, chunking, access modes, sequence ids, send timeouts, per-message properties, transactions, interceptors, and latency statistics. A consumer may need batch receive, cumulative acknowledgements, batch-index acknowledgements, nack backoff, ack timeout handling, retry topics, dead-letter topics, seek, pause/resume, subscription modes, and per-partition behavior. Applications with strict schemas need Avro, JSON, Protobuf, Protobuf-native, KeyValue, primitive schemas, and broker-side schema lookup semantics. Deployments with security requirements need token auth, mTLS, OAuth2, SASL, Kerberos/GSSAPI, Athenz, TLS provider choices, and a way to reason about crypto behavior without making every build drag every provider.
Those features are not independent checkboxes. They interact. Batching interacts with compression and send timeouts. Chunking interacts with sequence ids and redelivery. Transactions interact with acknowledgements. Reconnect interacts with producers, consumers, lookups, proxy routing, and failover. Observability has to describe the behavior without leaking secrets or flooding operators under churn.
That is why Magnetar’s feature work and architecture work are tied together. The more complete the surface becomes, the more important it is that the protocol state is explicit and that the runtime boundary is narrow. A broad client built on accidental state becomes harder to maintain as it becomes more useful. A broad client built on replayable state gives maintainers a better chance to keep feature growth from turning into hidden behavior.
Validation as architecture
Magnetar’s test strategy is intentionally layered because each layer catches a different class of bug.
The sans-io layer lets protocol behavior be tested without sockets or async tasks. The project testing documentation currently records more than 270 unit tests in magnetar-proto. Those tests exercise the state machine directly: bytes in, events out, transmit buffers out, state transitions checked.
Runtime integration tests exercise the glue that pure protocol tests cannot cover. The Tokio engine and Moonpool engine each have their own integration surfaces, and the project enforces runtime parity so one engine does not quietly fall behind the other.
The deterministic chaos pack targets the failure schedules that motivated the architecture in the first place: reconnect, failover, virtual-clock timers, TLS handshake ordering, in-flight publishes, broker migration, and adversarial network behavior. The Moonpool engine documentation describes a daily Moonpool seed sweep with 128 freshly rolled random seeds in parallel. The number is less important than what it represents: the client is being driven through schedules that a human would not write by hand.
Differential equivalence tests compare the two runtime worlds. They ask a direct question: if the same trace is run through Tokio and Moonpool, does the user see the same behavior?
End-to-end tests still matter. They keep Magnetar honest against a real Apache Pulsar broker. Simulation can make failures reproducible, but it does not replace compatibility with the system users actually deploy.
Taken together, these layers are not ceremony. They are part of the architecture. The point is not to collect large test numbers. The point is to put validation pressure exactly where distributed clients fail: protocol state, runtime boundaries, reconnect logic, timing, equivalence, and real broker behavior.
What reproducibility changes in practice
The practical value of deterministic simulation is not that it makes tests more sophisticated. It changes the debugging loop.
When a traditional end-to-end test fails because of timing, the first question is often whether the failure is reproducible at all. If it is not, engineers start adding sleeps, logs, larger timeouts, or retry loops. Some of those changes are useful. Many of them only move the failure somewhere else. The failure remains a story: “we saw it once in CI”.
With a deterministic simulation, the failure should become an artifact. The artifact is a seed, a trace, and a state transition that can be replayed. The question changes from “can we make it happen again?” to “what invariant did this schedule violate?”
That difference matters for a client library. Client bugs often sit at the boundary between user code and the broker. They are easy to misattribute. A publish can fail because the broker closed the connection, because the client lost an outcome, because the reconnect path rebuilt the producer too late, because an ack was delivered to the wrong waiter, because a timeout fired against stale session state, or because the runtime task died after the protocol had already transitioned.
If all of that behavior is spread across tasks, channels, timers, and sockets, a failure report is hard to compress. If the protocol state machine is explicit, the runtime boundary is narrow, and the same sequence can be replayed under controlled providers, the report can become much smaller: this seed, this trace, this event order, this invariant.
That is why Magnetar treats reproducibility as part of the client design. The point is not only to catch more bugs. The point is to make the bugs that do appear cheaper to understand and safer to fix.
It also changes code review. A reviewer can ask whether a behavior is covered at the right layer. Is this a pure protocol transition? Then the sans-io test should feed bytes and assert events. Is it runtime glue? Then Tokio and Moonpool need equivalent coverage. Is it an observable behavior shared by both engines? Then the differential harness should compare the event streams. Is it broker compatibility? Then an e2e test should touch a real Pulsar broker.
Those questions are more precise than “did we add tests?” They connect the test to the failure model.
What to inspect first
For engineers reading Magnetar for the first time, the best entry point is not a single file. It is the relationship between the documents and the code.
The README gives the public surface: what the client exposes, which features are implemented, how the Java-client parity matrix is tracked, which PIPs are supported, and how the two engines are positioned. That is the user-facing contract.
The architecture document explains the shape underneath that contract. It shows the crate topology, the dependency direction, the sans-io boundary, the driver loop, the event model, the producer and consumer paths, the runtime engines, TLS sites, schemas, PIP coverage, and the validation strategy. That is the system map.
The ADR series explains why the system has this shape. ADR-0004 records the sans-io split. ADR-0003 records the no-channels rule. ADR-0011 records clock injection. ADR-0010 records the Java parity decision. ADR-0019 records how the production Tokio engine and the Moonpool engine relate to the parity goal. ADR-0024 records the cross-runtime testing policy.
This matters because Magnetar is intentionally not only a code drop. It is a set of engineering decisions with a public audit trail. If you disagree with the design, the ADRs give you the right surface to discuss. If you want to contribute a feature, the parity matrix and validation policy tell you what “done” means. If you want to evaluate the simulation work, the Moonpool engine documentation and the differential harness show how the same protocol core is driven in different worlds.
The first release should therefore be read in two ways. As a Pulsar user, look at the client surface and the parity matrix. As a Rust infrastructure engineer, look at the state-machine boundary and the simulation strategy. The interesting part of Magnetar is that both views are meant to reinforce each other.
What the first public release gives you
This first public release is a technical entry point. It is for engineers who want to try Magnetar, read the architecture, inspect the ADRs, compare behavior with existing Pulsar clients, and help shape the Rust-native Pulsar ecosystem.
The high-level client API starts with the usual shape:
let client = PulsarClient::builder()
.service_url("pulsar://localhost:6650")
.build()
.await?;
let producer = client
.producer("persistent://public/default/orders")
.create()
.await?;That API is important because the client must be usable. But the release is also an invitation to inspect the lower layers. Read the README for the feature matrix. Read the architecture document for the sans-io design. Read the ADRs for the decisions behind no channels, swappable engines, clock injection, Java parity, Moonpool parity, TLS provider choices, and cross-runtime validation. Read the Moonpool engine documentation if you are interested in how deterministic simulation can be applied to a real network client.
The important point for this article is the direction: Magnetar is public so engineers can evaluate both the client surface and the engineering model behind it.
The inaugural technical entry point is available via v1.0.0, while the current state of the driver can be tracked in the v1.2.0 release.
Why this matters beyond Magnetar
Magnetar is an Apache Pulsar client. It is also a concrete example of how we want to build more Rust infrastructure software.
Distributed systems fail through state transitions. Some of those transitions are obvious and easy to test. Many are not. They happen when time advances in an unlucky place, when a retry overlaps with a close path, when a broker drops a connection after accepting part of the session, when a callback is registered just after a notification, or when a reconnect path rebuilds one handle but not another.
You can try to chase those bugs after they happen. Sometimes that is unavoidable. But the architecture can make the chase easier or harder.
A client with a pure protocol core, explicit events, injected time, replaceable I/O, deterministic seeds, and cross-runtime comparison gives engineers better tools. It lets them reduce a failure to a trace. It lets them replay that trace. It lets them ask whether a simulated failure corresponds to the production runtime. It lets them add an invariant where a silent state drift used to hide.
That is the direction Magnetar takes. Make the protocol explicit. Make the runtime replaceable. Make failures reproducible. Make correctness something the code has to demonstrate continuously.
Thank you to Pierre Zemb for Moonpool and for pushing this deterministic-simulation direction in Rust, and to the contributors and maintainers in the Rust Pulsar ecosystem whose work made the next step possible.
Magnetar is open source. The code, architecture notes, ADRs, and documentation live at github.com/CleverCloud/magnetar.
References
Magnetar
- Magnetar repository. Public Rust Apache Pulsar client repository. https://github.com/CleverCloud/magnetar
- Magnetar README. Public feature surface, Java client parity matrix, supported PIPs, engine coverage, and status. https://github.com/CleverCloud/magnetar/blob/main/README.md
- Magnetar architecture. Sans-io rationale, crate topology, driver loop, protocol state machine, and tests. https://github.com/CleverCloud/magnetar/blob/main/ARCHITECTURE.md
- Magnetar documentation index. Moonpool engine, testing, CLI, observability, logging, PIP features. https://github.com/CleverCloud/magnetar/blob/main/docs/README.md
- Moonpool engine documentation. Deterministic-simulation engine, chaos pack, differential harness, provider model. https://github.com/CleverCloud/magnetar/blob/main/docs/moonpool-engine.md
- Testing documentation. Test categories and validation commands. https://github.com/CleverCloud/magnetar/blob/main/docs/testing.md
- ADR-0004. Sans-io magnetar-proto and swappable I/O engines. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0004-sans-io-protocol-core.md
- ADR-0010. Full Java-client parity scope. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0010-v0-1-full-java-parity.md
- ADR-0024. Cross-runtime test and coverage policy. https://github.com/CleverCloud/magnetar/blob/main/specs/adr/0024-cross-runtime-test-and-coverage-policy.md
Ecosystem
- pulsar-rs. Existing Rust Apache Pulsar client project. https://github.com/streamnative/pulsar-rs
- Apache Pulsar. Distributed messaging and streaming platform. https://pulsar.apache.org/
- Apache Pulsar Java client. Reference client surface for many Pulsar users. https://pulsar.apache.org/docs/
Deterministic simulation
- FoundationDB testing. Deterministic simulation and fault injection lineage. https://apple.github.io/foundationdb/testing.html
- TigerBeetle TigerStyle. Assertion-first engineering style. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md
- TigerBeetle VOPR. Simulation testing approach. https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/internals/vopr.md
- Moonpool repository. Deterministic simulation for Rust systems, developed by Pierre Zemb. https://github.com/PierreZ/moonpool
- moonpool-sim. Simulation engine crate for Moonpool. https://crates.io/crates/moonpool-sim