Apache Pulsar Public Beta Release

Alexandre DUVAL — Tue, 05 Oct 2021 11:32:27 +0000

It's a big day for queuing and streaming enthusiasts today. Our engineering team is proud to release Apache Pulsar as an add-on on Clever Cloud in public beta! Apache Pulsar is a distributed messaging and streaming platform based on a "publish-subscribe" model. Clever Cloud Pulsar add-on provides one Pulsar namespace, with almost all management rights.

Common use cases

Replicating data among databases using Pulsar IO is commonly used to distribute change events from databases.
Parallel processing and workflows. You can efficiently distribute a large number of tasks among multiple workers (compressing text files, sending email notifications).
Data streaming from IoT devices. For example, a residential sensor can stream data to backend servers.
Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
Real-time event distribution. Events, raw or processed, may be made available to multiple applications across your team and organization for real-time processing.

Biscuit token support

At Clever Cloud, we are working on the Biscuit token, the foundation for our authorization systems. Consequently, we open-sourced our Biscuit token for Pulsar implementation which is directly pluggable to the Pulsar authentication and authorization system. This way, we can provide a Pulsar namespace with token-based authorizations. A generated access token enables you to operate your Pulsar namespace and its topics directly using the Pulsar Admin REST API. Using the power of Biscuit tokens, you can also attenuate your Biscuit token and create a new token that is only able to consume a single topic or multiple topics matching a regex. Here is an example with a topic prefix. The rights management we support is documented here.

Usage

We advise you to use pulsarctl provided by StreamNative. Here is an example to list topics in your namespace:

pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces topics $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE

As Biscuit is a token, you can use AuthenticationToken("") provided by clients libraries to authenticate to our clusters without any tweak.

C++ client
C# client
Go client
Java client (example)
Node.js client
Python client
Rust client (example)
WebSocket client

Storage policies

The Pulsar add-on comes with default retention policies and offload policies.

Retention policies

A freshly created Pulsar add-on has infinite retention policies, you can change it using:

# Example to set retention of namespace to 2 weeks and/or 100 GB
pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces set-retention $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE --time 2w --size 100G

Offload policies (cold storage)

Each Pulsar add-on is provided with a hidden Cellar add-on (object-storage add-on) used as a target for offloading data to cold storage. These policies are deactivated by default; you can enable offloading using:

# Example to set offload to run when hot storage is > 10G and put data to Cellar add-on as cold storage
pulsarctl \
--admin-service-url $ADDON_PULSAR_HTTP_URL \
--auth-params $ADDON_PULSAR_TOKEN \
--auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
namespaces set-offload-treshold $ADDON_PULSAR_TENANT/$ADDON_PULSAR_NAMESPACE 10G

Considering the previous retention of 100GB and the offload threshold of 10G. You will always have 10GB in the hot storage (Apache BookKeeper) and 90GB in Cellar. We will provide in the add-on dashboard the current storage usage of your hot and cold storage.

Pricing

Today, the pricing is in beta, meaning that it is subject to change depending on our client's usages. Please try the calculator to simulate the cost of your usage. Learn more on the Clever Cloud documentation

Alexandre DUVAL, Author at Clever Cloud