Enabling PromQL queries with Erlenmeyer

Aurélien Hebert — Tue, 12 Oct 2021 07:28:09 +0000

You are using Clever Cloud to deploy applications runtimes or add-ons. But do you know that all Clever Cloud services are producing metrics? And that you already have access to all of them? Indeed, we register an agent on our side, whose job is to collect all local data.Once it's done, they are stored in a Time Series database called Warp10.

To simplify the adoption of Warp10, the OVHcloud Metrics team built the Erlenmeyer tool. In the first place, this tool enables the use of several open source protocols as PromQL, OpenTSDB, InfluxQL or Graphite to query Warp10 stored metrics.

Secondly, Erlenmeyer works as a proxy in front of a Warp10 cluster. Then we deployed a couple of Erlenmeyer instances in front of our Warp10 infrastructure, naturally providing new ways to query your application metrics.

In this post, we will review what Erlenmeyer brings and how to handle Clever Cloud services metrics with PromQL.

Erlenmeyer protocols

Erlenmeyer almost entirely enables PromQL queries, OpenTSDB, InfluxQL and some of the Graphite functions. Moreover, you can retrieve the documentation about each protocol implementation and usage on Erlenmeyer's GitHub:

On Clever Cloud, we deployed an Erlenmeyer in front of our Warp10 backend. This means all these open source protocols can be used to query our application Metrics. The host is https://PROTO-c1-warp10-clevercloud-customers.services.clever-cloud.com/PROTO with PROTO being:

for PromQL: prometheus,
for OpenTSDB: opentsdb,
for InfluxQL: influxql,
for Graphite: graphite.

To authenticate your request, you need to use the Warp10 READ TOKEN as the password of the basic authentication. You can find it in the metrics panel of any Clever Cloud application:

Retrieve the memory data with PromQL

As a first step, let's store the Warp10 READ TOKEN and an APP_ID as an environment variable:

export TOKEN=MY_AWESOME_TOKEN # Replace the value by your Clever Cloud Warp10 token
export APP_ID=app_test # Replace the value by one of your own Clever Cloud application

Get raw data

To retrieve the mem.used_percent in PromQL, we need to send an HTTP GET request to Erlenmeyer, with the TOKEN as basic AUTH. In that case, the PromQL GET request requires four parameters: query, start, end and step. Similarly, to learn more about the PromQL language, you can read the Prometheus documentation. In our case we will create a simple query to get the raw data:

api/v1/query_range?
query=mem.used_percent{app_id="$APP_ID"}&
start=1623289800&
end=1623311400&
step=120s

The query parameter is used for the PromQL queries: start and end are milliseconds timestamps corresponding to the time limit to get data. And the step one is the sampling parameter.

As data are collected around every minute, let's set at least twice the duration: 120s.

Now, to get the result, execute the following cURL url encoded:

curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Group by example

A group by query can be useful in the Clever Cloud case to merge some series (multiple deployments or instances). Thus we will group data based on the app_id labels in the following example.

We will re-use the stored variable READ_TOKEN and APP_ID. To compute a group by in our case we apply the max PromQL function grouped by the series app_id label:

api/v1/query_range?
query=max(mem.used_percent{app_id="$APP_ID"}) by (app_id)&
start=1623289800&
end=1623311400&
step=120s

Finally to get the result, execute the following cURL url encoded:

curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=max(mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D)%20by%20(app_id)&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Grafana PromQL data source

You may use Grafana to create you own dashboards, which is an open source interactive visualisation web application (to deploy a Grafana as a Clever Cloud application, we provide a github example and you can also follow this French blog post). Of course you can configure a PromQL data source to get Clever Cloud applications metrics:

create a Prometheus source,
then set Prometheus source host to https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus,
in addition activate the basic authentication with a user set to metrics and a password containing your read Warp10 token,
to conclude, click on Save and Test, your Prometheus data source should now be working!

Prometheus UI with remote_read endpoint

Like with Grafana, you can use Prometheus and plot Clever Cloud metrics in the Web UI. And as a matter of fact, simply fill your Prometheus configuration with a remote_read source:

api/v1/query_range?
remote_read:
  - url: "https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/remote_read/"
  basic_auth:
    username: "metrics"
    password: "TOKEN"

Restart Prometheus, then simply execute a Prometheus query in the UI as below:

{ __name__= "mem.used_percent", app_id="$APP_ID" }

Our backend is able to correctly resolve . which are invalid prometheus characters. Nonetheless the backend response will always translate those . values in series data (name or labels) into valid _.

To Sum Up

As you can see with this post, you can get metrics of Clever Cloud applications and add-ons. Presently you now have one more way to query them, as Erlenmeyer allows the use of PromQL. However the drawback is that you will have to update regularly your token, as they expire in a few days. We are working on a solution to improve the user experience. First, we will soon provide our customers with access to a Grafana with configured data source and already pre-filled dashboards. The token will be handled on our side and you will then be able to build your own graphs with PromQL or WarpScript queries.

prometheus Archives | Clever Cloud