At Clever Cloud, we like to have some context about our application status and logs. That's why we have a lot of tools deployed to gain some observability. A lot of them are freely available for our customers (a log view, a metrics view or Grafana). Still, there are some tools that we use internally that could be helpful to many. We discovered recently glichtip, which is an open source error tracking. Glichtip is a great alternative to Sentry and less complex to deploy. We will add that Glichtip is compatible with the Sentry's open source SDK.

Context

We used Sentry to analyse the errors coming from our own application logs. We were still using the 9.1 version until we had too many trouble to keep it running. We then had to upgrade or find a new solution. All Sentry version post 9.1 require many more components to deploy (as Kafka, Snuba or Clickhouse). This is still possible to deploy it with a self hosted script. We wanted every component to be standalone outside of the main Sentry application, it meant that we had to provide authentication to all components. It was not a big issue, but it took us more time that what we first expected. And at the same time, we discovered a new project: Glitchtip.

Glitchtip

All started by a French tweet, about the discovery of a new tool: Glitchtip. This tweet said that Glitchtip is compatible to Sentry, free and easy to self host it, so it was evident that we were going to give it a try.

What does Glitchtip provide? Inside your application, with the Sentry SDK, you will transfer some of your error logs. Then the tool help you to track those errors, to better understand when a new version tool creates new errors, organize or search them... In a quick word, it's a very useful tool to debug your deployed software.

Clever Cloud deploy

Setup

First, we will download Glitchtip Docker project and customize it a bit to match how docker applications are run in Clever Cloud.

# Get latest Glitchtip release
git clone https://gitlab.com/glitchtip/glitchtip.git

# Go to the folder
cd glitchtip

Then open your favorite code IDE and open the Dockerfile:

# Remove
FROM glitchtip/glitchtip:latest

# And add instead:
FROM glitchtip/glitchtip:v1.12.2

COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

Here we will set-up the last Glitchtip release. When we were writing this blog it was the v1.12.2. You can find them here. We set up a new script to use docker-entrypoint.sh which will enable us to start or the Glitchtip web application or the Glitchtip worker. The docker-entrypoint.sh should contain the following script:

# docker-entrypoint.sh script:
#!/bin/bash

if [ "${START_TARGET}" = "worker" ]; then
  mkdir /tmp/sandboxed
	cd /tmp/sandboxed && nohup python -m http.server 8080 &
	/code/bin/run-celery-with-beat.sh
else
  /code/bin/start.sh
fi

# Don't forget to make it executable:
chmod u+x docker-entrypoint.sh

Deploy

Glitchtip comes with a Dockerfile, which means we will use a a docker runtime. It also requires a PostgreSQL Database and a Redis cache, which means we will deploy a PostgreSQL addon and a Redis addon. Adding a -o option in all create command line can be used to specify which organisation to use.

# Create the docker applications:
clever create --type docker glitchtip-web
clever create --type docker glitchtip-worker

# Create the PG addon
clever addon create postgresql-addon --plan s_sml --addon-version 14 glitchtip-postgres

# Link the addon
clever service link-addon glitchtip-postgres -a glitchtip-web
clever service link-addon glitchtip-postgres -a glitchtip-worker

# Create the Redis addon
clever addon create redis-addon --plan s_mono glitchtip-redis

# Link the addon
clever service link-addon glitchtip-redis -a glitchtip-web
clever service link-addon glitchtip-redis -a glitchtip-worker

Config

For now we will configure Glitchtip through environment variables. As we use several applications, let's deploy an addon configuration provider.

# Create a config addon:
clever addon create config-provider --plan std glitchtip-config

# Link the addon
clever service link-addon glitchtip-config -a glitchtip-web
clever service link-addon glitchtip-config -a glitchtip-worker

Now that we have deployed a config addon provider let's update its configuration. To do it go to the Clever Cloud console and then select the glitchtip-config. In the variable panel, select the expert mode and add the following required variable, (fill their value based on the provided commentary):

# In the config addon provider add:
CC_DOCKER_EXPOSED_HTTP_PORT="8080"
# Copy paste here the POSTGRESQL_ADDON_URI environment variable of the glitchtip-postgres addon 
DATABASE_URL="postgresql://user:password@host:port/database"
# Provide a valid email server
EMAIL_URL="smtp://email:password@smtp_url:port"
# Copy paste here the application domain name of the glitchtip-web application instead (should look like: https://app-da7a7a7a-da7a-4242-7a7a-da7a42427a7a.cleverapps.io)
GLITCHTIP_DOMAIN="MY_APP_DOMAIN_NAME"
# Copy paste here the REDIS_ADDON's environment variable of the glitchtip-redis addon 
REDIS_HOST= < REDIS_HOST >
REDIS_PASSWORD= < REDIS_PASSWORD > 
REDIS_PORT= < REDIS_PORT >
# Generate your own custom secret key for Glitchtip
SECRET_KEY= < choose a secret key >

Note that you can find more information about how to configure Glitchtip in its documentation.

Each docker application requires some custom configuration too:

#glitchtip-web custom env
clever env set CC_RUN_SUCCEEDED_HOOK "./manage.py migrate" -a glitchtip-web
clever env set START_TARGET "web" -a glitchtip-web
#glitchtip-worker custom env
clever env set START_TARGET "worker" -a glitchtip-worker

Deploy and run

Now that everything is set, it's time to deploy and run your Glitchtip application.

# Remove initial git
rm -r .git 

# Init a git repository
git init

# Add your files
git add .

# Commit the changes
git commit -m "clever init and deploy"

# Deploy the application
clever deploy -a glitchtip-web
clever deploy -a glitchtip-worker

# Open Glitchtip in your browser
clever open -a glitchtip-web

Your glitchtip is now almost ready to use. One last action is required: you need to create a super user to administrate Glitchtip.

# Connect in SSH to your Clever application:
clever ssh -a glitchtip-web

# Run 
./manage.py createsuperuser

Great job! You should now have an access to glitchtip admin page (available on your application /admin path). You can then create users or connect to your favorite social application to authenticate your users. You will find more information on the documentation. You can set your own keycloak on Clever Cloud and use it to connect to Glitchtip!

Connect Glitchtip to an application

We will reuse the akka application example we used for our scala exemple blogpost. And we will then see how we can forward our own error logs to Glitchtip.

Create a Glitchtip project

The first step will be to login with new root user (or any Glitchtip user you created or linked with a social application). Glitchtip will ask you to create your organisation. In our case let's set clever-cloud. As we want to monitore our application error's logs, we will need to create a new project directly in Glitchtip UI. Let's call it "myakka" and create it's associated "Scala-dev" team.

Once this project is create you will be able to get a DSN. We will load it later as environment variable for the akka application.

Create a Glitchtip Token

Go to Profile/auth token and craft a new glitchtip token with at least the following project rights: read, write and releases. Save it for later.

Setup the AKKA example application on Clever Cloud

The first step will be to deploy AKKA example application on Clever Cloud as mentionned in the scala exemple blogpost. Clone the project and instead of the main branch, use the blogpost-sentry one.

# Clone Clever Cloud akka example project:
git clone https://github.com/CleverCloud/scala-akka-http-postgres-example.git

# Checkout blogpost-sentry
git checkout blogpost-sentry

In this branch, we update a bit the application to connect it to Sentry. Then we send only the starting server HTTP logs with a custom status tag (success or failure). This is code we use in the blogpost-sentry branch:

# Register Sentry
Sentry.init(sentryConfig.sentryDsn)

# Send a message to Sentry with a tag
Sentry.getContext().addTag("status", status)
Sentry.capture(message)
Sentry.getContext().clear()

In addition, you have to add a .sentryclirc file. This file should contain:

[defaults]
url=GLITCHTIP_APP_DOMAIN_NAME
org=clever-cloud
project=myakka

Configure Glitchtip environment variables

Use the post_build.sh script as CC_POST_BUILD_HOOK. This will run the mandatory sbt flywayMigrate as well as create a new release on Sentry (based on the commit ID).

# Set CC_POST_BUILD_HOOK
clever env -a myakka set CC_POST_BUILD_HOOK "./post_build.sh"

You also need to set Glitchtip token and Glitchtip dsn as environnement variables:

# Set Glitchip required env variables
clever env -a myakka set SENTRY_AUTH_TOKEN < custom_glitchtip_token >  
clever env -a myakka set SENTRY_DSN < custom_glitchtip_dsn >  

Deploy the application

You can now deploy myakka project:

# Deploy myakka
clever deploy

Try it

As soon as this done, you should see your first issue on Glitchip! That's all, you can now add all the application you want to Glitchtip and enjoy this tool to centralise your application logs and errors. If you're used to this product or just try it feel free to share with us any feedback on Twitter.

In one of our previous posts, we announced the release of a dedicated Clever Cloud Grafana for all our metrics. What if now, you want to receive an alert when an application is in an abnormal state? Grafana provides an alerting service that is compatible with our Warp10 backend. Let’s see how we can create Slack alerts with our or any Grafana with a PromQL data source.

Grafana Notifier

First, before creating any alerts, we need to configure a Grafana notifier. You can have a full list of supported Grafana notifier here. However, you won't be able to configure notifiers that require an admin access, like email or SMTP. In this example, we will configure a slack channel to receive all Grafana alerts.

Configure Slack

To be able to receive alerts on Slack, Grafana will ask for a web hook. The process to create one with slack is here. At the end you should obtain an active incoming web hook.

Simply copy the web hook URL and then go to the alerting main panel (bell on the left) and add a new slack notification channel. Configure it as done below, within the field web hook URL, the one from the slack application. You can set your own custom name.

Before you validate, you can test your slack integration with the Test button. Without delay, in the configure slack channel, you will receive a test message.

Grafana Alerting

Now that we have enabled an alert notifier, let’s create an alert in Grafana.

Clever Cloud Available Metrics

We will start from one of the template dashboards we release with Grafana. We deployed a preview application on a Clever Cloud organisation.

For the rest of this post, we will work on the load average metrics. As we can see below, it’s a simple PromQL query. However, it includes custom dashboard variables APP_ID and DEPLOY_ID that are not supported in Grafana alerting!

Before enabling the alerting, we will keep somewhere this graph query system.load1_per_cpu{app_id="APP_ID"}. I will only replace the variable APP_ID by its value, which is available on top of the dashboard.

Then in order not to have our custom alerts removed at each update of the Clever Cloud dashboard templates, create a new dashboard.

Also save it in a dedicated dashboard which is not Clever Cloud, in our example, it’s alerting.

Finally, in this new dashboard create the same load graph with the query system.load1_per_cpu{app_id="APP_ID"}. As well as seen previously replace APP_ID by your own application ID. For the data source, select the PromQL one.

Set Up The Alerting

At this instant we have a new graph where we can set up a first alert on the load of our application. For that, go on the Alert panel, and configure the alert based on the parameter you want. Quickly, to get an alert working, I set up the condition to check if my metrics are below the value 0.2.

Then still on the Alert panel add the notifier you created in the first section. Here, you can customise the alert you will receive in slack: adding a message and custom tags.

Once you have set up your alert, you can test the rule with the test rule button.

Next all you have to do is to wait for an alert to appear on Slack! Besides each alert state change will be ploted on the alert graph. After about five minutes, as I was clearly matching my alert state condition, I got two states change: one pending and one alerting.

As soon as the alert state happens, a slack message is sent through the notifier.

Finally, I would like to retrieve a more ‘normal’ state. To do it, I update my alert condition and set it to above 0.8, which is a more classical value for an alert threshold for the load metrics.

Instantly or almost, the state of the alert should change and comes OK!

Immediately after Grafana also send a message to slack to end the issue!

To Sum Up

You should now be able to configure your own alerts on Slack for Grafana! You can play with the pending time, the alert condition, when to trigger the alert, the alert message the tags and so on. In fact, all you need is a working PromQL query. Nonetheless remember that the Grafana alerting doesn’t support dashboard variables. Other than that, you should be able to custom the alert as you wish!

I hope you enjoy and feel free to share with us your thought on the Alerting and the Metrics product. We will be happy to have your feedback. You can contact me directly on twitter, or send us a message through the metrics overview comments.

At Clever Cloud, we think it’s important to offer access to our metrics for our users. That’s why you can already query all our metrics, and can choose between WarpScript or PromQL. Besides, we wanted to provide more than only a metrics access to our users! We wanted to give a tool that will make you able to explore and visualize the state of your Clever Cloud applications. Internally, we’re using Grafana a lot to build monitoring dashboards, what if we can re-use this tool and build dedicated dashboards? That’s how we got the idea to release a Grafana for all of you!

In this blog post, we will talk about the Grafana tool. Then we will see how we deployed it. After that we will detail all actions to do to activate our Grafana Beta on your organisation. We will conclude with our dashboard’s contents.

About Grafana

Grafana is an open-source tool which enables among others applications metrics, alerting or log visualization. Thus Grafana’s aim is to ease to create efficient dashboards. Grafana also comes with multiple kinds of tiles or themes and provide many data sources. In our case, we will use only Warp10 and Prometheus.

Grafana at Clever Cloud

For the one who wants to utilize Grafana, it's easy to deploy it today on Clever Cloud. For the French reader you can follow this blog post. And also we released some scripts to automate it on Github. When we wanted to deploy a Grafana for our customers, we decided to re-use the same scripts. This is why our deployed instance runs as a Clever Cloud runtime application. We only activate an option as force https and add some scalability.

Our Grafana Architecture

As Clever Cloud, Grafana furnishes organisation too. Then it’s easy to link both and make it available to our users. Next, we created an API endpoint to connect our Grafana to our console. All data sources exposed (Warp10 and Prometheus) are handled on our side. This means that no users can change them. However we cyclically refresh all tokens. In the same way, we regularly update all custom dashboards we provide.

Everyone features editor rights in their organisations. This means that you can customise all dashboards as you want. We will see, later in this post, how to not generates conflicts with our templates.

Activate the Grafana Beta Organisation

To activate our beta, you need to go to your Clever Cloud organisation view in the Clever Cloud console. Then you will have access to a page called Metrics in Grafana.

Available Dashboards

We built three unique dashboards for you! First we will start by a tour of an organisation metrics. Then we will get more into the details of a runtime view. And finally we will see what we bring for the add-ons. Most graph of all Grafana dashboards contains a small information tile. There, you can retrieve a bit of help and description.

Clever Cloud Organisation

The organisation view is the home dashboard in Grafana. It provides some insights about your organisation. Here, we expose some key information about the applications, the flavours and the image types. Besides, as a home, you will access some links here. You can go to all Clever Cloud template dashboards as well as the ones you set as favourite in Grafana. You can also find out direct link to reach an application.

A tile I want to explain a bit more is the deployment state graph. What is done on this tile? Every new deployment generates a new state. We plot only each one occurring during the Grafana time range. And finally this dashboard includes a metrics counting the average requests rate per minutes, computed from the access logs.

Application Runtime Dashboard

I will now introduce the runtime dashboard. In this dashboard we retrieve several parts. First we start with a quick overview of your application state. Then we will have access to more details sections. Those sections will be about classical system metrics as the CPU, the load, the memory, the disks and the networks. Moreover in each segment, you will find many graphs as the usage or when the resource is saturated. To compute this last metrics, we apply a linear regression.

Add-on Dashboard

The last dashboard contains the add-on metrics view. A first panel contains system-related metrics as the CPU, the memory, the disks or the network. A second one show the database metrics. This is where you will retrieve data as connections count, transactions, errors, deadlocks or tuple operations. However we collect custom data on a limited pool of add-on. This means there are only four of them: MySQL, PostgreSQL, Redis and MongoDB.

Customize Your Grafana Organisation

You have full right to edit all dashboard saved in your Grafana organisation. This means you can edit our templates. However we will update them every once in a while. As a result all your dashboards will always be up to date.

We store all our templates on the Clever Cloud folder. However, you can create custom dashboards anywhere else, we will never update them!

The Grafana manage view lists all folders and files. Besides, this view is helpful to organise and to create new folders!

At any time you can reset our templates. For example in case you made unwanted changes. In that case, you can reach the Grafana page of your Clever Cloud organisation. Then click on reset all dashboards. Finally, all Clever Cloud ones will be back to their initial states.

Share Your Experience

Thank you for reading this post, we will be more than happy to get some feedback! How do you use our available data? Is there something missing for your use case on what we have provided? I hope you will enjoy this new data ride!

Superset is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data.

How to deploy Superset

# Clone last superset release
git clone --depth 1 -b 0.38.1  https://github.com/apache/superset.git

# Move in superset repository
cd superset

# Create the python application
clever create --type python Superset

# Create the PostgreSQL instance                                
clever addon create postgresql-addon --plan m SupersetPG
    
# link the addon 
clever service link-addon SupersetPG

Superset environment

# Use gunicorn as python mode 
clever env set CC_PYTHON_BACKEND gunicorn

# Start the app using create_app() 
clever env set CC_PYTHON_MODULE superset.app:create_app()

# Python version to use
clever env set CC_PYTHON_VERSION 3.7

# Load all superset requirements
clever env set CC_PIP_REQUIREMENTS_FILE requirements/base.txt

# Application port
clever env set PORT 8080

# PYTHONPATH
clever env set PYTHONPATH /home/bas/YOUR_APP_ID/config/

# App secret key
clever env set SECRET_KEY A_SECRET_KEY

# Post build commands
clever env set CC_POST_BUILD_HOOK ./init.sh

# Superset admin password
clever env set ADMIN_PASSWORD AN_ADMIN_PASSWORD

Configure superset

import os
# Superset specific config
ROW_LIMIT = 5000

SUPERSET_WEBSERVER_PORT = os.getenv("PORT")
SUPERSET_WEBSERVER_ADDRESS = "0.0.0.0"

# Flask App Builder configuration
# Your App secret key
SECRET_KEY = os.getenv("SECRET_KEY")

# The SQLAlchemy connection string to your database backend
# This connection defines the path to the database that stores your
# superset metadata (slices, connections, tables, dashboards, ...).
# Note that the connection information to connect to the datasources
# you want to explore are managed directly in the web UI
SQLALCHEMY_DATABASE_URI = os.getenv("POSTGRESQL_ADDON_URI")

# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
# Add endpoints that need to be exempt from CSRF protection
WTF_CSRF_EXEMPT_LIST = []
# A CSRF token that expires in 1 year
WTF_CSRF_TIME_LIMIT = 60 * 60 * 24 * 365

# Set this API key to enable Mapbox visualizations
MAPBOX_API_KEY = ''

Superset build scipt

#!/bin/bash

# Load configuration file in PYTHONPATH
mkdir $APP_HOME/config
cp clever_config.py $APP_HOME/config/superset_config.py

# Create an admin user (you will be prompted to set username, first and last name before setting a password)
superset fab create-admin \
    --username admin \
    --firstname admin \
    --lastname admin \
    --email admin@admin.com \
    --password ${ADMIN_PASSWORD}

# Upgrade the database
superset db upgrade

# Loading examples data set (optional)
superset load_examples

# Create default roles and permissions
superset init

# Build frontend
cd superset-frontend 
npm install -f --no-optional
npm run build

Superset requirements

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements/base.txt

pip3 freeze > requirements.txt 

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements.txt

# gunicorn valid versions are located in setup.py file
echo "gunicorn>=20.0.2, <20.1" >> requirements.txt

Deploy

chmod u+x init.sh

# Add your files
git add .

# Create the first commit
git commit -m "clever init"

# Deploy the application
clever deploy

# Run the application
clever open