logs Archives | Clever Cloud

GlitchTip on Clever Cloud, a Sentry alternative

Aurélien Hebert — Tue, 21 Jun 2022 15:13:11 +0000

At Clever Cloud, we like to have some context about our application status and logs. That's why we have a lot of tools deployed to gain some observability. A lot of them are freely available for our customers (a log view, a metrics view or Grafana). Still, there are some tools that we use internally that could be helpful to many. We discovered recently glichtip, which is an open source error tracking. Glichtip is a great alternative to Sentry and less complex to deploy. We will add that Glichtip is compatible with the Sentry's open source SDK.

Context

We used Sentry to analyse the errors coming from our own application logs. We were still using the 9.1 version until we had too many trouble to keep it running. We then had to upgrade or find a new solution. All Sentry version post 9.1 require many more components to deploy (as Kafka, Snuba or Clickhouse). This is still possible to deploy it with a self hosted script. We wanted every component to be standalone outside of the main Sentry application, it meant that we had to provide authentication to all components. It was not a big issue, but it took us more time that what we first expected. And at the same time, we discovered a new project: Glitchtip.

Glitchtip

All started by a French tweet, about the discovery of a new tool: Glitchtip. This tweet said that Glitchtip is compatible to Sentry, free and easy to self host it, so it was evident that we were going to give it a try.

What does Glitchtip provide? Inside your application, with the Sentry SDK, you will transfer some of your error logs. Then the tool help you to track those errors, to better understand when a new version tool creates new errors, organize or search them... In a quick word, it's a very useful tool to debug your deployed software.

Clever Cloud deploy

Setup

First, we will download Glitchtip Docker project and customize it a bit to match how docker applications are run in Clever Cloud.

# Get latest Glitchtip release
git clone https://gitlab.com/glitchtip/glitchtip.git

# Go to the folder
cd glitchtip

Then open your favorite code IDE and open the Dockerfile:

# Remove
FROM glitchtip/glitchtip:latest

# And add instead:
FROM glitchtip/glitchtip:v1.12.2

COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

Here we will set-up the last Glitchtip release. When we were writing this blog it was the v1.12.2. You can find them here. We set up a new script to use docker-entrypoint.sh which will enable us to start or the Glitchtip web application or the Glitchtip worker. The docker-entrypoint.sh should contain the following script:

# docker-entrypoint.sh script:
#!/bin/bash

if [ "${START_TARGET}" = "worker" ]; then
  mkdir /tmp/sandboxed
	cd /tmp/sandboxed && nohup python -m http.server 8080 &
	/code/bin/run-celery-with-beat.sh
else
  /code/bin/start.sh
fi

# Don't forget to make it executable:
chmod u+x docker-entrypoint.sh

Deploy

Glitchtip comes with a Dockerfile, which means we will use a a docker runtime. It also requires a PostgreSQL Database and a Redis cache, which means we will deploy a PostgreSQL addon and a Redis addon. Adding a -o option in all create command line can be used to specify which organisation to use.

# Create the docker applications:
clever create --type docker glitchtip-web
clever create --type docker glitchtip-worker

# Create the PG addon
clever addon create postgresql-addon --plan s_sml --addon-version 14 glitchtip-postgres

# Link the addon
clever service link-addon glitchtip-postgres -a glitchtip-web
clever service link-addon glitchtip-postgres -a glitchtip-worker

# Create the Redis addon
clever addon create redis-addon --plan s_mono glitchtip-redis

# Link the addon
clever service link-addon glitchtip-redis -a glitchtip-web
clever service link-addon glitchtip-redis -a glitchtip-worker

Config

For now we will configure Glitchtip through environment variables. As we use several applications, let's deploy an addon configuration provider.

# Create a config addon:
clever addon create config-provider --plan std glitchtip-config

# Link the addon
clever service link-addon glitchtip-config -a glitchtip-web
clever service link-addon glitchtip-config -a glitchtip-worker

Now that we have deployed a config addon provider let's update its configuration. To do it go to the Clever Cloud console and then select the glitchtip-config. In the variable panel, select the expert mode and add the following required variable, (fill their value based on the provided commentary):

# In the config addon provider add:
CC_DOCKER_EXPOSED_HTTP_PORT="8080"
# Copy paste here the POSTGRESQL_ADDON_URI environment variable of the glitchtip-postgres addon 
DATABASE_URL="postgresql://user:password@host:port/database"
# Provide a valid email server
EMAIL_URL="smtp://email:password@smtp_url:port"
# Copy paste here the application domain name of the glitchtip-web application instead (should look like: https://app-da7a7a7a-da7a-4242-7a7a-da7a42427a7a.cleverapps.io)
GLITCHTIP_DOMAIN="MY_APP_DOMAIN_NAME"
# Copy paste here the REDIS_ADDON's environment variable of the glitchtip-redis addon 
REDIS_HOST= < REDIS_HOST >
REDIS_PASSWORD= < REDIS_PASSWORD > 
REDIS_PORT= < REDIS_PORT >
# Generate your own custom secret key for Glitchtip
SECRET_KEY= < choose a secret key >

Note that you can find more information about how to configure Glitchtip in its documentation.

Each docker application requires some custom configuration too:

#glitchtip-web custom env
clever env set CC_RUN_SUCCEEDED_HOOK "./manage.py migrate" -a glitchtip-web
clever env set START_TARGET "web" -a glitchtip-web
#glitchtip-worker custom env
clever env set START_TARGET "worker" -a glitchtip-worker

Deploy and run

Now that everything is set, it's time to deploy and run your Glitchtip application.

# Remove initial git
rm -r .git 

# Init a git repository
git init

# Add your files
git add .

# Commit the changes
git commit -m "clever init and deploy"

# Deploy the application
clever deploy -a glitchtip-web
clever deploy -a glitchtip-worker

# Open Glitchtip in your browser
clever open -a glitchtip-web

Your glitchtip is now almost ready to use. One last action is required: you need to create a super user to administrate Glitchtip.

# Connect in SSH to your Clever application:
clever ssh -a glitchtip-web

# Run 
./manage.py createsuperuser

Great job! You should now have an access to glitchtip admin page (available on your application /admin path). You can then create users or connect to your favorite social application to authenticate your users. You will find more information on the documentation. You can set your own keycloak on Clever Cloud and use it to connect to Glitchtip!

Connect Glitchtip to an application

We will reuse the akka application example we used for our scala exemple blogpost. And we will then see how we can forward our own error logs to Glitchtip.

Create a Glitchtip project

The first step will be to login with new root user (or any Glitchtip user you created or linked with a social application). Glitchtip will ask you to create your organisation. In our case let's set clever-cloud. As we want to monitore our application error's logs, we will need to create a new project directly in Glitchtip UI. Let's call it "myakka" and create it's associated "Scala-dev" team.

Once this project is create you will be able to get a DSN. We will load it later as environment variable for the akka application.

Create a Glitchtip Token

Go to Profile/auth token and craft a new glitchtip token with at least the following project rights: read, write and releases. Save it for later.

Setup the AKKA example application on Clever Cloud

The first step will be to deploy AKKA example application on Clever Cloud as mentionned in the scala exemple blogpost. Clone the project and instead of the main branch, use the blogpost-sentry one.

# Clone Clever Cloud akka example project:
git clone https://github.com/CleverCloud/scala-akka-http-postgres-example.git

# Checkout blogpost-sentry
git checkout blogpost-sentry

In this branch, we update a bit the application to connect it to Sentry. Then we send only the starting server HTTP logs with a custom status tag (success or failure). This is code we use in the blogpost-sentry branch:

# Register Sentry
Sentry.init(sentryConfig.sentryDsn)

# Send a message to Sentry with a tag
Sentry.getContext().addTag("status", status)
Sentry.capture(message)
Sentry.getContext().clear()

In addition, you have to add a .sentryclirc file. This file should contain:

[defaults]
url=GLITCHTIP_APP_DOMAIN_NAME
org=clever-cloud
project=myakka

Configure Glitchtip environment variables

Use the post_build.sh script as CC_POST_BUILD_HOOK. This will run the mandatory sbt flywayMigrate as well as create a new release on Sentry (based on the commit ID).

# Set CC_POST_BUILD_HOOK
clever env -a myakka set CC_POST_BUILD_HOOK "./post_build.sh"

You also need to set Glitchtip token and Glitchtip dsn as environnement variables:

# Set Glitchip required env variables
clever env -a myakka set SENTRY_AUTH_TOKEN < custom_glitchtip_token >  
clever env -a myakka set SENTRY_DSN < custom_glitchtip_dsn >

Deploy the application

You can now deploy myakka project:

# Deploy myakka
clever deploy

Try it

As soon as this done, you should see your first issue on Glitchip! That's all, you can now add all the application you want to Glitchtip and enjoy this tool to centralise your application logs and errors. If you're used to this product or just try it feel free to share with us any feedback on Twitter.

Access Logs are now available from Clever Tools

Laurent Doguin — Thu, 09 Apr 2020 11:35:00 +0000

Yes you read it right, you can now access your access logs. How you ask? With the latest release of Clever Tools. But before we see how it works, let me explain what access logs are. Access logs contain all incoming requests to a web server. They are available in different formats. One of the most common being CLF which stands for Common Log Format. Here's an example where you can find the IP of the client, date, HTTP verb, URL target, status code and size of the answer: 255.255.255.255 - - [06/Feb/2020:07:59:22 +0100] "GET /aget/to/your/beautiful/website -" 200 1453

How to get your access logs

To get your access logs you need to use clever-tools and run this in a linked application: clever accesslogs You will get all the requests that came to this app in the last hour. This also work for most dedicated add-ons using --addon followed by the add-on id: clever accesslogs --addon addon_41c5a60f-11c6-442a-9803-f4604fe8638b You can also choose a date span using --after and --before like so: clever accesslogs --after 2020-03-27 --before 2020-03-29 Any date you give must follow ISO 8601. You could for instance have a microsecond precision like this: clever accesslogs --after 2020-03-29T11:01:02.120000Z --before 2020-03-29T11:01:02.123456Z You can also change the format of the logs with -F (or --format): clever accesslogs --after 2020-03-28 --before 2020-03-29 -F json Here's an example of JSON entry, it contains all of the information currently available for each request:

{
  "t": "2020-03-10T11:24:34.220324Z",
  "a": "app_87d748d4-3c7b-4baa-8f52-c748e1123ab3",
  "adc": "par-cleverapps-adc-n6",
  "o": "user_51f7098f-604e-4d9e-8fb6-4026fd1767bb",
  "i": "0805330e-cff6-4fd5-87c0-26974d8bcb4e",
  "ipS": "255.255.255.255",
  "pS": 38566,
  "s": {
    "lt": 48.6719,
    "lg": 1.3002,
    "ct": "Aunay-sous-Crecy",
    "co": "FR"
  },
  "ipD": "255.258.258.258",
  "pD": 11810,
  "d": {
    "lt": 48.7967,
    "lg": 2.3338,
    "ct": "Cachan",
    "co": "FR"
  },
  "vb": "GET",
  "path": "/status-codes/500",
  "bIn": 411,
  "bOut": 443,
  "h": "app-87d748d4-3c7b-4baa-8f52-c748e1123ab3.cleverapps.io",
  "rTime": "3ms",
  "sTime": "115μs",
  "scheme": "HTTPS",
  "sC": 500,
  "sT": "Internal Server Error",
  "w": "WRK-01",
  "r": "b6bcab8a-c6d0-4546-9000-57b43d009a49",
  "tlsV": "TLS1.3"
}

You will find the details for every field in our access logs documentation.

What you can do with your logs

Now let's say you want to know what are the most returned status codes from a log file, here's how you can do it:

clever accesslogs --after 2020-03-10 --before 2020-03-20 -F json | jq .sC | sort | uniq -c | sort -nr

Here we use the JSON format and jq to select the field we want, then sort and uniq to get a proper output, which would yield something like:

The first column being the number of hits and the second being the status code. Now for something a bit more practical, here's a script that allows you to split access logs for each day between two dates:

#!/bin/bash

input_start=2020-01-01
input_end=2020-03-27

startdate=$(date -I -d "$input_start") || exit -1
enddate=$(date -I -d "$input_end")     || exit -1

d="$startdate"
while [ "$d" != "$enddate" ]; do 
  clever accesslogs --alias prod --after $d --before $(date -I -d "$bv d + 1 day") -F clf > access-$d.out
  d=$(date -I -d "$d + 1 day")
done

It's hard to visualize so much data if you don't have the proper tools. So I really invite you to try GoAccess. Combine the script above and this command goaccess access* -o report.html --log-format=COMMON --anonymize-ip. You will get this neat, anonymized HTML report:

Viusalize CLF logs with GoAccess

There are of course lots of other tools you can use to visualize logs, like our latest Elastic add-on for instance :) Or you can go straight into the metrics tab of your application or add-on and start playing with Warp10. This is actually what we use when you see the live map under the Overview tab of your application. We hope you'll find this new feature practical and as always welcome your input to make this better. Happy coding!

Introducing Traffic Overview in Real-Time

Clément Nivolle — Thu, 19 Dec 2019 17:32:00 +0000

Time for some exciting new features! We've been working for a while now on giving more insights to your application's activity. Is your application used? Where? Which IPs are accessing it? Even the response time is here! In this post, I am presenting to you what's new and how to use it at best.

Console: new tiles and improvements

We've been building new tiles for both organizations and overview pages. These new tiles rely on web components which are open-source. You can embed each of these components easily in your product to show to your customers. You can find each of our components here: https://www.clever.cloud/developers/clever-components/.

Here's an example of our new inputs, with copy-to-clipboard button.

Please note that these are in beta, so bear with us as we fix possible hiccups over time!

We are also developing a new type of ACLs, built on top of our next-gen auth product, allowing you to make integrations easier. Stay tuned!

The new "overview".

Geolocation of requests

This one may be the most obvious change: there is a near-real-time map that shows dots for requests and their location around the world. In the organization's overview, you can see requests of all your apps combined.

There is also a heatmap that shows the density of the traffic location over 24 hours. Real-time is nice, but having a track of what's happened can be pretty handy.

The heatmap view

HTTP Response codes

Here we show you the distribution of HTTP status codes in the responses of your apps. It is displayed over 24h. If you click on a status code in the legend, it will hide it from the chart. Pro tip: If your shift+click on a HTTP code in the legend, it will unselect all the other codes.

HTTP Requests

This graph represents the variation of HTTP requests on your app over 24h, with a time window of a 2, 3 or 4 hours for each graph, (depending on your windows size, responsive ftw).

Access logs & tools

What is an access log?

An access log records all requests that people or bots have requested from an application. When you are using Clever Cloud, your application's requests transit via our load-balancers. They receive the incoming network traffic and route it to each application. For legal reasons, we have to keep track of each request for at least a year.

Retrieving access logs, before

When a customer needed the access logs, the original process was to dive into a vast logfile and find it ourselves. But now, each Clever Cloud user can retrieve its access logs, and much more.

Retrieving access logs, today

We introduced Metrics a few years ago, a tool built into the web console to give monitoring data to our users. Metrics relies on Warp10, a time-series database built for this kind of usage. The access logs are now stored into Warp10, as it's a part of Metrics.

On the Metrics page, you'll find your metrics token and URL, allowing you to request the warp10 database with Quantum (it's a (WarpScript™ IDE).

You can open Quantum within Metrics, and click on "access logs" located in the "Explore your data" section. Here's an example:

[
  ''
  'accessLogs'
  { 'app_id' '' }
  NOW 5 s
] FETCH

As access logs are still quite complex to retrieve that way, we are still working on a better way to make it accessible to you. Stay tuned for the next iterations!