In a previous post, we explained why you should switch from Google Analytics to Matomo for your audience measurement. Some of our customers have decided to take this step, and Clever Cloud has helped them get to grips with the solution. Based on this experience, find out in this article how to maximize the use of our Matomo addon on Clever Cloud.

Selecting the optimal instance size

As soon as the Matomo addon is created, Clever Cloud deploys three optimized instances (a PHP runtime, a MySQL addon and a Redis addon). Initially, they are sized to the minimum server requirements in order to evolve according to your needs, thus ensuring cost optimization. If you need to make adjustments (for example, if you start to experience errors or slowdowns when launching some processes), our integrated Grafana is your best ally for monitoring your metrics and adjusting your resources accordingly.

Easy access to your metrics

Thanks to the intuitive dashboard for each application, you're provided with a clear view of it all. We've recently integrated a metrics tile into this dashboard, allowing you to track RAM and CPU consumption in real time from the console.

For a more in-depth analysis, a simple click on the Grafana logo from the tile gives you access to all your application's metrics.

In addition, Matomo's PHP instance logs are always there to provide you with additional information.

All about auto-scalability

Self-scalability on Clever Cloud is a major advantage, giving you peace of mind and flexible billing. With Matomo, this feature guarantees a smooth experience even in times of high demand. Keep in mind certain specifics, such as the occasional need for certain extensions, and you'll you make the most of this feature.

Perks from enabling auto-scalability on the PHP instance

With auto-scalability, you can use Matomo smoothly and adaptively, even during peak periods! Benefit from powerful on-demand resources, while paying only for the consumption/second. Once you've completed your intensive tasks, your instance returns to an optimal size, guaranteeing efficiency and savings. So there's no need to reserve a large infrastructure to meet occasional needs.

When to disable auto-scalability?

Clever Cloud provides a service that can be adapted to your needs, with the option of limiting or completely disabling instance auto-scalability. In some cases, it may be advisable to deactivate it from time to time.

As Matomo is a managed service, some configuration options are unalterable to guarantee optimal operation and performance. As part of Matomo's continuous improvement, we have integrated the Google Analytics data import extension in response to common customer requests. This extension, generally used for a one-off migration from Google Analytics to Matomo, is deactivated by default.

If you activate the extension and start importing data, and your application scales, your instance reboots and redeploys the application... with the default configurations. Restarting will therefore disable this extension and kill the import process at the same time. If you decide to use an extension that appears as disabled by default in our Matomo, we recommend the following procedure:

1. Disable auto-scalability for your Matomo PHP instance
2. Increase the instance size until the process is complete
3. Restart instance
4. Once the instance has been redeployed, activate the extension in Matomo

Now you're ready to use the extension, which will remain active until the next server restart.

Matomo, with its MySQL addon, is designed to grow with you. While the database is created with sufficient capacity at start-up, Matomo will quickly exploit its potential. To avoid surprises, why not set up Grafana alerts to monitor memory levels?

Creating alerts

Among its many features, Grafana lets you create alerts on the metrics of your choice. You'll find a tutorial here to show you how to create your alerts from Grafana to Slack, which you can also apply to other contact points like Discord, Teams, etc.

We recommend that you leave around 20% disk space available, as the MySQL instance will need disk to migrate.

Migrating the database

Although scalability has its peculiarities for databases, Clever Cloud provides a smooth and intuitive manual solution. When you receive the alert, a simple connection to the Console on your MySQL addon dashboard will enable you to launch our one-click migration tool. This operation can take from a few seconds to a few minutes, depending on the size of your database.

💡 Tip: Include a direct link to the migration tab in your Grafana alert message. This will give you instant access to the migration tool, making the process even easier.

Leveraging Matomo and Clever Cloud

Although a managed service is pre-configured, you have Matomo administrator status and can use it as such. Here are just a few examples of what you can do with Matomo and Clever Cloud.

Adding new Matomo users

All our managed services use Clever Cloud SSO, enabling you to access deployed addons from your platform credentials. Perhaps you'd like to add users to Matomo without giving them access to the platform.

Matomo lets you add users with different levels of permissions to the application. You can invite them by email directly from the Matomo interface. Users invited by this method will not have access to Matomo's system settings or extensions, unlike Clever Cloud users.

Elevating permission levels

If you have added a user from the Matomo interface, but now wish to add them to your Clever Cloud organization with the same email address, access to Matomo will be denied for security reasons, since this user already exists in the database.

To avoid duplication, simply delete the Matomo user from your database.

1. Launch your MySQL addon's PHPMyAdmin interface
2. Click on your database name (not schema)
3. Search by keyword user and choose the table matomo_user
4. Delete the user you wish to add to the Clever Cloud organization

Now you can invite the user to your organization with his email address, giving him administration rights on the instance.

Measure the audience of any application

On Clever Cloud, you can separate your collaborative environments by creating organizations to which you invite members. Organizations remain isolated, ensuring that no member has uninvited access to an organization's applications and addons.

Matomo makes it possible to track apps to which a simple script has been added to the head. So, even if an application is deployed within a different organization, Matomo offers a unified and complete view of its audience tracking as soon as the application contains this script.

Optimizing role separation

You can set up a dedicated organization for the Matomo administrator of your company or association. From this organization, the administrator can monitor the audience of applications hosted in other organizations, even if these are inaccessible to him/her.

To implement this configuration, the administrator simply provides the appropriate script to be integrated into the head tag of each application. Once integrated, the tracking data will be visible in Matomo. This process ensures that developers have no access to Matomo, while restricting the Matomo administrator's access to applications.

📢 New: Migrate Google Analytics data to Matomo

As we mentioned, you can now use the extension to migrate your Google Analytics data to Matomo. It might be challenging to determine the right database sizing , due to Google Analytics' opacity regarding its metrics.

With this in mind, we suggest you keep an eye on your database metrics and opt for a gradual increase in storage capacity. To facilitate this procedure, Matomo provides a detailed guide to migration.

Conclusion

Clever Cloud not only facilitates the deployment of on-demand applications and services in the form of add-ons. In addition, we include an integrated suite for all our customers, offering in-depth monitoring of application and database activity, as well as automatic and adaptive scaling capabilities.

Whether you've opted for Matomo or one of our other add-ons, we're always listening to our customers' needs, so we can continually improve the services we offer. We look forward to hearing from you!

At Clever Cloud, we think it’s important to offer access to our metrics for our users. That’s why you can already query all our metrics, and can choose between WarpScript or PromQL. Besides, we wanted to provide more than only a metrics access to our users! We wanted to give a tool that will make you able to explore and visualize the state of your Clever Cloud applications. Internally, we’re using Grafana a lot to build monitoring dashboards, what if we can re-use this tool and build dedicated dashboards? That’s how we got the idea to release a Grafana for all of you!

In this blog post, we will talk about the Grafana tool. Then we will see how we deployed it. After that we will detail all actions to do to activate our Grafana Beta on your organisation. We will conclude with our dashboard’s contents.

About Grafana

Grafana is an open-source tool which enables among others applications metrics, alerting or log visualization. Thus Grafana’s aim is to ease to create efficient dashboards. Grafana also comes with multiple kinds of tiles or themes and provide many data sources. In our case, we will use only Warp10 and Prometheus.

Grafana at Clever Cloud

For the one who wants to utilize Grafana, it's easy to deploy it today on Clever Cloud. For the French reader you can follow this blog post. And also we released some scripts to automate it on Github. When we wanted to deploy a Grafana for our customers, we decided to re-use the same scripts. This is why our deployed instance runs as a Clever Cloud runtime application. We only activate an option as force https and add some scalability.

Our Grafana Architecture

As Clever Cloud, Grafana furnishes organisation too. Then it’s easy to link both and make it available to our users. Next, we created an API endpoint to connect our Grafana to our console. All data sources exposed (Warp10 and Prometheus) are handled on our side. This means that no users can change them. However we cyclically refresh all tokens. In the same way, we regularly update all custom dashboards we provide.

Everyone features editor rights in their organisations. This means that you can customise all dashboards as you want. We will see, later in this post, how to not generates conflicts with our templates.

Activate the Grafana Beta Organisation

To activate our beta, you need to go to your Clever Cloud organisation view in the Clever Cloud console. Then you will have access to a page called Metrics in Grafana.

Available Dashboards

We built three unique dashboards for you! First we will start by a tour of an organisation metrics. Then we will get more into the details of a runtime view. And finally we will see what we bring for the add-ons. Most graph of all Grafana dashboards contains a small information tile. There, you can retrieve a bit of help and description.

Clever Cloud Organisation

The organisation view is the home dashboard in Grafana. It provides some insights about your organisation. Here, we expose some key information about the applications, the flavours and the image types. Besides, as a home, you will access some links here. You can go to all Clever Cloud template dashboards as well as the ones you set as favourite in Grafana. You can also find out direct link to reach an application.

A tile I want to explain a bit more is the deployment state graph. What is done on this tile? Every new deployment generates a new state. We plot only each one occurring during the Grafana time range. And finally this dashboard includes a metrics counting the average requests rate per minutes, computed from the access logs.

Application Runtime Dashboard

I will now introduce the runtime dashboard. In this dashboard we retrieve several parts. First we start with a quick overview of your application state. Then we will have access to more details sections. Those sections will be about classical system metrics as the CPU, the load, the memory, the disks and the networks. Moreover in each segment, you will find many graphs as the usage or when the resource is saturated. To compute this last metrics, we apply a linear regression.

Add-on Dashboard

The last dashboard contains the add-on metrics view. A first panel contains system-related metrics as the CPU, the memory, the disks or the network. A second one show the database metrics. This is where you will retrieve data as connections count, transactions, errors, deadlocks or tuple operations. However we collect custom data on a limited pool of add-on. This means there are only four of them: MySQL, PostgreSQL, Redis and MongoDB.

Customize Your Grafana Organisation

You have full right to edit all dashboard saved in your Grafana organisation. This means you can edit our templates. However we will update them every once in a while. As a result all your dashboards will always be up to date.

We store all our templates on the Clever Cloud folder. However, you can create custom dashboards anywhere else, we will never update them!

The Grafana manage view lists all folders and files. Besides, this view is helpful to organise and to create new folders!

At any time you can reset our templates. For example in case you made unwanted changes. In that case, you can reach the Grafana page of your Clever Cloud organisation. Then click on reset all dashboards. Finally, all Clever Cloud ones will be back to their initial states.

Share Your Experience

Thank you for reading this post, we will be more than happy to get some feedback! How do you use our available data? Is there something missing for your use case on what we have provided? I hope you will enjoy this new data ride!

Glorious day, we are bringing real-time application metrics to all our users.

Some of you were already Beta testers of our new Metrics solution so a big thank you is in order. Thanks to you and our amazing team, we are officialy rolling out Metrics for all our users today.

Current State

Right now if you want to get an idea of the metrics of your application, you have limited choices. You can use a third party tool and manage it yourself. We have several users happily doing that with New Relic. Or you can SSH to your application instance and use local tools like HTOP. It's not a satisfying situation and we knew we needed to fix it.

Our internal monitoring system (you know that thing that allows your application to restart automatically among other things) was also slowly coming to its limit. So we had to find a new solution.

Enter Clever Cloud Metrics

An original name for an original use case.

Clever Cloud Metrics is the name of our new monitoring solution. It is based on the GeoSpatial Timeseries Database Warp10, Warp10 itself being based on Hadoop and Kafka. This is now what we use on a global scale to monitor all our infrastructure and your applications; to store all the hardware metrics we could get from your VMs. If you are wondering why we made this choice, it's based on mostly two things.

We had scale issues with most known solutions. Without calling them out, all the new ones try to reimplement their own clustering logic. Warp10 clustering is managed by ZooKeeper (remember it's Hadoop and Kafka), a proven solution that has been in production for years. Granted, it's not easy to manage. Good news is it's our job and that's why you chose us.

The other appealing reason to us was the Timeseries specialization. It's perfect for monitoring. When you use a regular datastore, with its regular queryability, it can be hard to do proper time series analysis. To put it simply, SQL does not cut it. Warpscript does:

WarpScript is an extensible stack oriented programming language which offers more than 600 functions and several high level frameworks to ease and speed your data analysis. Simply create scripts containing your data analysis code and submit them to the platform, they will execute close to where the data resides and you will get the result of that analysis as a JSON object that you can integrate into your application.

This happens to be exactly what we need. And they made it super easy to use thanks to Quantum, their query interface based on Polymer (Which means it's also dead easy to integrate in existing webapps). You will be able to use something similar from the Clever Cloud console to query the metrics of your applications. If you are interested, know that the console's dashboard is made with MetricsGraphics.js.

How it Works

From now on you have a new tab in the console called Metrics. It will show you real time graphs of your scalers' hardware resources:

If you don't see anything it's probably because you have not enabled Metrics for your application. To do so you need to add the environment variable ENABLE_METRICS=true.

So what exactly do you have access to?

Hardware Metrics

Our default configuration gives you access to a variety of hardware metrics like CPU, RAM, disk IO, network IO. This is of course available for all the scalers of your application.

Additional Metrics

Depending on the language and runtime you are using, you also get access to specific metrics. For instance if you are running on top of the JVM and have activated JMX (the classic protocol for everything metrics in the JVM) you will see all of the metrics exposed by JMX. This also works for Haskell's EKG. If your favorite is missing, please let us know so we can add its support.

Here is an example of a PHP application that gives you by default the number of idle and active workers as well as the number of requests per second.

All these metrics are collected with Statsd. Which means you can also send your own metrics. It's particularly useful to measure business events. A simple example can be a counter incremented each time a specific API is called.

$statsd = new League\StatsD\Client();
$statsd->configure(array(
    'host' => '127.0.0.1',
    'port' => 8125,
));
$statsd->increment('myApi.pageview');

To send StatsD data, you need a StatsD client configured for '127.0.0.1' as host and '8125' as port. Here I am using one that is nicely integrated to Lumen from the PHP League. Every metric you record yourself are accessible with the statsd prefix. To see it, simply click on Advanced on the top-right corner of the screen, you will be presented with a dropdown:

Advanced Usage

Sometimes you want to go the extra step. If you select Custom View in the drop down you will be redirected to the following screen:

Click on the Quantum perma link and you will be redirected to our Quantum instance, ready to execute custom queries. From there you can do pretty much whatever you want. Before trying anything if you are not familiar with Stack-based languages and Warpscript please visit their documentation.

What's to come

We plan to give you more metrics of course. Wether it comes from our reverse proxy to give you insights on all the requests that go through it or from other runtime frameworks of the like of JMX and EKG. For this we welcome suggestions. Anything you think should be a part of a default monitoring dashboard for this very language or framework, please let us know.

We plan to support the '/_metrics' used by Prometheus. The idea is for our monitoring to regularly poll this endpoint that should send back a JSON containing metrics.

While you can already access the metrics API, in the future you will be able to directly write your own metrics as well.

Please keep in mind that nothing is written in stone for roadmap and things might change :)

We really hope you like this new feature and would love to get your feedback. Please tell us what you think :)

Some resources to get you started

We plan to add more of these but here are sample projects that write to a local StatsD intance. There are many others available on Github:

• NodeJS with Hapi and statsd-client
• PHP with Lumen
• Kotlin, Spring Boot 2 with micrometrics
• Go with Goji
• Ruby with Rack
• Python with Falcon