This is precisely the role of observability. It is also why Elasticsearch has gradually established itself as an analytical foundation for logs, metrics, and traces.

In this article, we will look at how Elasticsearch fits into an observability approach beyond simple logging, and how it enables technical signals to be correlated in order to better understand application behaviour.

What is observability, and why Elasticsearch is involved

Observability refers to the ability to understand the internal state of a system based on its external signals. Unlike traditional monitoring, it is not limited to predefined metrics or fixed thresholds.

Observability relies on collecting rich, contextual data, analysing it across multiple dimensions, and exploring situations that were not anticipated in advance. In this context, Elasticsearch plays a key role. Its indexing and search engine can analyse large volumes of heterogeneous data, structured or unstructured, in near real time, which aligns precisely with the needs of a modern observability approach.

The three pillars of observability: logs, metrics, and traces

An observability strategy is built on three complementary types of signals. Each addresses a different question and provides a specific perspective on system behaviour.

Logs: understanding what happened

Logs are events produced by applications and infrastructure components. In Elasticsearch, they are associated with a timestamp, either derived from the log event itself or from the ingestion time. They provide a high level of detail and make it possible to understand the precise context of an error, unexpected behaviour, or incident.

Elasticsearch has historically been well suited to this use case:

• ingesting large volumes of data,
• fast full-text search,
• fine-grained event exploration.

Logs provide valuable context, but they become difficult to exploit on their own as architectures become more distributed and data volumes grow significantly.

Metrics: measuring system state

Metrics are numerical data aggregated over time. They describe the overall state of a system and make it possible to track its evolution. Latency, error rates, and resource consumption provide a high-level view of application or infrastructure health.

In Elasticsearch, these data are stored as time-series. This enables aggregations, long-term trend analysis, and anomaly detection, while still allowing metrics to be linked to other technical signals.

Traces: following a request end to end

Traces describe the full journey of a request through a distributed system. They are essential for understanding dependencies between services and for pinpointing the exact source of latency or errors.

Each trace is composed of multiple segments representing different execution steps. Once indexed in Elasticsearch, these traces can be correlated with associated logs and metrics, making it easier to analyse complex behaviours in microservices environments.

How Elasticsearch correlates logs, metrics, and traces

The value of observability does not lie in individual signals taken in isolation, but in their correlation. Elasticsearch facilitates this correlation through several structural mechanisms:

• a shared indexing engine,
• common schemas such as ECS (Elastic Common Schema), which provides a shared structure for logs, metrics, and traces,
• cross-signal search capabilities.

In practice, this approach makes it possible to navigate naturally between signals. An alert triggered by a metric can lead to the analysis of related traces, followed by the exploration of logs associated with a specific request. Kibana plays a central role by making these correlations visible and actionable, through visualisations, dashboards, and exploration tools designed for cross-signal analysis.

Historically, Elasticsearch is best known for powering application search engines, particularly for indexing and querying website content. The same principles of fast, contextual search apply to observability data: logs, metrics, and traces are also indexed and queried as datasets, which makes large-scale exploration and correlation possible.

OpenTelemetry: a key standard for observability with Elasticsearch

In modern architectures, data collection is just as important as data analysis. OpenTelemetry has emerged as an open standard for application instrumentation, covering traces, metrics, and logs.

Elasticsearch natively supports OpenTelemetry, enabling signal collection to be standardised without relying on proprietary formats. This compatibility improves interoperability, reduces technological lock-in, and allows observability tooling to evolve without requiring changes to existing application instrumentation.

Observing your applications with Elastic on Clever Cloud

In a PaaS hosting context, observability must remain easy to enable and simple to operate. On Clever Cloud, Elasticsearch is available as a managed add-on. Applications can send their logs using Elasticsearch drains, enabling automatic centralisation of application logs. Several components can then be enabled depending on requirements:

• a managed Elasticsearch cluster,
• Kibana for exploration and visualisation,
• Elastic APM for application performance analysis.

This approach makes it possible to centralise application logs, collect relevant metrics, and trace requests without having to manage the underlying infrastructure. The goal is not to multiply tools, but to provide a coherent observability foundation integrated into the application lifecycle.

Conclusion

Observability is not about stacking monitoring tools. It is about correlating logs, metrics, and traces in order to understand increasingly complex systems.

Thanks to its indexing, search, and analysis capabilities, Elasticsearch provides a solid technical foundation for this approach. Combined with open standards and interfaces such as Kibana, it enables teams to move from fragmented visibility to a comprehensive understanding of application behaviour.

In modern cloud environments, this correlation is no longer a luxury. It is a necessary condition for operating production systems reliably.

It is in this context that the ELK stack has established itself as a technical foundation for analysing, searching, and visualising technical data, particularly logs.
In this article, we answer three key questions:

• What exactly is the ELK Stack?
• What is it used for today, especially in observability?
• How can it be used effectively without managing the underlying infrastructure?

ELK Stack: a clear definition

The ELK Stack is a historical acronym that refers to three components:

• Elasticsearch: a distributed search and analytics engine;
• Kibana: a data exploration and visualisation interface;
• Logstash: a data collection and transformation tool (depending on the context).

At present, Elasticsearch and Kibana form the functional core of the ELK stack, particularly for data analysis and visualisation use cases, once the data has been ingested into Elasticsearch.

The term Elastic Stack is also used, referring more broadly to the entire Elastic ecosystem. In common usage—especially in cloud environments—the ELK Stack generally refers to the combination of a data collection mechanism, often agent-based, with Elasticsearch for storage and analysis, and Kibana for visualisation.

What is the ELK Stack used for?

The ELK Stack is used to centralise, analyse, and exploit technical data coming from systems and applications. It enables large volumes of data to be indexed and analysed across wide time ranges, while correlating information from multiple sources, services, or environments.

This analytical capability makes it a widely adopted tool for understanding application behaviour, diagnosing incidents, investigating anomalies, or exploring operational data. Its main strength lies in the ability to move quickly from raw data to actionable insights, without relying on specialised tools for each individual use case.

ELK Stack and observability: what is the connection?

Observability aims to understand the internal state of a system through its observable signals. Among these signals, logs play a central role, as they describe precisely what an application is doing at a given point in time.

In this context, the ELK Stack provides a particularly well-suited foundation for log-centric observability. Elasticsearch enables large-scale search and correlation of events, while Kibana provides a visual layer that makes analysis and interpretation easier. Together, they make it possible to detect abnormal behaviour, reconstruct the timeline of an incident, and analyse trends over time.

In an observability approach, the ELK Stack is therefore mainly used as a log analysis foundation, complemented by other signals depending on the needs.

How to use the ELK Stack without managing infrastructure

One of the main barriers to adopting the ELK Stack has long been its operational complexity. Deploying, maintaining, and scaling such a stack requires handling capacity planning, upgrades, security, and backups.

In cloud environments, this operational burden can quickly distract teams from their primary goal: analysing data rather than managing infrastructure. This is why many teams now turn to managed approaches.

Managed approach

In a managed approach, Elasticsearch and Kibana are provided as ready-to-use services. The underlying infrastructure and part of the day-to-day operations—such as service provisioning, maintenance, backups, and access control according to the platform’s model—are handled by the platform. This allows teams to focus on usage rather than operations.

In this model, log collection is handled by the platform’s mechanisms. On Clever Cloud, applications and add-ons can expose their logs through drains, which redirect them to a target Elasticsearch instance without deploying any collection tooling inside the PaaS.

On Clever Cloud, it is for example possible to create an Elastic Stack add-on that provides:

• a managed Elasticsearch service;
• an associated Kibana instance;
• built-in security and backup mechanisms;
• a connection using the access credentials provided by the add-on.

This approach makes it possible to leverage the ELK Stack without managing low-level infrastructure concerns, while retaining the analytical power of Elasticsearch.

Concrete observability use cases

Application log analysis

Centralising application logs in Elasticsearch makes it possible to quickly search for errors, explore specific events, or filter data using multiple criteria. This capability is essential for understanding the real behaviour of an application in production.

Incident diagnosis

When an incident occurs, event correlation becomes critical. The ELK Stack allows teams to analyse event timelines, identify the components involved, and better understand root causes, without being limited to a fragmented view of logs.

Application behaviour monitoring

Over time, analysing indexed data in Elasticsearch helps detect trends, abnormal spikes, or behavioural changes. Kibana dashboards facilitate this analysis and provide a synthetic view tailored to technical teams.

Conclusion

The ELK Stack remains a solid foundation for analysing and exploiting technical data, particularly logs. Its role in observability practices has grown alongside the evolution of cloud-native and distributed architectures.

By relying on the functional core of the ELK Stack—namely Elasticsearch and Kibana—it is possible to build an analysis environment suited to modern needs without necessarily managing the underlying infrastructure. Managed approaches help reduce operational complexity and allow teams to focus on data value.

ELK Stack use cases continue to evolve. Recent work by Elastic on new log management models, such as streams, opens the door to more flexible approaches better suited to current data volumes. These evolutions build on existing foundations without calling into question Elasticsearch’s central role in observability data analysis.

For those looking to explore these use cases in a controlled environment, creating an Elastic Stack add-on on Clever Cloud offers a pragmatic way to approach Elasticsearch-based observability without turning operations into a constraint.

In my last article I created a simple Canvas workpad in my Kibana. To do so I already had meetup data indexed in my Elastic instance. Today I am going to tell you how I did gather data and added it to Elasticsearch.

Selecting meetups I want to track

The first thing to do is to select a list of meetup groups you want to keep track of. There are many ways to do that using the Meetup API. For instance, you can search to similar groups using the similar_groups endpoint. I let you read the API documentation to find a way to select the events you will track. You just need to format the response to extract the cities and the names of the Meetups groups in a JSON file formatted as follows:

{
  "City name 1": [
    "meetup group name 1",
    "meetup group name 2",
    ...
  ],
  "City name 2": [
    "meetup group name 1",
    ...
  ]
}

Getting data about meetup events

Once you have this, you can use the node.js application we created. Of course it has the Elasticsearch dependency in package.json. You are strongly invited to check out the source code at this point. You can see in the index.js many parts of interest:

• The creation of an express server listening on port 8080. We need it, so Clever Cloud will know our app is up running.

app.get('/', (req, res) => {
    res.send('Hello !');
});
app.listen(8080, () => console.log('Listening on port 8080!')); 

• The meetup API call:

axios.get(`https://api.meetup.com/${meetupName}/events/?status=past,upcoming\&fields=comment_count`)

• the creation of Elasticsearch indexes:

await client.indices.create({
  index: "meetup",
  body : {
    "mappings": {
      "properties": {
        "time":  {"type": "date", "format": "epoch_millis"},
        "group.name": {"type": "keyword"},
        "yes_rsvp_count" : {"type": "integer"},
        "grouploc": {"type": "geo_point"},
        "venueloc":{"type": "geo_point"}
      }
    }
  }
});

• The creation of another server where our meetup API calls happen, listening on port 8081. We can also notice that we've restricted our server to allow only localhost connections.

localapp.listen(8081, 'localhost', function() {
  console.log("... port %d in %s mode", 8081, localapp.settings.env);

Now in ./clevercloud/cron.json you can notice a cron task, wich will trigger a curl on http://localhost:8081/ every night at 1 AM:

"0 1 * * * /usr/host/bin/curl http://localhost:8081/"

It is this cron that will call our second server to trigger the meetup API calls.

True fact: to use it in its current version, you must keep your application running all the time for one hour of usage maximum. A way to improve the application regarding this issue would be to implement authentication to our application, so we still are the only one having access.

Then remove the cron from this project to have it running in your main application instead. Your main application will be the one consuming this indexed data. Taking advantage on the fact that every virtual machine running on Clever Cloud already has the Clever Tools CLI installed, we could improve our cron to start the application for an hour then stop it when it has finished its indexation job.

So you will end up with two machines, one with your main application, and the second one running for one hour each night.

We must also know that Clever Cloud does not monitor what's going on on port 8081. You could add a logging system or use Elastic APM to monitor your application during its execution time.

This is an approach among many others, do not hesitate to talk with us about your own implementation.

Okay, let's go back to our main goal, and to do so, you can use our sample data meetup list or use your own by replacing the json in the meetups.jsonfile.

Try it out

You can $ git clone the repo in your console, and go into your Clever Cloud console.

Under the organization of your choice, select New, Application, Node. When prompted if you need add-ons, select Elastic Stack, select the plan you need and enable Kibana as an option.

In the environment variables menu of your application, add NODE_ENV=production and add the provided clever remote to your local git folder. Then push using git push -u clever master.

Your deployment will start and thanks to the ES_ADDON_URI we provided in our index.js file, we have nothing else to configure, our application will start sending data to elastic.

Visualize your data and go further

Either in your Kibana or Elastic instance menu in the Clever Cloud console, in the information page you will find a Open Kibana button. Click it and login using your Clever Cloud credentials.

Into Kibana click on the Management (gear) icon in the left side menu. Under the Kibana title, select Index Patterns, then meetup* to see how the data is indexed.

Of course at this point, you are able to do the exact same as I did in the previous article video.

Here is the ElasticSQL query I used in the Canvas demonstration:

SELECT AVG("yes_rsvp_count") AS average, "group.name" FROM "meetup*"
GROUP BY "group.name"
ORDER BY average DESC
LIMIT 5

Happy indexing!

Good news everyone, we are really excited to offer the Elastic Stack — 🔥Platinum Version 🔥 — on Clever Cloud. It's the first as a service offer that is officially supported by Elastic from a French company, with datacenters located in France.

If you are looking for an Elastic Stack provider dealing with the American Cloud Act problematics and the GDPR regulations, look no further. :)

We are proud to partner with Elastic and offer you the full Elastic Stack on Clever Cloud.

Am I talking about X-Pack? Yes and no, I am talking about so much more. You will have access to Kibana Canvas & Lens, Kibana Spaces with full security (encryption, RBAC, field and doc-level security), Alerting, Elasticsearch SQL, Machine Learning, Metrics, logs, ... If you want the full details about what is available on Clever Cloud, take a look at the Platinum column on this page.

Our starting price is 17 euros per month. With this you can get the full extent of the Elastic Stack, at a very small scale. On the other hand, you can go all the way up to 64 CPUs and 256Go of RAM per node. Our support team will manage the first levels of support while being able to escalate to Elastic's team when needed. You are in good hands!

And of course we worked on an integration with the rest of Clever Cloud. Please have a look at our documentation for the details. Here's a quick glance at what we did.

Specific Clever Cloud Integrations

We have worked on our Elastic Stack integration on several fronts. When you provision the Elastic add-on, we allow you to provision Kibana and an APM server as traditional Clever Cloud applications. It means that they benefit from all the goodness that we bring to applications. You can turn them off if you want to, you can enable auto scalability, you can link them to other applications, really anything you would do with traditional applications. Let's see each integration a bit more in details.

Authentication

Authenticating to Kibana is available through an automatically configured SSO. Every member of the organisation the addon has been deployed to can use their Clever Cloud account to authenticate. No configuration is required on your part.

Elastic APM

If you link the APM server application to any of your application, the right environment variables will be injected and automatically picked by the APM agent in your dependencies. Then simply authenticate to Kibana and start setting up your APM! This feature will be showcased in a dedicated blog post in the coming days.

Backups

We are introducing a new way to manage your add-on backups. When we create your new add-on, we also create a Cellar add-on (our S3-compatible object storage solution) named Backups. All your backups will be stored there. We are starting with Elasticsearch but other databases will soon follow.

What's next?

We plan to provide an even better integration with the Elastic Stack. We are currently thinking about the best way to integrate Beats to our applications. You can currently do it manually but wouldn't it be nice if this was automated? This should really ease the usage of Elastic SIEM for instance.

And of course we are working on automatic cluster provisioning, not just for Elastic. In the meantime you can provision as many nodes as you need and contact our support team to put them in the same cluster.

We are supper excited about working with Elastic and hope you will be as excited to try it on Clever Cloud! We will publish more blog posts in the coming days, highlighting some of the awesome capabilities of our Elastic Stack integration.

➡️ https://console.clever-cloud.com