Dans cet épisode nous parlons de la nouvelle release d'Elastic, de base de donnée serverless, de la plainte de la CNIL contre de gros sites qui utilisent Google Analytics, de Cloudflare côté panne et nombres aléatoires, de “destructuring assignment” dans Java, de Rust 1.63, l'outil de l'époside est Picosnitch et enfin, un peu de geek porn avec l'installation de baies Clever dans notre nouveau Data Center, avant de finir en musique avec de l'electro vintage.

👋 Venez discuter avec nous sur @clever_cloudFR pour nous dire ce que vous avez pensé de ce nouvel épisode.

➡️ Pour découvrir ou réécouter d’anciens épisodes c’est par ici !

Timecode & Liens

00:00:00 Introduction

00:06:29 Nouvelle release Elastic (Sylvain)
https://www.elastic.co/blog/whats-new-elastic-8-3-0

00:10:42 Bases de données serverless avec SQLite et Postgres (Sylvain)
https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/
https://twitter.com/simonw/status/1529134311806410752

00:16:42 Plainte à la CNIL contre 42 sites utilisant Google Analytics (Sylvain)
https://www.latribune.fr/technos-medias/internet/seisme-dans-la-presse-42-medias-attaques-aupres-de-la-cnil-pour-utilisation-illegale-de-google-analytics-923423.html

00:20:30 Cloudflare down: internet en panique (Arnaud)
https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/
C’est la faute de BGP
Ils ont pas testé sur les DC qui implémentent leur nouvelle stack réseau

00:25:00 Cloudflare génère des nombres aléatoires avec des ‘lava lamps’ (Manu)
https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/
https://en.wikipedia.org/wiki/Lavarand
Bonus: banana powered random number generation
https://www.valerionappi.it/chi-squared/
https://www.valerionappi.it/brng-en/

00:32:08 Record patterns en Java : apparition du “destructuring assignment” dans Java (Sylvain)
https://openjdk.org/jeps/405

00:40:10 Scoped threads dans rust 163 (Julien)
https://www.reddit.com/r/rust/comments/vftc28/rust_163_scoped_threads/

00:49:58 Picosnitch - Visualiser le trafic réseau par exécutable (Manu)
https://github.com/elesiuta/picosnitch

00:52:06 On installe nos baies dans un DC
https://twitter.com/GwinizDu/status/1541467816376897543

Musique de fin :
Bluxte (1-20) : Space - Magic Fly (1977)
https://www.youtube.com/watch?v=eViso883FR4

In my last article I created a simple Canvas workpad in my Kibana. To do so I already had meetup data indexed in my Elastic instance. Today I am going to tell you how I did gather data and added it to Elasticsearch.

Selecting meetups I want to track

The first thing to do is to select a list of meetup groups you want to keep track of. There are many ways to do that using the Meetup API. For instance, you can search to similar groups using the similar_groups endpoint. I let you read the API documentation to find a way to select the events you will track. You just need to format the response to extract the cities and the names of the Meetups groups in a JSON file formatted as follows:

{
  "City name 1": [
    "meetup group name 1",
    "meetup group name 2",
    ...
  ],
  "City name 2": [
    "meetup group name 1",
    ...
  ]
}

Getting data about meetup events

Once you have this, you can use the node.js application we created. Of course it has the Elasticsearch dependency in package.json. You are strongly invited to check out the source code at this point. You can see in the index.js many parts of interest:

• The creation of an express server listening on port 8080. We need it, so Clever Cloud will know our app is up running.

app.get('/', (req, res) => {
    res.send('Hello !');
});
app.listen(8080, () => console.log('Listening on port 8080!')); 

• The meetup API call:

axios.get(`https://api.meetup.com/${meetupName}/events/?status=past,upcoming\&fields=comment_count`)

• the creation of Elasticsearch indexes:

await client.indices.create({
  index: "meetup",
  body : {
    "mappings": {
      "properties": {
        "time":  {"type": "date", "format": "epoch_millis"},
        "group.name": {"type": "keyword"},
        "yes_rsvp_count" : {"type": "integer"},
        "grouploc": {"type": "geo_point"},
        "venueloc":{"type": "geo_point"}
      }
    }
  }
});

• The creation of another server where our meetup API calls happen, listening on port 8081. We can also notice that we've restricted our server to allow only localhost connections.

localapp.listen(8081, 'localhost', function() {
  console.log("... port %d in %s mode", 8081, localapp.settings.env);

Now in ./clevercloud/cron.json you can notice a cron task, wich will trigger a curl on http://localhost:8081/ every night at 1 AM:

"0 1 * * * /usr/host/bin/curl http://localhost:8081/"

It is this cron that will call our second server to trigger the meetup API calls.

True fact: to use it in its current version, you must keep your application running all the time for one hour of usage maximum. A way to improve the application regarding this issue would be to implement authentication to our application, so we still are the only one having access.

Then remove the cron from this project to have it running in your main application instead. Your main application will be the one consuming this indexed data. Taking advantage on the fact that every virtual machine running on Clever Cloud already has the Clever Tools CLI installed, we could improve our cron to start the application for an hour then stop it when it has finished its indexation job.

So you will end up with two machines, one with your main application, and the second one running for one hour each night.

We must also know that Clever Cloud does not monitor what's going on on port 8081. You could add a logging system or use Elastic APM to monitor your application during its execution time.

This is an approach among many others, do not hesitate to talk with us about your own implementation.

Okay, let's go back to our main goal, and to do so, you can use our sample data meetup list or use your own by replacing the json in the meetups.jsonfile.

Try it out

You can $ git clone the repo in your console, and go into your Clever Cloud console.

Under the organization of your choice, select New, Application, Node. When prompted if you need add-ons, select Elastic Stack, select the plan you need and enable Kibana as an option.

In the environment variables menu of your application, add NODE_ENV=production and add the provided clever remote to your local git folder. Then push using git push -u clever master.

Your deployment will start and thanks to the ES_ADDON_URI we provided in our index.js file, we have nothing else to configure, our application will start sending data to elastic.

Visualize your data and go further

Either in your Kibana or Elastic instance menu in the Clever Cloud console, in the information page you will find a Open Kibana button. Click it and login using your Clever Cloud credentials.

Into Kibana click on the Management (gear) icon in the left side menu. Under the Kibana title, select Index Patterns, then meetup* to see how the data is indexed.

Of course at this point, you are able to do the exact same as I did in the previous article video.

Here is the ElasticSQL query I used in the Canvas demonstration:

SELECT AVG("yes_rsvp_count") AS average, "group.name" FROM "meetup*"
GROUP BY "group.name"
ORDER BY average DESC
LIMIT 5

Happy indexing!

You like Kibana? You will love Canvas

Canvas is a feature that comes inside our beloved Kibana dashboard and allows us to visualize and present our data from our Elasticsearch our way.

Whether you are working with application metrics, security events, infrastructure logs, business-related data or even your side projects, as long as you put the generated info in an Elasticsearch instance, you will be able to display them in an elegant way.

Create workpads with combined data from your Elasticsearch and convert them into beautiful charts, graphs, progress monitors, infographics, reports… The only limit is your imagination (yes, for real)!

You already have a visual identity ? Import your logo, your color schemes, your css to get a consistent interface all over and reuse them everywhere! Share them with your users and customers with ease so your branding is really homogeneous.

Not having a visual identity? There are tons of color palettes and templates you can play around with.

Make your workpads dynamic using time-series data, add your Elastic Maps into your workpads, choose how often it should be updated.

Canvas also lets you customize your workspace with backgrounds, colors, fonts, images and more.

Is it easy to use

What a question! You should know how simple is the Elastic Stack experience!

Canvas arrives out of the box inside Kibana on Clever Cloud, with features like syntax highlighting, mouse-over context-sensitive help, drag and drop publishing, dark mode and more to help you create pixel-perfect visuals with your data.

Use Elasticsearch SQL to design aggregations like you already do in your applications and display them in your workpads.

Canvas comes with its own library of functions to empower your imagination but if it’s not enough, with the Elastic plugin framework, you can also add your own plugins or community ones no matter how experienced in development you are.

And thanks to Clever Cloud, keeping everything in operating condition is pretty simple, as we manage this for you. While as usual providing you a Europe-based infrastructure, GDPR compliant and protected from Cloud Act.

Reuse your workpads everywhere

You might wonder: what can I do with my awesome creations? Well, actually almost everything!

You can, of course, use them in your Kibana, but here are some examples of usage you can have outside of your dashboards:

• export your workpads as JSON
• generate one shot PDF
• automate PDF reporting with a POST endpoint
• display your workpads on websites (and customize them there to match the design of the page)
• get live reporting on screens in your office, retail points or any place
• add real data to your weekly, monthly … emails

Once again, the only limit is your imagination!

Do not hesitate to try out Canvas, it is really easy to use and available in one click and stay tuned, next time I’ll show you how I indexed meetups data into Elastic.

Good news everyone, we are really excited to offer the Elastic Stack — 🔥Platinum Version 🔥 — on Clever Cloud. It's the first as a service offer that is officially supported by Elastic from a French company, with datacenters located in France.

If you are looking for an Elastic Stack provider dealing with the American Cloud Act problematics and the GDPR regulations, look no further. :)

We are proud to partner with Elastic and offer you the full Elastic Stack on Clever Cloud.

Am I talking about X-Pack? Yes and no, I am talking about so much more. You will have access to Kibana Canvas & Lens, Kibana Spaces with full security (encryption, RBAC, field and doc-level security), Alerting, Elasticsearch SQL, Machine Learning, Metrics, logs, ... If you want the full details about what is available on Clever Cloud, take a look at the Platinum column on this page.

Our starting price is 17 euros per month. With this you can get the full extent of the Elastic Stack, at a very small scale. On the other hand, you can go all the way up to 64 CPUs and 256Go of RAM per node. Our support team will manage the first levels of support while being able to escalate to Elastic's team when needed. You are in good hands!

And of course we worked on an integration with the rest of Clever Cloud. Please have a look at our documentation for the details. Here's a quick glance at what we did.

Specific Clever Cloud Integrations

We have worked on our Elastic Stack integration on several fronts. When you provision the Elastic add-on, we allow you to provision Kibana and an APM server as traditional Clever Cloud applications. It means that they benefit from all the goodness that we bring to applications. You can turn them off if you want to, you can enable auto scalability, you can link them to other applications, really anything you would do with traditional applications. Let's see each integration a bit more in details.

Authentication

Authenticating to Kibana is available through an automatically configured SSO. Every member of the organisation the addon has been deployed to can use their Clever Cloud account to authenticate. No configuration is required on your part.

Elastic APM

If you link the APM server application to any of your application, the right environment variables will be injected and automatically picked by the APM agent in your dependencies. Then simply authenticate to Kibana and start setting up your APM! This feature will be showcased in a dedicated blog post in the coming days.

Backups

We are introducing a new way to manage your add-on backups. When we create your new add-on, we also create a Cellar add-on (our S3-compatible object storage solution) named Backups. All your backups will be stored there. We are starting with Elasticsearch but other databases will soon follow.

What's next?

We plan to provide an even better integration with the Elastic Stack. We are currently thinking about the best way to integrate Beats to our applications. You can currently do it manually but wouldn't it be nice if this was automated? This should really ease the usage of Elastic SIEM for instance.

And of course we are working on automatic cluster provisioning, not just for Elastic. In the meantime you can provision as many nodes as you need and contact our support team to put them in the same cluster.

We are supper excited about working with Elastic and hope you will be as excited to try it on Clever Cloud! We will publish more blog posts in the coming days, highlighting some of the awesome capabilities of our Elastic Stack integration.

➡️ https://console.clever-cloud.com