Step 1: Creating a new Phoenix Application

$ mix phx.new <your application's name> # creating the new phoenix project
$ cd <your application's name>
$ mix ecto.create # creating database locally

Note to OSX users

Step 2: Getting your Application Ready for Production

Step 3: Creating the Prod Application and Database

• Under the organization of your choice, click new, then application. Select Elixir, then name your application and choose its deployment zone.
• When prompted if you need an add-on, select PostgreSQL. Select your database size and name it.
• On the environment variable screen create two new ones:
• SECRET_KEY_BASE with the value obtained from $ mix phx.gen.secret previously
• CC_ELIXIR_VERSION with the value of $ elixir -v result. (Available values as of today are 1.8, 1.9 or 1.10)
• Click on update changes
• Bonus: if you have migrations to run you could also set the environment variable CC_PHOENIX_RUN_ECTO_MIGRATE to true to trigger the command $ mix ecto.migrate

Step 4: deploy!

The magic explained

$ mix deps.get
$ mix deps.compile
$ npm install

It is now possible to force HTTPS redirection! No more .htaccess black magic, no redirection handling code… Let's have a tour of this new feature.

Forcing HTTPS on Clever Cloud

Let's start practical. There are two ways to achieve our goal.

Using the CLI

In your linked application, in your own terminal just type in clever config set force-https enabled.

clever config is a new CLI command that allows you to display in your console the current configuration of your application. You can edit any setting with clever config set {name} {value} and bulk edit with clever config update [options].

Here we just set the force https option to enabled and if we want to set if off we can just clever config set force-https disabled.

If this does not work for you, make sure you are running version 2.5.0 or newer (with clever version).

Using the Clever Cloud console

In the information page of each application, you will find a checkbox form with options you can tune in and out. One of them is Force HTTPS. Enable its checkbox and save.

From now on and as long as you do not disable it, every non-secured HTTP request to this application will be redirected to HTTPS with a 301 Moved Permanently status code.

How the magic happens

The Clever Cloud way

The redirection is handled at the reverse proxy level so you don't need to update your application to use it. Each time the browser will request a resource using HTTP, it will get a 301 response back with the same resource prefixed by https in the Location header field. The 301 redirection is recognized as the best practice for HTTPS upgrade.

Why enforce HTTPS

Before explaining why it is important, I shall provide a definition of HTTPS.

Simple definition

First of all, HTTPS is an extension of the HTTP protocol. Which itself is the application protocol the world wide web is relying on for communications. Full definition to be found on wikipedia.

The main difference between HTTP and HTTPS communication is encryption.

Using HTTPS, the communication is bidirectionaly encrypted between the client and the server. Both the headers and the response/request data are ciphered. This is achieved using TLS cryptographic protocol in addition to operations involved in a simple HTTP transmission.

The identification process requires the server administrator to create a public key certificate signed by a trusted certificate authority.

We have automated this part. So every time you create a new application on Clever Cloud, a cleverapps.io subdomain name is given to your application and comes along with encryption because a certificate covers cleverapps.io and all its subdomains. And each time you add a custom domain to your application like example.com, Clever Cloud asks Let's Encrypt for a TLS public key certificate that we automatically add to your application's configuration on our reverse proxies. Thanks to that, you can accept HTTPS requests on a brand new application without further configuration.

Enforcing HTTPS is just the beginning

I explained to you how HTTPS enforcing was working on Clever Cloud and that felt very simple, right?

To be honest this is because the tech team has decided only to handle the first step of the process.

Strong security on an application requires knowledge on many topics.

Sometimes, your application itself will link to HTTP content, which will result in mixed-content. When HTTP resources are served by the application on a page requested using HTTPS the browser will usually not load those resources.

You will also probably want to know more about HTTP Strict Transport Security (HSTS), and the Content-Security-Policy header.

You can start by reading this great article from Scott Helme to start your journey. If you cannot handle advanced security yourself, think about asking for the help of a security expert.

HTTPS is good for everyone, except hackers

In 2020 the HTTPS protocol is ubiquitous. It's more and more widely used thanks to the Let's Encrypt initiative which made it free for everyone; which makes the web more secure for both applications creators and clients.

The clients know they can trust the integrity of the content that's being displayed to them and feel safer about using the Internet in general. This safety feeling extends to their privacy. Today the regular user's main concern is about what the owner of the website do with their data. Not about that black hooded bad guy they were freaking out about a few years ago. The most used websites have raised their confidence about communications ciphering to such a level it's now the norm. Everyone puts credit-card numbers in forms eyes wide shut.

Yeah, but me ? Having a website to offer to the world, I want to meet my client’s expectations, of course. But HTTPS has more to offer me. From the SEO perspective, having it will naturally boost ranking. I also ensure my user's privacy and confidentiality. Most importantly, I can trust that the data I receive from my client has not been altered. That doesn’t mean that I shall trust my user intentions obviously. But now if I need to watch out for a hacker, it will not come from a man in the middle or this kind of attack.

Anyway, this is everywhere now and already almost mandatory. Take this train now! This feature is free, does not require extra code and works out of the box with the automatically generated Let's Encrypt certificates.

Introduced with the release of Rails 5.2 , Active Storage replaces previous solutions for storing user videos, images, PDF, etc..

In the documentation only three implementations are demonstrated and they all rely on cloud services. Today I am going to show you how to use it to be Clever Cloud compatible.

To follow this tutorial you must already have a Rails >= 5.2 application running on Clever Cloud with a PostgreSQL database. You can read this if you need a basic setup.

Tell Rails you want to use Active Storage

In your console, at the root of your rails project repository, type in $ rails active_storage:install. This will create two migration files under ./db/migrate. These migrations will create two new tables: active_storage_blobs and active_storage_attachments, which is everything we need to start storing user attachments without adding new columns to our already existent models. Let's run them with $ rake db:migrate to have a local database up to date. You can now use has_one_attached :<the name you want to give> on any of your models to add an attachment on it.

Setting up the application for Clever Cloud

As we do provide AWS S3 like storage at Clever Cloud, we will use AWS gem and define custom endpoints. Add gem "aws-sdk-s3", require: false to your Gemfile. Run $ bundle install. Then in config/environments/production.yml add config.active_storage.service = :clevercloud. Now in config/storage.yml add the following lines:

clevercloud:
  service: S3
  access_key_id: <%= ENV['CELLAR_ADDON_KEY_ID'] %>
  secret_access_key: <%= ENV['CELLAR_ADDON_KEY_SECRET'] %>
  region: us-east-1
  bucket: <%= ENV['CELLAR_ADDON_BUCKET_NAME'] %>
  endpoint: <%= ENV['CELLAR_ADDON_ENDPOINT'] %>

I must make clear here that we do not use regions at Clever Cloud, but the aws-sdk-s3 gem needs it; otherwise it throws an AWS::Errors::MissingRegionError error. And we do not want a 500 in production. It is the usage of endpoint that will give us the right endpoint generated by the gem.

Creating our storage on Clever Cloud

Under the organization where your Rails application is deployed, select Create, an add-on, Cellar S3 storage. When prompted Select the applications that will use this add-on: select your Rails application. Name it and let it be deployed. Once this is done, in your addon dashboard, create a bucket. Keep its name. Now, under Environment variables in your Rails application, add the following:

CELLAR_ADDON_BUCKET_NAME=<your bucket name>
CELLAR_ADDON_ENDPOINT=https://cellar-c2.services.clever-cloud.com

You can now commit the local files we have changed and push them to your Clever Cloud remote to enjoy Active Storage in production.

Do not forget to edit your safe params in your controllers, forms and permissions.

Happy storing!

Today, I am proposing a very simple tutorial. I needed to integrate a Metabase dashboard in one of my applications and I realized there is no article which explains how to do that. So, ta-da!

Requirements

To follow this tutorial, you will need

• a Metabase dashboard (read this article from my fellow Laurent Dogin to have yours set up in a few minutes)
• a Ruby on Rails application (read this article by me if you need to set up one)

Enabling integration in Metabase

From your Metabase, click on the top right gear icon, then Admin. On the left menu select Embedding in other Applications. Toggle the button to have it saying Enabled. You will now get an Embedding API key. Save it for later.

Leave Metabase admin and select the dashboard you want to integrate. Click the sharing and embedding icon on the top right of your dashboard. Now select Embed this dashboard in an Application. Finally, click on the Publish button.

Integrate Metabase in Rails

In your Gemfile, add gem 'jwt' then run $ bundle install. If your application is deployed on Clever Cloud go to your application's console and under your application's menu go to the environment variables menu. If you deploy locally, use the gem figaro and edit your application.yml. Anyway you must add two environment variables

METABASE_SITE_URL: <your metabase url>
METABASE_SECRET_KEY: <the API key from metabase admin>

In the controller action where you want to integrate your metabase dashboard, add the following lines:

payload = {
          :resource => {:dashboard => <your dashboard number>},
          :params => { },
          :exp => Time.now.to_i + (60 * 10) # 10 minute expiration
    }
token = JWT.encode(payload, ENV["METABASE_SECRET_KEY"])
@iframe_url = ENV["METABASE_SITE_URL"] + "/embed/dashboard/" + token + "#bordered=true&titled=true"

Now in your view, add

<iframe
    src="<%= @iframe_url %>"
    frameborder="0"
    width="800"
    height="600"
    allowtransparency
></iframe>

You are all set. Just restart your application so the new environment variables are taken in consideration. Voilà, your application now has an integrated Metabase dashboard.

Requirements

• you need to have npm and git installed on your machine.
• you need a Clever Cloud account with at least a validated email to benefit the free credits.

Setting up our MDX-Deck

$ mkdir presentation //creating a new folder
$ cd presentation
$ git init // creating fresh git folder
$ npm init //creating new project

"main": "index.js",

"scripts": {
    "start": "mdx-deck deck.mdx",
    "build": "mdx-deck build deck.mdx",
    "test": "echo \"Error: no test specified\" && exit 1"
},

# Congratulations!

---

## You're up and live on Clever Cloud!

---

Deploy to clever cloud as static with hooks

Explanation

In my last article I created a simple Canvas workpad in my Kibana. To do so I already had meetup data indexed in my Elastic instance. Today I am going to tell you how I did gather data and added it to Elasticsearch.

Selecting meetups I want to track

The first thing to do is to select a list of meetup groups you want to keep track of. There are many ways to do that using the Meetup API. For instance, you can search to similar groups using the similar_groups endpoint. I let you read the API documentation to find a way to select the events you will track. You just need to format the response to extract the cities and the names of the Meetups groups in a JSON file formatted as follows:

{
  "City name 1": [
    "meetup group name 1",
    "meetup group name 2",
    ...
  ],
  "City name 2": [
    "meetup group name 1",
    ...
  ]
}

Getting data about meetup events

Once you have this, you can use the node.js application we created. Of course it has the Elasticsearch dependency in package.json. You are strongly invited to check out the source code at this point. You can see in the index.js many parts of interest:

• The creation of an express server listening on port 8080. We need it, so Clever Cloud will know our app is up running.

app.get('/', (req, res) => {
    res.send('Hello !');
});
app.listen(8080, () => console.log('Listening on port 8080!')); 

• The meetup API call:

axios.get(`https://api.meetup.com/${meetupName}/events/?status=past,upcoming\&fields=comment_count`)

• the creation of Elasticsearch indexes:

await client.indices.create({
  index: "meetup",
  body : {
    "mappings": {
      "properties": {
        "time":  {"type": "date", "format": "epoch_millis"},
        "group.name": {"type": "keyword"},
        "yes_rsvp_count" : {"type": "integer"},
        "grouploc": {"type": "geo_point"},
        "venueloc":{"type": "geo_point"}
      }
    }
  }
});

• The creation of another server where our meetup API calls happen, listening on port 8081. We can also notice that we've restricted our server to allow only localhost connections.

localapp.listen(8081, 'localhost', function() {
  console.log("... port %d in %s mode", 8081, localapp.settings.env);

Now in ./clevercloud/cron.json you can notice a cron task, wich will trigger a curl on http://localhost:8081/ every night at 1 AM:

"0 1 * * * /usr/host/bin/curl http://localhost:8081/"

It is this cron that will call our second server to trigger the meetup API calls.

True fact: to use it in its current version, you must keep your application running all the time for one hour of usage maximum. A way to improve the application regarding this issue would be to implement authentication to our application, so we still are the only one having access.

Then remove the cron from this project to have it running in your main application instead. Your main application will be the one consuming this indexed data. Taking advantage on the fact that every virtual machine running on Clever Cloud already has the Clever Tools CLI installed, we could improve our cron to start the application for an hour then stop it when it has finished its indexation job.

So you will end up with two machines, one with your main application, and the second one running for one hour each night.

We must also know that Clever Cloud does not monitor what's going on on port 8081. You could add a logging system or use Elastic APM to monitor your application during its execution time.

This is an approach among many others, do not hesitate to talk with us about your own implementation.

Okay, let's go back to our main goal, and to do so, you can use our sample data meetup list or use your own by replacing the json in the meetups.jsonfile.

Try it out

You can $ git clone the repo in your console, and go into your Clever Cloud console.

Under the organization of your choice, select New, Application, Node. When prompted if you need add-ons, select Elastic Stack, select the plan you need and enable Kibana as an option.

In the environment variables menu of your application, add NODE_ENV=production and add the provided clever remote to your local git folder. Then push using git push -u clever master.

Your deployment will start and thanks to the ES_ADDON_URI we provided in our index.js file, we have nothing else to configure, our application will start sending data to elastic.

Visualize your data and go further

Either in your Kibana or Elastic instance menu in the Clever Cloud console, in the information page you will find a Open Kibana button. Click it and login using your Clever Cloud credentials.

Into Kibana click on the Management (gear) icon in the left side menu. Under the Kibana title, select Index Patterns, then meetup* to see how the data is indexed.

Of course at this point, you are able to do the exact same as I did in the previous article video.

Here is the ElasticSQL query I used in the Canvas demonstration:

SELECT AVG("yes_rsvp_count") AS average, "group.name" FROM "meetup*"
GROUP BY "group.name"
ORDER BY average DESC
LIMIT 5

Happy indexing!

You like Kibana? You will love Canvas

Canvas is a feature that comes inside our beloved Kibana dashboard and allows us to visualize and present our data from our Elasticsearch our way.

Whether you are working with application metrics, security events, infrastructure logs, business-related data or even your side projects, as long as you put the generated info in an Elasticsearch instance, you will be able to display them in an elegant way.

Create workpads with combined data from your Elasticsearch and convert them into beautiful charts, graphs, progress monitors, infographics, reports… The only limit is your imagination (yes, for real)!

You already have a visual identity ? Import your logo, your color schemes, your css to get a consistent interface all over and reuse them everywhere! Share them with your users and customers with ease so your branding is really homogeneous.

Not having a visual identity? There are tons of color palettes and templates you can play around with.

Make your workpads dynamic using time-series data, add your Elastic Maps into your workpads, choose how often it should be updated.

Canvas also lets you customize your workspace with backgrounds, colors, fonts, images and more.

Is it easy to use

What a question! You should know how simple is the Elastic Stack experience!

Canvas arrives out of the box inside Kibana on Clever Cloud, with features like syntax highlighting, mouse-over context-sensitive help, drag and drop publishing, dark mode and more to help you create pixel-perfect visuals with your data.

Use Elasticsearch SQL to design aggregations like you already do in your applications and display them in your workpads.

Canvas comes with its own library of functions to empower your imagination but if it’s not enough, with the Elastic plugin framework, you can also add your own plugins or community ones no matter how experienced in development you are.

And thanks to Clever Cloud, keeping everything in operating condition is pretty simple, as we manage this for you. While as usual providing you a Europe-based infrastructure, GDPR compliant and protected from Cloud Act.

Reuse your workpads everywhere

You might wonder: what can I do with my awesome creations? Well, actually almost everything!

You can, of course, use them in your Kibana, but here are some examples of usage you can have outside of your dashboards:

• export your workpads as JSON
• generate one shot PDF
• automate PDF reporting with a POST endpoint
• display your workpads on websites (and customize them there to match the design of the page)
• get live reporting on screens in your office, retail points or any place
• add real data to your weekly, monthly … emails

Once again, the only limit is your imagination!

Do not hesitate to try out Canvas, it is really easy to use and available in one click and stay tuned, next time I’ll show you how I indexed meetups data into Elastic.

APM by Elastic

Elastic APM on Clever Cloud

Creating the Rails application

Creating the ES stack on Clever Cloud

• Go to your Clever Cloud console, under the organization where your application lives, select Create then an add-on. Scroll down until you find and select Elastic Stack.
• On the next screen select the plan you need, there is no minimal size to benefit from these features.
• On the next screen enable APM and Kibana
• You will now be prompted to name your add-on, and you will be able to select the deployment region. Fine tune as you need and press next.
• Go to the Service dependencies section of your application, then select the APM server in the Link applications drop down menu.

Configure your Application for APM

Ruby on Rails

• with your favorite text editor, edit the Gemfile to add gem 'elastic-apm'.
• Create a file named config/elastic_apm.yml that you will fill with this:

server_url: <%= ENV["ELASTIC_APM_SERVER_URLS"] %> 
secret_token: <%= ENV["ELASTIC_APM_SECRET_TOKEN"] %>

Other languages

Deploying the APM agent

Enjoying your APM / Kibana

Going further

At Clever Cloud, we really love tech conferences. It is a great opportunity for us to share our best stories, show and tell what we do all year long, meet in person a ton of interesting people including our users and offer the best tee-shirts out there, our amazing Sticker Mule laptop stickers and the famous 7in1 pens ;)

If you would like to meet us this year, here is the list of the conferences we will be at until July 2020.

USA

• Dev Nexus, Atlanta, GA, February 19 to 21
• Lonestar Elixir, Austin, TX, February 27 to 28

France

• Devoxx France, Paris, France, April 15 to 17
• Sunny Tech, Montpellier, France, July 2 to 3

Poland

• Elixir Conf EU, Warsaw, Poland, April 29 to 30

United-Kingdom

• Devoxx UK, London, UK, May 13 to 15

Germany

• International PHP Conference, Berlin, Germany, May 25 to 29

Republic of Ireland

• Euro Python, Dublin, Republic of Ireland, July 22 to 24

As we often travel a day before and leave a day after, if you want us to give a talk or meet around the conference, please send an email to deverel (at) clever-cloud.com. Same if you want us to come to a particular conference.

Stay tunned for next upcoming conferences and general news on Twitter Follow @Clever_Cloud

Fat Models and skinny Controllers right?

You know what they say about the MVC design pattern, keep your Controllers light.
So at the beginning of your project, you added oil and gears to your Models and it was rolling just fine. Even if I'm not a big fan of the big Model that even makes coffee (sending an email, really?), I can say that you were in perfect harmony with the framework principles.
Well, a few iterations later, this idyllic codebase seems very far away and your Controllers and Models are way more '200 line'ish than they used to be.

Service Objects is a good solution to extract some logic out of them by transferring code into simple services that handles only one thing. This way, you end up with maintainable, super understandable and separated critical code.
On this article I'm focusing on Controllers, but the principles are the same for Models. Now, some may ask, why not using Controller Helpers instead?

My answer to that would be: don't choose, use both!

Service object and Controller helper

Controller Helpers do hold code shared between Controllers and are also a good way to keep slim Controllers. They are great to respect the DRY (Don't Repeat Yourself) principles. If you target the same workflows in many Controllers, you must consider putting this logic into a helper. The code you put here is a simple succession of actions you do and conditions you are evaluating. They are useful only if used by at least two Controllers, otherwise, you should think of some code refactoring.
Anyway, we often hear that backend logic should not belong to the Controller and just moving everything into a helper won't solve the problem.

For every operation tied to your business logic (calling an API, calculation) performed in this duplicated code or present in any Controller, you should definitely consider creating a service object for each chunk of logic.

Okay, now you know when to use a service object, but what is it exactly?

A service object is a PORO. No, not that cute fluffy League of Legends NPC, but a Plain Old Ruby Object.
You just create a class not related to ActiveRecord (the RubyOnRails ORM) with it's own methods, put your code there, and WOW, you now have a PORO!
We will see how we construct it later, first we need to know where to put it in our project.

Break it to rebuild it

First, let's take this Controller. One thing we can notice is an API call in this get_weather method and this should not be Controller business.

class HomeController < ApplicationController
    require 'curb'
    def index
    end

    def get_weather
        http = Curl.get("#{ENV["api_base"]}#{ENV["api_key"]}")
        if http.status = "200 OK"
            response_body = JSON.parse http.body_str
            @current_weather = response_body["weather"][0]["description"]
            @current_temperature = "#{(response_body["main"]["temp"].to_f - 273.15).round(1)} °C"
        else
            @current_weather = "There was an error sorry: #{current_weather[:error]}"
        end
    end
end

According to the name of the design pattern, we will create a services folder at root_path/app/services/.
Okay, looks like it's about getting weather informations so we will name another folder accordingly.
Finally, we can create our service object file. For this, we will be the most explicit and we will use something like x_service.rb.

Being precise on naming and using the _service suffix is here to help other code maintainers to easily understand what is the purpose of the service and where to find it in the project.

Skeleton

This is how we should start every new service object. Here, we can note the import of the library "ostruct". This allows us to use the OpenStruct object in order to return a rich object containing error messages and a success? method.

class FolderName::NameOfTheService
    require "ostruct"
    
    def initialize(params={})
        @params = params
    end

    def call
        begin
        rescue => exception
            OpenStruct.new(success?: false,
                           error: exception.message)
        else
            OpenStruct.new(success?: true, 
                           error: nil)
        end
    end

end

Filling it up

Instead of just copying our Controller's code into our private method, we will use the wide known begin/rescue pattern to manage errors and will use it at our advantage to raise our own errors.

class Weather::GetCurrentWeatherService
  require "ostruct"

    def initialize(params={})
      @params = params
    end

    def call
      @api_base = ENV["api_base"]
      @api_key = ENV["api_key"]

      begin
        http = Curl.get("#{@api_base}#{@api_key}")
        if http.status == "200 OK"
          response_body = JSON.parse http.body_str
          if response_body["weather"] && response_body["main"]
            weather_string = response_body["weather"][0]["description"]
            temperature = "#{(response_body["main"]["temp"].to_f - 273.15).round(1)} °C"
          else
            raise
          end
        else
          raise
        end
      rescue => exception
          OpenStruct.new(success?: false,
                 error: exception.message)
        else
          OpenStruct.new(success?: true, 
                 weather_string: weather_string, 
                 temperature: temperature, 
                 error: nil)
        end
    end

end

Let's refactor this Controller

def get_weather
    current_weather = ::Weather::GetCurrentWeatherService.new.call(params: {})
    if current_weather.success?
        @current_weather = current_weather[:weather_string]
        @current_temperature = current_weather[:temperature]
    else
        @current_weather = "There was an error sorry: #{current_weather[:error]}"
    end
end

We simply call a new instance of our service in our Controller and use it's success? method to handle regular Controller rendering logic. Note the use of the prepending :: it's for preventing the Controller to look for the service only in its own repository.
Here, the params argument is just an empty hash that I do not use but this was to show that, like every PORO, a service can take many parameters.

This pattern is so great, but I've made a mess!

If our weather service was real, we would probably want to handle city search, forecast weather and so on. Maybe we would have extended our service object or created a bunch of others and now it's an obscure code to maintain and you can feel like you're drowning. No worries, it's just time for you to be rescued by NameSpacing!

In fact we already used it by putting our service in a weather folder in the first place. You should definitely classify all your new services by folders and never hesitate to create a sub folder when your folder holds too many service files.
If your services can be sliced by fields of usage or are really tied to a Model or Controller, create new folders and move your services under the corresponding ones.

For naming your folders, no master rule, but logic, efficiency, and consistency between the names.
Of course, you will need to rename your services. For instance, under a folder named my_folder/my_folder_2 your my_service will be renamed MyFolder::MyFolder2::MyService and you will now use the same name in your Controllers to use them.

So, SO or not?

As you may have notice, I was mostly giving advices on how to implement it and never refer to the official documentation.
This is because this pattern does not have any convention and everyone can implement it its own way. In order to avoid maintainability issues, I would really recommend keeping these concerns in mind:

• always think concerns separation.
• don't let your services grow big.
• always return rich objects.
• all your return objects should have the same structure.

Happy refactoring!