We have been issuing and automatically installing Let's Encrypt® certificates for a while now. The only manual thing was the trigger of this process. But today, we are glad to announce fully automated Let's Encrypt certificates for everyone!

When you add a domain — which targets Clever Cloud — to an application; it will have its own certificate a few minutes later (up to 12 minutes later).

This has been live since 2018-11-16. Hundreds of certificates have been issued since then. This has been possible thanks to Let's Encrypt, who also extended their rate limiting on their API.

How do we do this?

As explained in the previous blog post, queries to the path used by Let's Encrypt to check the ownership of the domain are routed to our Let's Encrypt integration service. This allows us to process the Let's Encrypt queries, get the certificate and give it to our certificates manager.

Here is what's new. Once a user adds a new domain, we periodically check that we can reach our Let's Encrypt integration service. When we do, we start the usual process et voilà.

What if I want another kind of certificate?

That's not a problem:

If you already have a certificate, we will not create a Let's Encrypt certificate.

How about existing domains?

Existing domains which do not yet have a certificate will all get a Let's Encrypt certificate.

This will be done over the next weeks to come. We can't do this in a single batch for two reasons:

• Let's Encrypt rate limiting (we have extended limits but we still cannot send such a big batch all at once)
• We need to spread this out so that we don't have a big batch of renewals every 3 months

If you don't want to wait, you can simply ask us to enable it.

Next steps

There are a few things yet to come:

• Interface in the console to track the status of the certificates
• Support of wildcard certificates (which will not be quite as automatic because it requires DNS validation; this will require an action from you at first)

One last thing

We are now proud sponsors of Let's Encrypt!

We have been working on Let's Encrypt support for quite some time now.

We have a working prototype used for hundreds of domains and it has been running since August 2017 and has handled thousands of renewals without a single hiccup.

To go further, we need to make a small change to the way we route incoming requests.

How does Let's Encrypt work?

Let's Encrypt needs to validate the ownership of a domain before delivering a certificate. As does any public certificate authority.

For our Let's Encrypt integration, we are using the HTTP challenge to validate that we own the web server that a domain points to.

Here is how the HTTP challenge works, basically:

• The client asks for a certificate for a domain (with a CSR)
• Let's Encrypt responds with a challenge id and a token
• The client sets up this token to be sent when receving a request on http://domain.tld/.well-known/acme-challenge/<the challenge id>
• The client tells Let's Encrypt that it's ready
• Let's Encrypt makes the request to http://domain.tld/.well-known/acme-challenge/<the challenge id>
• If the challenge is validated, it will then reply to the client with the certificate

How does Clever Cloud implement this?

What we have been doing since last August is routing the /.well-known/acme-challenge to our Let's Encrypt integration service when a customer asks us to.

Sadly, this means that we have a bunch of rules in our reverse proxies to handle this. As the list of domains grow, it's becoming quite clear that this is simply not technically feasible. This huge list of rules is adding a lot of work to HAProxy's configuration parsing.

As we have hundreds of configuration changes per minute (batched together, but still), this has too much of an impact on the performance of our reverse proxies and the feature is not even released yet!

What changes

Starting today, all requests starting with the path /.well-known/acme-challenge will be sent to our Let's Encrypt integration.

This can be disabled on dedicated reverse proxies for our Premium customers only.

When will this feature be available in the console?

Right now, we only enable this on demand, domain per domain.

The goal, obviously, is to have an interface for this in the console and in clever-tools.

We still have ways to go, but the current target is by the end of this year.

To protect your web app or API, there is almost only one way at this time: TLS. But users and browsers don't always use TLS by default. So what you want is to redirect them to a TLS encrypted version of your site if they try to connect via plain http.

Here is how to do it with Play Framework 2.4 in scala:

Play! and HTTP filters

We will start by creating a "TLSFilter.scala" file and write a TLSFilter class in it:

class TLSFilter extends Filter {
  def apply(nextFilter: RequestHeader => Future[Result])
    (requestHeader: RequestHeader): Future[Result] = {
      if(!requestHeader.secure)
        Future.successful(Results.MovedPermanently("https://" + requestHeader.host + requestHeader.uri))
      else
        nextFilter(requestHeader).map(_.withHeaders("Strict-Transport-Security" -> "max-age=31536000; includeSubDomains"))
  }
}

This part is easy: we implement the apply function by just checking the secure value of RequestHeader. If the connection is not secure, we need to redirect the client to the same url only with "https" as the protocol. If the connection is secure, we pass the request to the next header. Nothing simpler.

Note that we use requestHeader.host instead of requestHeader.domain because the host value is actually the value of the Host header as set by the client, with optional port and stuff.

Note that we create a Filter implementation and not an EssentialFilter one because we do not care about the body.

Next, you need to create a HttpFilters implementation that will hold the instance of your TLSFilter:

// In TLSFilter.scala
class MyFilters extends HttpFilters {
  val filters = Seq(new TLSFilter)
}

And finally, you need to tell Play! to use your Filters class:

# In conf/application.conf
play.http.filters=my.package.MyFilters

Now it will check your requests and permamently redirect the clients to HTTPS.

But, how does Play! know that the request is secure?

Reverse proxies: where did the 's' go?

Now, we need to ensure that Play! knows to differentiate a secured connection from a plain one. If you configured HTTPS in your application, it's quite simple to understand. But it is not always the case:

You most probably did not, configure TLS in your application. And that is because you deployed it on a very powerful and developer-friendly PaaS. So, chances are your Play! application is getting requests in plain HTTP, because the TLS encryption ended at the front reverse-proxy that's piping the request towards your app.

How then is your application going to know that the connection is secured? Enter the non-standard and the standard ways.

X-Forwarded-Proto

The first, non-standard but widely used (e.g. at Clever Cloud) way to know if the request handled by the reverse proxy in front came in a secure channel is to check the X-Forwarded-Proto HTTP header. Like all non-standard headers, you can recognize it by the X- at the beginning of the name.

This header describes how the final client is communicating with the reverse-proxy.

It takes two values: http and https. You can check for that header in your application. But we will see below that Play! can do it by itself.

RFC 2739

Also called the Forwarded HTTP Extension, it standardize the way that a proxy tells the final endpoint what is going on between the final client and itself. It's been published in June 2014.

You can read it here: https://tools.ietf.org/html/rfc7239. But the only thing that is relevant for us is the proto parameter. Like X-Forwarded-Proto earlier, its values that interest us are http and https. Like for the other one, Play! can handle those values for you, if you ask nicely.

How to make Play! handle Forwarded headers?

At the time of this writing, Play! framework support for Forwarded headers have known many states:

• In Play! 1.x, you need to add XForwardedSupport=all in your application.conf
• In Play! 2.0 to 2.3, you need to add trustxforwarded=true in your application.conf

Both these ways only support the X-Forwarded-Proto header.

Now, in Play! 2.4.x, the philosophy is different:

• Define the version of the Forwarded header you want to use: play.http.forwarded.version=x-forwarded|rfc7239
• Set the proxies you trust: play.http.forwarded.trustedProxies=["proxy_ip1","proxy_ip2",…].

Of course proxy_ipX can be an actual IP or a subnet mask, like "0.0.0.0" or "::" to trust every IPv4 or v6, respectively. Defaults are "127.0.0.1" and "::FF".

Also, as the X-Forwarded-Proto header is the one that is widely used in the world, the version default value is "x-forwarded".

What the hell is Strict-Transport-Security?

As you read the filter code, you must have seen that in the case the request is already in HTTPS, we still add a header to the response: Strict-Transport-Security: max-age=31536000.

This is the HTTP Strict Transport Security (HSTS) header. What it does is basically telling the client (most likely a browser): "Next time (and for the next 31536000 seconds), if your user tries to load the unencrypted version of the site, don't wait for me to redirect you and use https already".

The browser (meaning: chrome >= 4.0.211.0, firefox >= 4.0, Opera >= 12, IE >= 11) will then save the website and automatically replace "http" by "https" in the requests the next times.

This mechanism is documented here: https://www.rfc-editor.org/rfc/rfc6797.txt.

You MUST set the max-age value. You can also add includeSubDomains (after a ";" of course), which means "if you get that header while requesting domain.com, please use HTTPS when requesting *.domain.com too". It is a good practice to always add includeSubDomains just in case.

Please note that the STS header can only be set if the website is already TLS protected. You MUST NOT set this header on a non-TLS response.

If you want the browsers to use HSTS before the first request, you can register your domain to be included in browsers preload lists. To achieve that, register your domain here: https://hstspreload.appspot.com/. Also add the preload value to the header, like that: Strict-Transport-Security: max-age=31536000; preload.