A few weeks ago I've open sourced issues-helper. It allows you to easily interact with issues on GitHub and GitLab, from the command line.

gli o --assignee clementd --label improvement "Degauss the frobulator fluxfield"

Let's see why and how it's done.

TL;DR

Issues are a good way to keep track of decisions in a project. Opening issues should be easy so people don't forget to write down why they've done things. Opening issues from the command line with minimal fuss is a good thing:

gli init # generate initial configuration
gli o "Issue title" "Optional issue text"
  # add --open to open the issue in the browser once created
  # add --assignee to assign issue
  # add --label to add labels to the issue
gli l # list issues
gli b # open the project homepage in your browser

Issues. Use them.

At Clever Cloud, we use git (and GitLab) a lot. In addition to just using git for code, we also rely a lot on issues. Many discussions end with "please open an issue and sum up what we've just said".

Issues are not just for bugs, we use it as a way to track decisions. Some projects don't even have code and are just there to track discussions and decisions.

My goal, as a CTO, is to make sure we keep track of all of our decisions. So I need people to open issues, to open them often, and to open them early.

Processes. Make 'em easy.

I work with people, not with robots. While I trust them and I know they're good at what they do, I just can't expect them not to make mistakes, or in that case follow processes with no clear incentive. So I needed to remove friction when opening issues. When we interact with code, it's from the command line: we craft commits, we deploy to production, we even close issues from the command line (while crafting commits). So we should also open issues from the terminal.

My first try was to use the ruby gitlab CLI, but it had several issues:

• configuration was done with environment variables
• it's written in ruby and can be complicated to install
• the output is more or less database rows in ascii tables
• you have to specify which project you're talking about

Even though I've written documentation and showed people how to use it, it was not used a lot. When a tool feels like a hack, nobody wants to use it.

What I wanted

I needed two things to get a tool people would use:

• automatically know which project we're talking about
• config files to avoid having to mess with env variables (and leak GitLab tokens)

To avoid ruby installation issues, I've decided to use Rust. There's a lot of available libraries, the build system works well, it's sufficiently expressive to let me express what I want, and in the end I get a nice binary.

XDG basedir. Respect it.

Reading GitLab credentials from a file is not really hard. One common mistake is to put config files directly in ~ like some kind of animal. Instead of cluttering users' home directories, I followed the XDG basedir spec. Most languages have a library which handles it for you, so it really is a no-brainer (and it works out of the box on Windows as well).

Infer information from context

To know where the issues should be opened, issues-helper looks at the origin remote of the current git repo. From that, it knows what to do. Even if you're not currently in the right repo,

j my-project # autojump is great
gli o "My issue" # that's it

Tools. Open Source them.

Once I had this working, not only people started using this tool, but people improved it: @keruspe fixed a lot of issues quite quickly. So the next step was to un-hardcode every Clever Cloud specific things in the codebase (then rebase everything, fun afternoon).

Once it was on GitHub, I got more contributions (from inside and outside). Also, it made me add GitHub support: not being able to use issues-helper to manage its own issues quickly got very frustrating.

Supporting GitHub also made me contribute on the GitHub crate, thus closing a vertuous circle.

There are a few issues left. Feel free to chime in, I'll help you get started. It's a good way to start working on a Rust project!

How it's done

The tool itself doesn't hold a lot of business logic. It parses a project location from a git remote, reads config from a file, sends a few HTTP requests and displays useful information.

CLI arguments parsing

It's one of the most important parts of a program, since it's the UI. The Rust ecosystem has clap which handles everything: parsing, help and even autocompletion.

Unfortunately, the parsed values are exposed through a Map<String,String>, so even though parsing checks structure, you have to recreate the structure yourself. In practice, you end up with a bunch of value.unwrap(); // this should never fail.

Fortunately, there's Structopt, which generates clap config from Rust data structures: all the dirty stuff is handled for you, and you get nice structs and enums. Yay!

Config files

Config is in a toml file, config file location is handled by the xdg crate. Remember: every time you add files to ~, a cute puppy dies.

So

Having processes in a team is a good thing, but if you don't document them, you can't expect people to respect them. If processes don't make everyone's life easier, you can't expect people to respect them. One simple way to make processes stick is to provide tooling to make them the easiest path. A bash script is cool, but mainly for its author. A stable CLI tool with a good UI, written in a robust language is not a lot of work and will age way better.

Post illustration: The Sorcerer's Apprentice, Disney Movies. Friden calculating machine. Credits: Disney

We've been quietly rolling out support for hooks during deployments. This may come as a surprise if you listen to our ramblings about deployment processes, as we've always pushed for self-contained, zero-adherence builds. Don't worry, we haven't changed our minds. Let's see why and how doing things outside of the build system can make sense.

Our stance is unchanged

At Clever Cloud, we're uncompromizing on automation. The first step to automate production is to have robust, reproducible builds. The best way to be robust and reproducible is to get rid of implicit dependencies: if you have more than two operations to run in sequence to build and run your project, then you have a problem. Not just for deployment, mind you, but for your day-to-day productivity: working on your application becomes complicated, and onboarding new developers takes hours (or worse).

How are Node.JS apps deployed on Clever Cloud? Simple: npm install && npm start. What about scala applications? sbt stage. No problem, no vendor lock-in. That's nice and clean. Unfortunately it's not the whole story.

The case of frontend build

It's got to be the most common question we have about our deployment system: "how can I integrate my frontend build in the deployment phase"?

For Node.JS, easy, just add your build phase to scripts.install. For scala applications, you can configure sbt-web. For PHP applications, you can hook in composer. Hooking the frontend build in the main build can range from super easy to dwarf-fortress-fun. Some build tools allow to run arbitrary code, some are more restrictive (but more predictable). For instance, building Haskell applications with stack makes it quite hard to shell out and run npm install.

Monolithic builds are not the only option

In a full-stack setup (eg Ruby on Rails or Play! framework), integrating the front-end build in the build system is a good idea, as it gives you better integration (eg. play run also refreshes compiled assets). If you're going with a minimalist tech stack however, integrating everything makes less sense, as your needs for the development environment are not aligned with your needs for the production environment. Let's say you work on a scotty web app, with some assets compilation. For development, no need to bake everything in the web lib, you can use Devloop and have automatic recompiling on reload, both for frontend and backend.

In that case, instead of shoehorning the frontend build in a stack setup, using a hook as an escape hatch is the most time-effective way to do it; it also gives you more control on when build tasks happen. This allows you to leverage our build system more efficiently.

Available hooks and when to use them

A complete rundown of hooks is available in the documentation, but I'll give you a concrete example to give you a better understanding of what you can do.

Please give us feedback

I've created the new hooks system to scratch an itch, so it may change a bit in the following months. Please reach out to the support team if you have questions or suggestions.

Since its first release, we've been quietly improving our CLI tool, clever-tools. Today, we've released version 0.7.0, with better support for node 7 and many improvements.

New features in 0.7.0

The most important change is better support for node 7. The installation phase is now way faster for node 7 users.

Another important feature is clever ssh, available in preview mode for clever-tools users. clever ssh allows you to ssh into a running instance for debugging purposes (eg. list files, view logs locally, …). Please get in touch with the support if you want to try it out. We'll release complete documentation and examples once this feature hits GA.

Why use Clever Tools?

The web console is nice to use and doesn't require any installation. Why bother with clever-tools then? Here are a few use cases where clever-tools really shines.

Perfect for terminal-heavy multitaskers

If, like me, you're working on several projects in several terminal windows, going back to the web console to check logs and deployments can get tedious. With clever-tools, you can work on different projects without ever leaving your comfy terminal window. You can launch 5 deployments in parallel and see how they go, side-by-side without having to switch from app to app in your browser.

Fine-grained access to logs

The web console is great for displaying logs as they come, but reading old logs requires a lot of scrolling. If you want to inspect logs around a specific date/time to investigate e.g. a user report, worry not:

clever logs --after "2017-02-08T08:28" --before "2017-02-08T08:32"

Also, if you want to search for a specific term you can just grep the output:

clever logs | grep NullPointerException

The best part: new lines will appear in real time.

No SSH configuration necessary

From version 0.6.0 upwards, clever deploy pushes over HTTPS and doesn't require SSH configuration. If your platform makes ssh configuration more difficult than it should be (I'm looking at you, Windows), then just use clever deploy.

Nowadays, we know that null is to be avoided. It's been dubbed the billion dollar mistake by its own creator, and the dreaded NullPointerException everyone knows about. Yet, when it comes to getting rid of nulls, nobody agrees.

null is bad, m'kay

• NPEs
• hard to read (is this value always defined?)
• handling possibly undefined values is tedious

The issue is that we focus on null, and not on the actual problems it causes. If your only goal is to get rid of NPEs at all costs, then you will pay those costs. Dearly.

The memory safety issues caused by null are solved on most platforms. In many languages, null is a pointer to a specific value with specific properties, so not literally a null pointer anymore, and on modern OSs, the memory protection system will prevent your code from directly accessing the address 0x00.

So the big risk is having your program blow up (NPE, segfault, panic, …) because of an unexpected missing value. The core issue is that you were unable to express that the value was possibly missing; the crash is a consequence of this issue. Yet many people try to solve the issue by preventing the crash (null check, null object pattern, …) instead of handling the core issue: a value could be missing and you weren't able to express it.

Stop chasing NPEs and fix your domain model

If you're trying to get rid of NPEs by removing the difference between "there is a meaningful value" and "there is no value", then not only you're not solving your issue, but you're making it far, far worse.

Your domain model didn't let you express statically the possible absence of a value, but the presence of null at least gave you a stuctural difference at runtime. That's why putting default values or worse, following the null object pattern, is terrible. You had structural information, but couldn't use it in a rigorous way, so you're just throwing it all away because of an implementation detail.

If you're searching for a replacement to null to denote missing values, it has to be structurally different from a regular value (it cannot belong to the domain of the value you may have). So if you're in a typed system, then a value that may be not there has to have a different type from a value that's definitely there. In an untyped language, you can't have a static difference, so using null is kind of OK (though there are better solutions).

For example, in java you can do:

// Bad
String notThere = null;
String there = "my string";
// Turbo-bad
String notThere = ""; // or "N/A", etc
String there = "my string";
// Good
Optional<String> notThere = Optional.empty();
Optional<String> there = Optional.of("my string");

With optional, not only you have a structural difference between defined and undefined strings (instead of having to check if the string is equal to "" or "N/A", without any guarantee that it's not the actual value), but it's clearly documented in the type.

Optional is available in Java 8 (and in Guava if you're not using java 8 yet). In scala, rust, ocaml and many other languages, it's called Option, in Haskell it's called Maybe.

This is why you can have nice things

Ok so now you have a proper representation for your optional values. In a typed language it means that you're forced to check if the value is already there before using it.

For instance, in java:

Optional<Integer> parseOptionalString(Optional<String> myOptionalString) {
    if(myOptionalString.isPresent()) {
        Optional<Integer> result = parseInt(myOptionalString.get());
        return result;
    } else {
        return Optional.empty();
    }
}

This works perfectly well, but is still as verbose as using explicit null checks. So if you're only concerned about code terseness (you really shouldn't), it can feel like a marginal improvement (it's way better than that).

Thankfully that's not the idiomatic way of handling Optional values: now that we can denote the abstract concept of being possibly not there (in this case, Optional<_>), then we can do useful stuff about it:

• transforming it only if it's defined (with map)
• eliminate the option by providing a default value when we don't need the information anymore (with orElseGet)
• sequencing several operations returning options (with flatMap)

All of this comes for free because we were able to clearly define our domain model. Some languages like Kotlin provide approaching solutions with things like the Safe Call Operator for chaining operations (instead of using map and flatMap) and the Elvis Operator for providing default values. This, however, is less extensible and less composable than having proper options (map and flatMap are not specific to Optional).

Optional<Integer> parseOptionalString(Optional<String> myOptionalString) {
    return myOptionalString.flatMap(parseInt);
}

Once you have properly defined your domain model, then you can have cleaner code thanks to the information encoded in your types. Aiming at terser code without proper abstraction will lead you to perlish nightmares.

To sum up

Using null causes problems. Resorting to solutions that erase information (setting a default value too early or worse, the null object pattern) will cause graver and subtler problems.

In order of decreasing importance, the problems with null are:

1. it's not clear for people if a value can be missing
2. the program can blow up if somebody forgot to handle a missing value
3. it's tedious to sequence operations returning null

Don't settle for solutions that only address #2 and #3.

At Clever Cloud, we've started using microservices before it was cool™. Today we're excited to launch service dependencies, to simplify microservices management.

The microservices graph

When you ditch your monolith and go full microservices, your application becomes a graph of loosely coupled microservices.

For instance, an typical service API can depend on:

• an auth microservice to validate requests
• a notification gateway to send notification

In turn, the notification gateway can depend on:

• an email gateway
• a push notification gateway

So you have a graph of services, each with its own lifecycle (eg dev, staging, production). So if you want to use a different email gateway from your staging API, you need to have in turn a specific notification gateway. To make a test in one project, you need to modifiy code in all its dependency chain. Uncool

Service dependencies

On Clever Cloud, you already know how to link applications to their dependencies on the fly, without touching code or fidling with config files: by linking an addon to your application, you let the addon inject its location and credentials so you don't have to handle it yourself. Service dependencies is just a generalization of the addon mechanism to any application.

Link applications

You can add dependencies to an application either from the CLI:

  clever service --alias api link-app notification-gateway

or from the console:

Exposed configuration

With addons, the exposed configuration is always the same. With applications, you can declare the configuration you want to expose to your dependents. To make an app expose configuration, you can either do it from the CLI:

  clever exposed-config --alias api set API_DOMAIN_NAME api.example.com

or from the console:

Note: every time you update the exposed configuration, dependent applications will be automatically redeployed.

As big users of PostgreSQL, we had the opportunity of re-thinking the idioms common in the world of relational DBs.

Today, I'll talk about why we stopped using serial integers for our primary keys, and why we're now extensively using Universally Unique IDs (or UUIDs) almost everywhere.

TL;DR

Use UUIDs as primary keys. They can be freely exposed without disclosing sensitive information, they are not predictable and they are performant.

Primary keys, technical keys and semantic keys

A relational database is a graph where nodes are called entities and edges relations. To be able to express a relation between entities, we need to be able to uniquely refer to any entity: that's the role of a primary key.

The role of a primary key is to provide a stable, indexable reference to an entity. You have two ways to construct your primary key: either as a semantic key, or as a technical key.

A semantic primary key is extracted from the entities attributes (ie you use one or several fields of your entity as its primary key).

A technical primary key is completely unrelated to the fields of its entity. It's constructed when the entity is inserted in the DB.

Ideally, semantic keys would be better, as it allows for a very simple way to understand and compare entities. But relational DBs are built upon a fundamentally mutable model. The current state of the world is stored in the database, and entities are subject to mutation. Relational DBs are very good at storing a relatively small amount of data, and are not well suited for immutable databases.

That's why to be able to have stable primary keys, we commonly resort to serial IDs. Serial IDs are a technical key, as they're not related to the actual contents of its entity.

To be fair, you can configure your foreign key to have ON UPDATE CASCADE to have primary key updates automatically propagated everywhere within the database, but it doesn't solve the problem if you've exposed primary key values outside the DB.

What's a serial ID?

Basically, a serial id is a number that increases everytime you insert a row. The database does the bookkeeping for you, so it's transparent for the developer.

create table entity(
  entity_id serial primary key,
  attribute text not null
);

We can then insert values in the table; postgres will handle the entity_id field.

> insert into entity (attribute) values ('ohai');
INSERT 0 1

> select * from entity;
    entity_id | attribute 
-----------+-----------
            1 | ohai
(1 row)

The row has been inserted, and the entity has an automatically attributed id. In postgresql, serial is implemented through a sequence, which acts a bit like a counter. We can inspect its current state:

> select currval('entity_entity_id_seq'::regclass);
currval 
---------
        1
(1 row)

OK, so we have a perfectly good primary key for our entities. What's the problem?

Why serial IDs can be a problem

Information disclosure

The current value corresponds to the last value used as a primary key. Since the sequence starts at 1 and is incremented 1 by 1, currval roughly corresponds in our case to the number of rows in the table (not exactly, because of deletions).

So, if you want to count the number of rows in a given table, all you have to do is to have the system generate a new entity and inspect its ID.

So, do users have access to primary keys? The answer is often yes: primary keys are very often exposed to the user, most often in URLs, as we want URLs to be stable (see cool URIs don't change).

Let's say you're a company operating a SaaS, all it takes to know your user count is to create an account. A number of social networks boasting impressive user bases were pwn3d this way. Just by creating an account and looking at a link.

Entity enumerations

An even bigger problem is that it's very easy to enumerate the entities in any given table. You start from 1, and you keep on incrementing the value. It becomes very easy to scrape all your entities. It leads to spamming (just don't start from 1, as you don't want to spam the admin team ;-) ), or worse. If you have light access control, only based on the knowledge of a given URL, then anybody can display private information.

Non uniqueness across tables

Since each table has its own sequence, an identical value will be found as the primary key of different entities. Now imagine you make a tab-completed typo when deleting a row. Yeah, you've just deleted an arbitrary row somewhere in your DB. Let's hope you don't have too many ON DELETE CASCADE configured.

Workarounds

You can configure your sequence to start at an arbitrary point, have an increment bigger than one, or even share the sequence between tables. In any case, you're still disclosing information about your growth, and you'll just reach integer overflow quicker. Not really a perfect solution.

A word on UUIDs

You may already have seen stuff like c0b656b1-7351-4dc2-84c8-62a2afb41e66 somewhere. This is a UUID (or Universally Unique IDentifier).

Which is nice. UUIDs are guaranteed to be Universally Unique, which makes them good candidates for primary keys (and neatly solve the issue of having primary keys non unique across tables).

More precisely, UUIDs are 128 bits values, with textual representation in hex digits. It bears repeating, because many people think that UUIDs are stored as text.

UUIDs are 128 bit values

There are different versions of UUIDs, based on how they're created. Within the 128 bits, 4 are used to encode the version. The kind we're interested in is UUIDv4, which is based on randomness.

UUIDv4 are perfectly well indexed by PostgreSQL

UUIDv4 values are random

UUIDv4 values being random, you don't have a guaranteed uniqueness. However, the probability of a collision is rather small (see Random UUID probability of duplicates).

Also, keep in mind that in the extremely unlikely case (you're more likely to get hit by a meteorite) of colliding UUIDs, it will be caught by the DB thanks to the primary key constraint.

UUIDv4 adds two reserved bits to the already used ones, so we're left with 122 bits of information.

λ> 2 ^ 122
5316911983139663491615228241121378304

This should be enough for your needs.

The fact that UUIDv4 values are random gives us interesting properties:

• you can't enumerate values
• you don't know how many values are present in each table
• you don't have to talk to the database to construct a value

The two first properties are nice, scroll up if you don't remember why. The third one deserves some extra explanations.

Sequences are mutable state, you need atomicity on get-and-increment, to preserve uniqueness. So everytime you need to insert an entity, you need the DB to give you its primary key. The entities in your code are incomplete until you talk to the DB. Since a sequential key is technical, not semantic, this can usually be handled by your ORM (or data mapper, ORMs are bad, m'kay?). When inserting several related entities, this can cause additional problems, because you're not only lacking the primary keys, but also the foreign keys too, which are semantic, not just technical.

The UUIDv4 generation algorithm is well documented and available virtually anywhere, so you can generate whole entities without ever talking to the database. This makes the code simpler and clearer as you don't have to handle partially-created entities, and you can test your entity-generation code without having to talk to a giant blob of mutable state over the network.

Also, serial is a 32 bit integer. You'll reach overflow at some point. Debugging it and migrating a production database to 64 bit integers is an interesting experience, but maybe you can take my word for it and avoid it altogether. With UUIDs it's a problem you just don't have.

Also, if you're brave enough to run a SQL database in a multi-master setup (please, please, please, DON'T), then you're relieved from the shenanigans of having conflict avoiding sequence generation. But please don't run SQL databases in multi-master setups. I have done that: one star, would not recommend.

UUIDs in PostgreSQL

PostgreSQL supports UUIDs, through the type uuid.

> select '697dffc9-8ba5-410c-b677-227475a73530'::uuid;
uuid                 
--------------------------------------
    697dffc9-8ba5-410c-b677-227475a73530
(1 row)

By default PostgreSQL doesn't have the ability to generate UUIDv4 values. You need the uuid-ossp extension for that. From 9.4, UUIDv4 generation is also provided by the pgcrypto extension.

CREATE EXTENSION "uuid-ossp";
-- or CREATE EXTENSION "pgcrypto";
create table entity(
  entity_id uuid primary key default uuid_generate_v4(),
  -- or entity_id uuid primary key default gen_random_uuid(),
  attribute text not null
);

> insert into entity (attribute) values ('toto');
INSERT 0 1
> table entity;
                entity_id               | attribute 
--------------------------------------+-----------
    b2a3d43b-5b81-4127-be29-d849de607d78 | toto
(1 row)

You can use UUIDs without the uuid-ossp extension, but you will have to provide UUIDs when inserting entities (which is a good thing to do anyway).

When UUIDs are not what you need

If you don't have mutable state, then you don't need a technical primary key, and you can use a semantic key.

If you have strong storage constraints, UUIDs can cause a problem, since they use 2 to 4 times as many space as a serial or big serial. So if you have strong size constraints and if you don't expose the primary keys, then UUIDs may be overkill. Also, UUIDs being random, you lose locality, and your index ends up scattered. This causes a definite performance hit when confronted to high insert rates. Also, this creates extra disk use.

In my experience, UUIDs are a safe default. You can question it when you have strong performance constraints and a safe environment.

How to generate UUIDs in $LANGUAGE

Java, Scala

It's in the standard library.

java.util.UUID.randomUUID

Node.js

npm install uuid

var uuid = require("uuid");

uuid.v4();

PHP

Install ramsey/uuid from composer.

$uuid4 = Uuid::uuid4();

Haskell

Install uuid from hackage

uuid :: IO UUID
uuid = nextRandom

Here at Clever Cloud, we're enthusiastic users of functional programming. It helps us a lot when it comes to have clean and maintainable code.

That's why we were delighted to fly to Bologna for the LambdaCon, to meet the Italian FP community as well as some twitter friends.

After a stop at Yoox offices where Quentin and I got the opportunity to have a chat with developers and the other speakers, it was time for the actual conference.

Fun with Categories

Many interesting functional programming concepts come directly from a branch of mathematics called Category Theory. The keynote speaker Bartosz Milewski went as far as saying that CT is the next revolution after mainstream FP adoption.

I was surprised to witness that "Kleisli Category" frightened people less than the dreaded M-word.

Category Theory is a vast field and can be intimidating for newcomers due to its extremely high level of abstraction, but the good news is that Bartosz is writing an introduction book aimed at developers.

Dissecting the rabbit

Since we use RabbitMQ a lot at Clever Cloud, we were happy to be able to attend Alvaro Videla's talk about RabbitMQ's internals, especially the actor hierarchy in place inside RabbitMQ.

Streams on top of Scala

We're big users of Scala and we deal with a lot of streamed data. That's why we've been using Scala's streaming libraries a lot, first with Play's iteratees, then with scalaz-streams and akka-streams.

Quentin did a nice round-up of what's available and the different use cases for streaming libs in Scala.

The slides are available on slideshare: Streams on top of Scala

TDD as in Type-Directed Development

Even though I've started my career with dynamicallly typed languages, I've been very happy with static, expressive type systems these last years. In this talk I showed how to use types not only for safety, but as a very powerful design and communication tool.

You can have a look at the slides: TDD, as in Type-Directed Development

The european FP community is great

The people from CodersTug did a great job for their first conference. And it's great to see such forward-thinking events take place in Europe.