Welcome to this new edition of One Framework a Day keeps the Boredom Away. In this series I will show you how to deploy a particular framework on Clever Cloud every day until I want to go back to boredom. Today it's about Sinatra.

In each post of this series we'll see how to deploy a particular framework on Clever Cloud. Today we are taking a look at Sinatra.

If you want to tag along, make sure you have git, a Clever Cloud account and that you have installed our CLI Clever-Tools.

What is Sinatra?

Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort

It's been around for 10 years now and has already inspired several other framework since then. Ratpack which was showcased last week has been heavily inspired by sinatra for instance. Our own Keruspe wanted to play with a couple of Ruby technologies like Puma and Sinatra. And so he built a sample project for it. That's what we are going to deploy today.

Setup

• Start by cloning the sources: git clone https://github.com/CleverCloud/demo-sinatra-puma-activerecord
• Create the database you want to use, in my case Postgres: clever addon create postgresql-addon --plan dev --region eu sinatraPG
• Create the Ruby application on Clever Cloud: clever create --type ruby sinatraSample --region par
• Link the Postgres add-on to the application: clever service link-addon sinatraPG
• Tell Clever Cloud to deploy the API using puma (the default is uwsgi as of now): clever env set CC_RACKUP_SERVER puma
• Tell Clever Cloud to run database migrations before launching the API:clever env set CC_PRE_RUN_HOOK 'bundle exec rake db:migrate'
• Add a domain name: clever domain add mySinatraSample.cleverapps.io

Specify the maximum number of PG connections your application will use (depending on the selected plan) by adding an environment variable like DB_POOL=5 if you want to use 5 connections. For production, you should also set:

clever env set RACK_ENV production
clever env set RAILS_ENV production

Deploy

• Deploy the application with clever deploy

And you are up and running. This application provides an API to store users. They only have a name field. Users can be created with a POST request and retrieve with a GET request like so:

• Create an object with curl --data "regis" https://mySinatraSample.cleverapps.io
• Retrieve that object with curl https://mySinatraSample.cleverapps.io/regis

Everything works out of the box because this application has been written for Clever Cloud. If you take a look at the condfiguration file under ./config/database.yml you should see some environment variables being used. They are the one provided by our Postgres add-on. You can get the full list of variables with clever env.

Let us know your thoughts on this in the comments below. We will be back tomorrow for another post :)

At Clever Cloud, we love giving new technologies a try. Since its inception, rust has always been quite appealing and we always wanted to find projects for which rust would be an adequate solution.

Now that rust is mature enough (to our taste), we started giving it a go, and we needed a way to easily package rust software for our distribution of choice: Exherbo.

Rust package manager: cargo

Amongst the tooling shipped with rust is cargo. Cargo allows you to simply build, test and install a package and its dependencies. It's actually very nice to use when developing.

If we were to install software with cargo, there are two solutions:

• Either we want to install something like a private or local projects and we just have to go to the sources directory, where the Cargo.toml file is, and run cargo install
• Or we want to install something publicly available, on crates.io for example, we can just run cargo install crate_name, crate_name being the actual crate (cargo package) name. We could also pass it a git URL as an alternative.

Regarding production environment and system-wide utilities, we want everything to be managed by the distribution package manager, paludis, so we had to hook up those two and try to make them work together as well as possible.

How packaging rust is different from packaging other software

Currently, rust doesn't support installing libraries system-wide as they're not confident enough in the stability of their ABI yet. What that concretely means is that it's not yet possible to install any rust library, and then use it to build dependent stuff. You have to rebuild all the dependencies each time you build a rust binary.

That particular point is the most noticeable difference on the packaging point of view. Instead of building and installing the dependencies, then the dependents, you only install the final product with everything built in.

What a "traditional" install process looks like in a source-based distribution

For traditional packages, the build and install process is composed of several sequential steps, that we can summarize like this, in that order:

Fetch

This is the step where we download everything, usually the software sources.

This is the only step for which network sandboxing is disabled.

Unpack

Here we unpack the sources in a sandboxed environment. Network sandboxing is on as well as file system sandboxing for this step and all the steps onwards.

Prepare

This is an optional phase, not needed by all packages. It's the moment when we generate build-system files if needed and apply patches to the software sources.

Configure

We decide there which features we want to enable or not for the build.

Build

Where the actual build happens.

Install

We install the resulting binaries/files in a sandboxed location replicating the / hierarchy.

Merge

Once everything gets validated in terms of file system access, everything gets installed to the root file system.

What a rust install process looks like

A rust install pretty much ressembles a traditional one, with one huge exception.

As I said before, cargo install supports either reading a local Cargo.toml, or being passed a crate name or a git URL.

On the other hand, cargo fetch, which is the command that downloads all the dependencies doesn't have these options, it only supports reading a local Cargo.toml and fetching the dependencies listed in it. If libraries were installable system-wide, we wouldn't need this step at all, but since they're not, we have to do this step after unpacking the sources, to be able to access the Cargo.toml.

This has serious implications, it means that we extend the unpack phase with three additional actions:

• disable the network sandboxing, which is really sad and we should never do that
• cargo fecth
• re-enable the network sandboxing

Of course, we don't have to download the dependencies at each reinstall, cargo supports a CARGO_HOME environment variable allowing us to store the downloads in a shared location with all other distfiles.

We'll eventually be able to install libraries system-wide, but I don't see that happening any time soon. The better workaround I could come up with to clean up this hack is to make cargo fetch support being given a crate name or a URL, matching its companion cargo install. Here is the corresponding github issue but I'm not sure whether it will be implemented nor when.

Actual implementation and examples

The implementation of my exlib (which is to exherbo packages what libraries are to software) can be found here.

cargo-watch is a good example of what a rust exherbo package using this exlib looks like. You can see it's fairly trivial and everything is automatic.

At Clever Cloud, we like giving a try to new technologies when they're released, especially if it means safer software. When facebook recently announced the release of infer, we surely wanted to try it out!

The concept of static analysis

The idea behind infer is quite simple: it analyses your code before you actually run it and looks for common code issues or misconceptions which can lead to unsafe code.

You can think of it as some kind of compiler plugin which will do some stricter checks and produce more warnings than your compiler usually does.

The idea of static analysis is really great as you get to know some of your code's problems before even trying to run it. An issue with most static analyzers though is the false positives, those warnings you know shouldn't be there as you perfectly know that this problem is 100% impossible to happen.

Testing our code with infer

We tend to use a lot of different technologies to run our platform. From the languages supported by infer, Java was the most interesting to us. I decided to run a first pass on all of our Java projects.

After getting the code on github and following the installation instructions in INSTALL.md, I started with one of our core small pieces of software.

I didn't expect to see many defects but was afraid of getting a ton of false positives when I ran the test: infer -- mvn clean package. Turned out we got a couple of legitimate warnings about resource leaks when spawning a process and then waiting for it but not destroying it once done, and nothing more! No false positives at all.

For the record, those warnings looked like that:

error: RESOURCE_LEAK resource acquired by call to exec(...) at line 89 is not released after line 89

Fix was as easy as changing return Runtime.getRuntime().exec(cmd).waitFor(); to

Process p = Runtime.getRuntime().exec(cmd);
int res = p.waitFor();
p.destroy();
return res;

As the beginning was encouraging, I went on and ran it on one of our core libraries. One similar problem found, one potential null dereference in an error path and another resource leak warning. The first two were trivial to fix, the last one was a weird one as in some cases it didn't detect a stream to be closed. After refactoring this piece of code which was quite old, the warning was gone, but what appeared to me as being a false positive frightened me when it came to running it on our big API.

The relevant annoying warning: error: RESOURCE_LEAK resource acquired by call to FileReaderHelper(...) at line 28 is not released after line 30

FileReaderHelper makes use of a BufferedReader; changing the scope of the inside reader fixed the issue.

Most of our other projects are really small and no defects were detected at all; then came our API. Our API is the oldest piece of software we're running. As time goes by, we externalise the codebase into separate projects but it's still quite a huge one with its 415 files.

Good surprise: only 20 defects, 18 of them in the same error path which would mean that our platform is unavailable anyways. Amongst the two others, one was clearly a false positive. I added an assertion to silence the warning with a comment explaining why it was there. But the warning wouldn't go. I've since then opened an issue for this which hopefully will be fixed soon.

An example of NULL pointer dereference for which we added checks because "better safe than sorry" when it really should never happen: error: NULL_DEREFERENCE object returned by token.getUserId() could be null and is dereferenced at line 86

Conclusions

Only two false positives in all of our codebase, and a couple of really pertinent warnings makes this project really promising. It doesn't throw a lot of warnings yet (or we didn't hit much) but it reduces the noise by giving only relevant ones and avoiding false positives, which is a really good thing for a static analyser.

With an integration into our CI, it'll now provide yet more guaranties about code safety.

Go and try it for yourself!

At Clever Cloud, git is core in the deployment process. With the years, our needs evolved, especially in terms of performance. This is how we dealt with it.

A bit of background first

When we started Clever Cloud, we chose to use gitolite to manage our git repositories. It appeared to be a complete and functional solution. We used it internally first, for managing our own internal repositories even when the product wasn't released yet.

After several months of testing, we were convinced that it was a really good solution, and chose to go with it.

Managing gitolite configuration

gitolite is primary designed to be configured manually by a sysadmin. You have to describe every user, group of users and repositories in configuration files. Gitolite will then use these files to generate its internal stuff for dealing with access rights. Gitolite ensures repositories are created and hooks propagated.

We were in the need of automatic reconfiguration, for quite obvious reasons. We therefore developed a tool called etilotig which aim was to keep gitolite's configuration up to date. On startup, this tool would load its initial configuration from the API and then listen to AMQP events to update its configuration cache and write the new gitolite configuration accordingly.

It worked actually quite well for longer than we expected, and we used it in production until May 5th 2015.

The drawbacks of gitolite

gitolite was great but had a few major drawbacks we weren't happy with.

• as previously said, it wasn't meant to be dynamically configurable, which required some non-trivial hacks
• it required duplicating the configuration in both the API and gitolite itself
• all the repositories were created in a same directory
• at each repository creation, it did a full pass on all the repositories to check whether the hooks were up to date or not
• rewriting only part of its configuration wasn't trivial at all so we ended up rewriting most of it for each change

Those problems were annoying but not critical at the beginning, but some of them became quite interesting to us.

The fact that all repositories are created in a same directory is a huge performance issue when the number of repositories become really high.

We actually dropped the part that checks for the hooks at each repository creation from the gitolite code a while back as it took half the time of the gitolite internal configuration regeneration on update.

The new etilotig

While gitolite used to be efficient, lately it had a lot of performance problems, and depending on the timing of the events, it could take up to five (!) minutes to create a new repository. This went far beyond our acceptable limits, so we had to find another solution.

The idea was simple: drop gitolite totally and improve our configuration management tool etilotig to do what was needed by itself.

The checklist we needed to accomplish was quite tiny:

• ssh keys management
• authorization management
• repositories creation
• hooks installation

Ssh keys management

When its goal was only to manage the gitolite configuration, etilotig only forwarded them to gitolite, which in turn handled them. Now, we have to manage the authorized_keys file by ourselves.

We basically write a new one each time an ssh key is added or removed and then replace the old one with the new one.

Each line is printed as such:

command="AUTH_SCRIPT USER_ID",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty PUBLIC_KEY

With AUTH_SCRIPT pointing to the authorization script that I'll speak of later, USER_ID being the user id of the key owner and PUBLIC_KEY being their public key.

The authorization script will thus be called with the user id as first parameter.

That solves the first point of our TODO list.

Repositories management

At startup, etilotig writes some bash configuration files defining basic things such as the directory in which repositories have to be created.

Then we have a tiny bash script in charge of the repository creation. We now create them in a deeper directory hierarchy to get as few repositories as possible per directory. Say we had /data/app_18c6021b-0860-4f97-a08d-0663f45cf3f0.git before, we now have /data/app_18/c6/02/app_18c6021b-0860-4f97-a08d-0663f45cf3f0.git which makes things waaaaay faster.

#!/bin/bash

create_repo() {
local repo_dir="${1}"

mkdir -p "${repo_dir}"
pushd "${repo_dir}" &>/dev/null
git init --bare
popd &>/dev/null
}

main() {
    local repo="${1}"
    local repo_dir
    
repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"

[[ -d "${repo_dir}"/hooks ]] || create_repo "${repo_dir}"
}

. "${HOME}"/.etilotig/.etilotigrc

main "${@}"

Then we have another one for hooks installation.

#!/bin/bash

shopt -s nullglob

main() {
    local repo="${1}"
    local repo_dir
    
repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"

for hook in ${HOME}/.etilotig/hooks/*; do
ln -sf "${hook}" "${repo_dir}"/hooks/
done
}

. "${HOME}"/.etilotig/.etilotigrc

main "${@}"

With those two simple scripts, we only have to call them for each repository at startup to ensure everything is OK, and at each repository creation, which solves two of the four points from our TODO list, only one left.

Authorization

Now, the goal was not to duplicate the configuration anymore, but rather use the configuration from the API, thus externalising the whole thing. Dropping all the configuration management from etilotig reduced its size by more than 50%.

The way etilotig manages authorization is quite simple: when it generates its internal configuration, it actually generates a perl script which is called on each ssh connection attempt. The script then authorises or not the transaction.

We made a similar script which prints the users some information about which repositories they have access to if they just run ssh git@push.par.clever-cloud.com or such, checks if they are authorized when they try to git push/pull or rejects any other request.

It looks like this (with some extra stuff added):

#!/bin/bash

sanity_check() {
    if [[ -z "${SSH_CONNECTION}" ]]; then
        echo "Who the hell are you?" >&2
        exit 1
    fi

if [[ -z "${SSH_ORIGINAL_COMMAND}" ]]; then
        export SSH_ORIGINAL_COMMAND="info"
fi
}

ask_for_info() {
    local userid="${1}"

# make the request to the API to retrieve user info message
echo "some info"
}

ask_for_authorization() {
    local userid="${1}"
    local appid="${2}"

# make the request and return the HTTP status code here. 200 means authorized.
echo 200
}

authorize() {
    local userid="${1}"
    local verb="${2}"
    local appid="${3}"
    local ret=1
    case "${verb}" in
        "git-receive-pack"|"git-upload-pack")
            local code
            code=$(ask_for_authorization "${userid}" "${appid}")
            [[ "${code}" == "200" ]] && ret=0
            ;;
    esac
    return "${ret}"
}

final_abort() {
    echo "What are you trying to achieve here?" >&2
    exit 2
}
main() {
    sanity_check
    local userid="${1}"
    local verb
    local repo
    local repo_dir
    verb=$(echo "${SSH_ORIGINAL_COMMAND}" | cut -d ' ' -f 1)
    repo=$(echo "${SSH_ORIGINAL_COMMAND}" | cut -d ' ' -f 2 | tr -d "'\"")
    [[ "${repo}" == /* ]] && repo=${repo:1}
    repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"
    if [[ "${verb}" == "info" ]]; then
        ask_for_info "${userid}"
    elif authorize "${userid}" "${verb}" "${repo}"; then
        export CC_USER="${userid}"
        export CC_NOTIFY_SCRIPT="${HOME}/.etilotig/send-push-event"
        exec "${verb}" "${repo_dir}"
    else
        final_abort
    fi
}
. "${HOME}"/.etilotig/.etilotigrc
main "${@}"

With this in place, nearly everything was ready. Two tiny hooks on top of that to only allow users to push on the master branch, and to trigger a deployment on git push:

hooks/update

#!/bin/bash

main() {
    local rev="${1}"
    if [[ "${rev}" == refs/tags/* ]]; then
        exit 0
    fi
    if [[ "${rev}" != "refs/heads/master" ]]; then
        echo "You tried to push to a custom branch."
        echo "Only master is allowed."
        exit 1
    fi
}

main "${@}"

hooks/post-update

#!/bin/bash

sanity_check() {
    local rev="${1}"
    if [[ "${rev}" == refs/tags/* ]]; then
        exit 0
    fi
}

main () {
    local rev="${1}"
    sanity_check "${rev}"
    local repo=$(basename $(pwd))
    local appId=${repo/.git/}
    local commitId=$(git rev-parse "${rev}")
    "${CC_NOTIFY_SCRIPT}" "${appId}" "${commitId}" "${CC_USER}"
    echo "[SUCCESS] The application has successfully been queued for redeploy."
}

main "${@}"

Conclusion

That's it, we have our new git server manager up and running, which works pretty well.

The performance gain? We went from between 3 and more than 5 minutes to less than 1 second per action, while dropping the whole gitolite codebase and reducing the size of etilotig by 50%, with an average performance gain of 30k%.

gitolite has been very useful both in its utilisation and its codebase to better comprehend the whole authentication mechanism, so a great thanks to this awesome tool.

Now gitolite is dead, long live etilotig!

At Clever Cloud, we use a lot of Java. A large part of our API is written using JEE (aka Java Enterprise Edition). Here is how we implemented the http client we use for our Github integration.

The standard HTTP implementation in JEE is Jersey, so that's what we'll use here.

A bit of background first

Our API is designed in a "micro-services" way. There's a central part and various external components.

Here, we'll be dealing with the central part, which itself is internally designed in a sort-of "micro-services" way. JEE offers a powerful dependency injection mechanism called CDI, which we use to build a lot of small tools we can then use just as we need.

Serialization

When you're building APIs and clients, one of the problematics is about serialization. We made a tool to do just that which you'll see in the code as MapperHelper, namely mh. It's basically a wrapper around jackson's ObjectMapper.

Let's get to work

We'll be working on an internal tool called GithubAPIHelper. We'll not get through the whole of it, but you'll quickly get the concept.

First, some standard JEE boilerplate injecting required tools using CDI.

@Stateless
public class GithubAPIHelper {

   @Inject
   private MapperHelper mh;
   @Inject
   private LogsHelper lh;

   private final static String GITHUB_URL = "https://api.github.com";
}

Now, we want to build an HTTP client using jersey-client, this is pretty straightforward. Btw, we create a method to make the client directly point at Github.

private Client getClient() {
   if (client == null) {
      client = ClientBuilder.newClient(new ClientConfig().property(ClientProperties.FOLLOW_REDIRECTS, true));
   }
   return client;
}

private WebTarget getTarget() {
   return getClient().target(GITHUB_URL);
}

What about authentication? Let's create a new method to get an authenticated request to Github:

private Invocation.Builder getBuilder(WebTarget target, String token) {
   return target.request().header("Authorization", "token " + token);
}

Ok, now that we have those basic pieces in place, we can start requesting for real.

Let's get the basic user profile, for starters

First, we create a basic data structure that we'll use to deserialize the payload from Github. I'll skip most of the available fields here as they're not needed.

public class GithubOwner implements Serializable {

   public int id;
   public String name;
   public String login;
   public String email;
   public String avatar_url;
}

Now, you should all be yelling at me about those public fields. Let me remember you that this data structure is only used for deserializing purpose, it won't ever be used for anything else, so what's the point in complicating things?

Now let's get back to business, we have a way to construct authenticated requests, and a model in which to store the data. So let's do it!

First, let's get to the right endpoint: /user

   getTarget().path("/user");

Now, let's actually issue the request, and get a response

   Response r = getBuilder(getTarget().path("/user"), token).get();

Let's deserialize the data from github

   GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);

And here we are, with some integrity checks added:

   public GithubOwner getSelf(String token) {
      Response r = getBuilder(getTarget().path("/user"), token).get();
      if (r.getStatus() < 300) {
         try {
            GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);
            return owner;
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return null;
   }

Dealing with pagination

Getting the user's profile was quite easy and straightforward, but everything isn't that smooth.

Let's say we now want all the repositories, instinctively, here is what we'd do:

public class GithubRepository implements Serializable {

   public long id;
   public String name;
   public String description;
   public GithubOwner owner;
   @JsonProperty(value = "private")
   public boolean isPrivate;
   public String ssh_url;
   public String git_url;
}

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      if (r.getStatus() < 300) {
         try {
            List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), new TypeReference<List<GithubRepository>>() {
            });
            repos.addAll(new TypeSafeArrayList<>(ms));
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return repos;
   }

That will actually work, but you won't get all of them, only the first 20 or so. If you want all of them, you'll have to follow the pagination.

First, we want to get the link marked as rel=next in the Link http header.

   private String getNextLink(Response r) {
      String link = r.getHeaderString("Link");   /* <http://foobar>; rel="next", <http://blah/>; rel=last */
      if (link != null) {
         String[] links = link.split(",");
         for (String l : links) {               /* <http://foobar>; rel="next" */
            if (l.contains("rel=\"next\"")) {
               String[] tmp1 = l.split("<", 2);
               if (tmp1.length == 2) {
                  return tmp1[1].split(">", 2)[0];  /* http://foobar */
               }
            }
         }
      }
      return null;
   }

Next, we want to request this url and get the next data to our initial request

   private Response getNextData(Response r, String token) {
      String link = getNextLink(r);
      if (link == null)
         return null;
      return getBuilder(getClient().target(link), token).get();
   }

Now let's put all the pieces together, addind a loop to retrieve all the data

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      while (r != null) {
         if (r.getStatus() < 300) {
            try {
               List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), newhTypeReference<List<GithubRepository>>() {
               });
               repos.addAll(new TypeSafeArrayList<>(ms));
            } catch (IOException e) {
               lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
            }
         }
         r = getNextData(r, token);
      }
      return repos;
   }

And voila, you now have a working Github HTTP client. Once you got these, the other endpoints are really easy.

A couple of days ago, Github made breaking changes regarding the way other companies can interact with github organisations.

Usually, this wouldn't mean anything to you and you wouldn't need to do anything, but this time, Github requires you to explicitly approve Clever Cloud in order for your github applications to work properly.

If you want to be able to create new applications from one of your github organisation's repositories, or if you want a git push to github to launch a deployment on Clever Cloud, you have two available solutions:

• The recommended one: Go to github, click on the settings (the little cog) button at the top right of your screen and then click on Applications in the right column. If your github account is linked to your Clever Cloud one, you should see a line saying Clever Cloud API with a View button next to it. Click this View button. At the bottom of your screen, you can now grant Clever Cloud access to your organisations, or request an admin to do it.
• The "simpler" but not recommended one: Go to github, click on the settings button at the top right of your screen, click on an organisation to access its settings, then click Third-party access and then Remove restrictions. Note that this solution can have several security issues.

As a side note, github now requires more rights to be enabled to have access to an organisation repositories. You can make sure the rights associated to your account are up to date by logging out and then logging in again using github login.

Once you've done this, everything should work back as expected.

• One-click import of ssh keys from GitHub
• Application creation from a GitHub repository
• Automatic deployments of several apps via a push to GitHub

1. Grab this secret and go to the GitHub repository settings.
2. Click on "add a service" and select "Clever Cloud" from the list.
3. Paste the secret

Did you like [Heartbleed?]({{ site.basepath }}/features/2014/04/08/openssl-101g-update.html) Meet Shellshock — aka CVE-2014-6271 — a new bug discovered this week in the widely used Bash command line interpreter.

First things first

Are you safe at Clever Cloud?

Yes. Yesterday afternoon (September, the 24th), a patch was released by the bash developpers to address this issue.

A member of our team, Kevin Decherf, then submitted an updated bash package with this patch to the distribution we use: exherbo.

The patch was reviewed by several members of the core exherbo team and finally validated by me, both as member of Clever Cloud and of the exherbo core team at around 5PM (CEST).

The update was then propagated inside our Cloud platform and all the critical virtual machines got bash updated today.

What about you, <localhost>?

You really should care about this new vulnerability.

It can compromise especially Apache web servers using CGI scripts with Bash invocation, making your system vulnerable to remote-code injection. OpenSSH and some DHCP clients are affected as well on machines that use Bash.

When I started to really dig into my system to fully understand how it works, breaking everything to know of to fix it, I decided to do even worse and started using scm packages.

Scm packages download the source code directly for the upstream development repository using git, subversion, mercurial or whatever other vcs.

Doing this I ended up with a more-that-bleeding-edge system, causing a lot of breakages when stuff went incompatible with recent versions of other. That's perfect, since that's exactly the state I wanted to reach. If I wanted to recover a functional system, the easy solution that I adopted when I did not have much spare time was to revert my last changes and get back to an earlier version of the guilty package, but this was not the goal of the operation. What I really started to do at this point was to patch the broken stuff to make it compatible with the newer version of the guilty stuff which broke everything. It was sometimes really trivial, sometimes way less. I mostly did this for Gnome projects. It's a really good experience since you have to dig in a lot of projects and documentation, making you know better how various parts of your system work internally. Once I had everything back working, I then submitted my patches to the developers of the projects upstream, so that it gets fixed for everyone.

Now that I have way less spare time, I no longer have such a bleeding edge system, or at least I have way less scm packages. Sometimes I still need to patch stuff though, so I still use the same procedure.

My workflow

My workflow for contributing to upstream is not much different to the one I use for contributing to Exherbo. I also use an autopatch mechanism which is slightly different. My hook for automatically patch software is available there: https://github.com/Keruspe/paludis-config/blob/exherbo/hooks/ebuild_prepare_pre/patches.bash. If I remember correctly it was initially written by Ciaran McCreesh, the paludis's lead developer. When I want to patch something, I get a copy locally, I write and commit my patch, I generate a patch file and move it to /etc/paludis/autopatch/<category>/<package>/ where <category> is for example "x11-dri" and <package> is "mesa", according to exherbo's packages name. Each time I install a package, it applies all this patches before configuring and compiling the software. If it works, I submit it upstream, if not, I fix it and retry.

I highly recommend you to try contributing to open source projects, to fix or improve them, like adding new functionalities. The only risk is to learn a lot.