At Clever Cloud, we love giving new technologies a try. Since its inception, rust has always been quite appealing and we always wanted to find projects for which rust would be an adequate solution.

Now that rust is mature enough (to our taste), we started giving it a go, and we needed a way to easily package rust software for our distribution of choice: Exherbo.

Rust package manager: cargo

Amongst the tooling shipped with rust is cargo. Cargo allows you to simply build, test and install a package and its dependencies. It's actually very nice to use when developing.

If we were to install software with cargo, there are two solutions:

• Either we want to install something like a private or local projects and we just have to go to the sources directory, where the Cargo.toml file is, and run cargo install
• Or we want to install something publicly available, on crates.io for example, we can just run cargo install crate_name, crate_name being the actual crate (cargo package) name. We could also pass it a git URL as an alternative.

Regarding production environment and system-wide utilities, we want everything to be managed by the distribution package manager, paludis, so we had to hook up those two and try to make them work together as well as possible.

How packaging rust is different from packaging other software

Currently, rust doesn't support installing libraries system-wide as they're not confident enough in the stability of their ABI yet. What that concretely means is that it's not yet possible to install any rust library, and then use it to build dependent stuff. You have to rebuild all the dependencies each time you build a rust binary.

That particular point is the most noticeable difference on the packaging point of view. Instead of building and installing the dependencies, then the dependents, you only install the final product with everything built in.

What a "traditional" install process looks like in a source-based distribution

For traditional packages, the build and install process is composed of several sequential steps, that we can summarize like this, in that order:

Fetch

This is the step where we download everything, usually the software sources.

This is the only step for which network sandboxing is disabled.

Unpack

Here we unpack the sources in a sandboxed environment. Network sandboxing is on as well as file system sandboxing for this step and all the steps onwards.

Prepare

This is an optional phase, not needed by all packages. It's the moment when we generate build-system files if needed and apply patches to the software sources.

Configure

We decide there which features we want to enable or not for the build.

Build

Where the actual build happens.

Install

We install the resulting binaries/files in a sandboxed location replicating the / hierarchy.

Merge

Once everything gets validated in terms of file system access, everything gets installed to the root file system.

What a rust install process looks like

A rust install pretty much ressembles a traditional one, with one huge exception.

As I said before, cargo install supports either reading a local Cargo.toml, or being passed a crate name or a git URL.

On the other hand, cargo fetch, which is the command that downloads all the dependencies doesn't have these options, it only supports reading a local Cargo.toml and fetching the dependencies listed in it. If libraries were installable system-wide, we wouldn't need this step at all, but since they're not, we have to do this step after unpacking the sources, to be able to access the Cargo.toml.

This has serious implications, it means that we extend the unpack phase with three additional actions:

• disable the network sandboxing, which is really sad and we should never do that
• cargo fecth
• re-enable the network sandboxing

Of course, we don't have to download the dependencies at each reinstall, cargo supports a CARGO_HOME environment variable allowing us to store the downloads in a shared location with all other distfiles.

We'll eventually be able to install libraries system-wide, but I don't see that happening any time soon. The better workaround I could come up with to clean up this hack is to make cargo fetch support being given a crate name or a URL, matching its companion cargo install. Here is the corresponding github issue but I'm not sure whether it will be implemented nor when.

Actual implementation and examples

The implementation of my exlib (which is to exherbo packages what libraries are to software) can be found here.

cargo-watch is a good example of what a rust exherbo package using this exlib looks like. You can see it's fairly trivial and everything is automatic.

As I said in part 3 of the “knowing your system” saga, I really started to dig into my system when I used Gentoo. I quickly spotted the limits of portage, its default package manager and switched to paludis.

It really is easy to mess up your python installation, especially when you’re not a python developer and you don’t care about it. Of course there are tools to help you, such as python-updater, but it won’t be of great help if you’re not in a basic “breakage because of python update” case. Because of portage being written in python, you end up with an unmaintainable system which is painful.

A pretty good example of this is when I tried install Gentoo on my Playstation 3. At this time, the powerpc stage was kinda old. During the base system installation and update, you ended with the linux headers version being incompatible with your python installation, and portage was unusable. After 3 attempts without success, I tried once starting by installing paludis, and then doing everything with it. It worked like a charm.

Portage is not the only limitation of Gentoo. Its workflow is kinda insane. First of all, Gentoo aims to get every piece of software packaged in a same repository, external overlays being there only for testing purpose before merging. You end up with tons of packages being available on your machine, 95% of them won’t ever interest you. You could say it’s cool to get everything available without needing internet access to retrieve them, but you’ll still have to download the source tarballs so it’s pointless, it just make dependencies resolution slower.

Another problem is contribution. To contribute to Gentoo, you’ll open a bug on their bugzilla with a patch attached (I’m fine with this first step, the next ones are awful). When a developer or a proxy maintainer sees it, he applies it locally and push it to the centralised CVS server (which you sync using rsync on client side…). There is a HUGE lack of consistency in this process:

• As a simple contributor, you won’t ever be the author of the real commit, at most you’ll be mentioned in the commit message
• You don’t use the same tool to pull changes in that the one developer use to push them.

Exherbo is awesome

After having used paludis on Gentoo for one year or so, I decided to involve myself in its community. I started to send a couple of patches for my purpose, and to help beginners to deal with it. I then heard of another bigger project they were launching: a brand new source-based distribution, a kind of Gentoo rewritten from scratch in a more modern, modular, clean and strict way.

While I was still using Gentoo, I started to install exherbo in a chroot, to give it a try. First thing I really enjoyed was that everything is split into several repositories. If I do not have any C# application installed, the repository containing the mono stuff won’t be installed and those packages won’t be available on my system. If I finally decide to install mono, I will automatically be suggested to install the mono repository, and running cave resolve -x1 repository/mono will do all the stuff for you. Everything is managed by git both on the developer and “user” side. Being an exherbo “user” is kinda special, since every user is considered as a developer by the community.

Contributing to exherbo is really easy and smooth, I’ll explain my contribution workflow next week.

As exherbo is a “new” distribution, not every package is available, so you’ll probably end up creating your own repository to package your software. Once you get enough packages in it and you want to share it with the community, you just have to submit it, people will review it and tell you how you can make it better. It will then be added to the list of available repositories so that people will be guided to it when they try to install software you’ll have packaged. For example, at Clever Cloud, we use openstack for now, and we had to package it and its dependencies. Everything is available in our repository.

One more thing: exherbo policy is to be as vanilla as possible. Every patch used in packages must contain the upstream status of it, since they all have to be submitted. We want to maintain as few patches as possible.

My exherbo repository

Clever Cloud exherbo repository

Source-based distributions for huge server pools

At Clever Cloud, we decided to use a source-based GNU/Linux distribution called exherbo, which I’ll blog about next week, because of its strictness and flexibility. Since we have to manage hundreds of servers and virtual machines, it would have been a big overload in energy consumption and in time invested if we managed it the “conventional” way. Indeed, compiling everything on the hypervisors could cause instabilities because of the CPUs being monopolised by the compilation process, leaving no power to virtual machines. The virtual machines would have the same problem, and for much longer, since you do not have the same power in a virtual machine than in an hypervisor, so compilations last longer.

This is why we decided to manage everything using binary packages. Wait… What? Binary packages in a source-based distribution? How?!

The search is over: paludis’ pbins

Setup

Paludis comes with a very nice feature: pbins.

The concept is simple:

• You create an empty repository, anywhere in your filesystem, containing an empty packages directory, a profiles
  directory containing a file named repo_name which contents is your binary repository name, like mybinaries.
• In your repository, create a metadata containing a layout.conf file, with masters = arbor in it.
• On your compilation node, create a configuration file for you binary repository
• On the other nodes, create a configuration file for it too. It’s the same, without the lines starting with binary_
  and modifying the sync line

A sample configuration file:

format = e
location = /var/db/paludis/repositories/mybinaries
sync = file:///var/db/paludis/repositories/mybinaries
importance = 100
binary_destination = true
binary_distdir = /var/cache/paludis/distfiles
binary_keywords_filter = amd64 ~amd64
binary_uri_prefix = http://mybinaries.com/exherbo/

location is the place where you put your empty repository.

sync is the same that location on the compilation node, and may refer to a git repository where you’ll publish your binary repository on the other nodes.

binary_distdir is the directory where binary tarballs will be placed. I recommend you to make it point to the same place as where paludis downloads its distfiles, since it will be easier to maintain in further use. That’s why I left the default value here.

binary_prefix is the URL where the generated tarballs will be available for the other nodes. Ensure that the /var/cache/paludis/distfiles directory is available via http at this URL.

Usage

You’re now fully ready! All you have to do is create the binary packages on your compilation node, push the updated repository to your git server, sync it on the other boxes and you will be able to install your freshly made binary packages without having to compile them on all of your machines.

It’s pretty simple to make binary packages, all you have to do is first to generate the binaries for everything you have installed:

cave resolve -xc world --make binaries --make-dependencies all

It will automatically generate packages and put tarballs in your distfiles directory. If you run this after updating your compilation box, it will only generate new binary packages for those that have changed.

Last thing you might want to know: to create a binary package for a package you do not have installed yet, just run

cave resolve -x --make binaries --make-dependencies all <insert a package name here>

It will create packages for all the dependencies, installing it afterwards, and finish by making the package for the software you asked.

Don’t forget to push all changes on your git server!

As we last saw, I’m really fond of source-based distributions. Beyond the administration and the management of an installed system, you also have to install it in the first place. Because of the fact that you have to compile absolutely everything in your system, it may be really scary to install one.

I formerly told you to keep in mind that your system is fully usable whilst you’re upgrading it…

When you install a source-based distribution, it’s not using a shiny interface and clicking repeatedly on the “next” button. You have to use a “bootstrap” system, which can be a livecd such as the awesome sysrescuecd, but you also can decide to first install a binary distribution like Fedora or Ubuntu (you may probably already have one, so you obviously can use it).

When you have booted your “bootstrap” system, there will be 3 steps:

• Mounting the target partition to anywhere in your filesystem, like /mnt/newsystem
• Unpacking the base filesystem of your distribution into that folder (these are commonly called “stages tarballs”)
• Chrooting into this folder (you may have to mount several subsystems such as /dev beforehand, refer to the distribution
  documentation)

One you have chrooted, you will actually be (for the current shell) in your bare new system. However you can start to manage and install it from a shell within your “bootstrap” system that you can hopefully use to work during the installation. Note that you can also leave your computer do all the compilations during your idle time at night.

Here are two install guides that I recommend you to follow:

• Gentoo install guide
• Exherbo install guide

Paludis, the other package mangler

Now that you have all the minimal informations needed to understand what is a source-based GNU/Linux distribution and to understand how it works globally, We’ll give a closer look to the heart of the distribution: the package manager.

On Gentoo, the official package manager is portage, with the emerge command line. I used it the first weeks I tried Gentoo, but when I really wanted to play with my system, it was too restrictive so I decided to switch. Moreover, portage is written in python (so are a lot of core components of gentoo), and when your python installation gets broken… Game over.

When exploring the list of available packages, I found out paludis and decided to give it a try. Thanks to a provided script, I translated roughly my portage configuration to a paludis one (Paludis can use portage configuration but this is not recommended). It was not perfect, but was a good start. After cleaning and updating it a little by myself (the documentation is really exhaustive), I could start using it. The man pages are also really complete.

You can see my current configuration here: https://github.com/Keruspe/paludis-config. A first configuration will be a lot lighter, this is a configuration that have evolved during 4 years, from Gentoo to Exherbo.

Paludis command line cave is a modular tool which allow you to do a lot of things. Basically everything you would ever want to do, and some more.

The main cave subcommands are:

• cave resolve looks for the whole dependency tree of a package, in pretend mode by default, and you then can apply
  this resolution (to install stuff) by specifying the -x argument
• cave uninstall is the pendent of cave resolve for uninstalling
• cave purge looks for unused packages (as for all commands, pretending by default, and execution with -x)
• cave fix-linkage looks for broken binaries (because of libraries updates) and suggest you to rebuild them.

A lot of other subcommands are available, and a lot of options for each of these. Amongst other things, cave allows you to stop the installation process at a certain phase, or resume at another. This allows you to abort at compile phase to apply custom patches, and them resume at compile phase to compile the package with your patches applied. It also support a hooks mechanism.

We’ll see some more advanced paludis features in a next post of the “knowing your system” saga.

Last thing, The way I update my system is:

cave sync
cave resolve world -c -km -Km -Cs

It syncs repositories with upstream to get latest versions of the packages, and then resolve world which is the set containing all the packages you have installed, with a complete dependency tree, as deep as it can, with -c aka --complete and reinstalling all packages for which metadata have changed because of -km -Km aka --keep if-same-metadata --keep-targets if-same-metadata. If a failure occurs, it continues to build the rest while the dependencies still are satisfied thanks to -Cs aka --continue-on-failure if-satisfied. I then run this last command again with -x aka --execute to apply the available updates.

What is a source-based GNU/Linux distribution?

The principle may be a bit scary, but is actually simple.

In most distributions (called binary distributions), software are packaged to be “ready to use”, all configured by the distribution maintainers, and you then get a list of available packages that you can install. When you ask for a package to be installed, it will download the software and its dependencies and install everything.

In source-based, the package manager will also handle all the dependencies, downloading and installing stuff. The thing which changes with them is the thing you actually download, and what is being done between the downloading and the installing phases of the process.

Instead of downloading the software, the package mangler will download its source code, unpack it, configure it and then compile it. You’re basically compiling your whole system with those distributions.

Why “wasting” that much time makes sense?

As you probably know, compiling can be very long. It mostly depends on your hardware, and most of the projects will compile in a couple of minutes, but some of them, like the compiler itself or your web browser can take up to several hours on certain boxes.

This can be seen as a PITA but this is the price of real liberty. With source-based distributions, you can choose which components of each software you want to build, and exactly which options you want. With binary distributions, if you’re missing a feature, you’ll have to do everything by yourself outside your package manager, and this will really be a PITA to maintain. With source-based distribution, it’s way easier to contribute, as we’ll see in a later post of this saga. The package does not contain all the binaries, it’s just a text file you have to edit to add an option you’re missing. Everything becomes easier to customize, you become the God of your system.

You must also keep in mind that while you’re compiling stuff, your system is still fully usable, so you can just do it in background.

The Gentoo example: My beginning with source-based distributions

My first GNU/Linux distribution was Ubuntu, which seemed to be popular (and which still is), a binary distribution of course. I was quite happy with it for the first months, but as soon as I wanted to explore my system more deeply, like compiling my own kernel, or when I wanted to do really specific operations, I was immediately limited by the design of this distribution.

I was a student at this time, and my class-mate (and now colleague) Kevin Decherf told me he was using Gentoo. on his server. I immediately asked him if he agreed to plan an Install Party the week just after that. During this session, I installed a minimal system, discovering the distribution. At the end of the day, it was barely booting to an xterm.

I wanted to really explore my distribution to understand exactly how everything works, so I started “playing” with it, modifying packages, checking how things reacted. I broke my system and reinstalled it 5 times in 3 weeks, playing harder and harder, until I knew how to make my system work again after a wanted major breakage.

You really should try source-based distributions, to really get how things work. It can take a few months to understand them if you just play a little with them on an irregular basis, but it’s really worth it. Gentoo was my choice since it’s the most popular source-based distribution, and the only one I heard of at this time. It’s quite good, but you’ll see in my next post that I didn’t really like its default package manager, and I no longer use Gentoo.