Some open source git repos host really useful code but are sometimes not maintained.

If you use that code on a regular basis, you may need to fork it, improve it, and use your enhanced version of that code.

Then someday the original repo is updated, a brand new version of the code you forked is released, and either you will lose all your changes or you will miss the new features.

To get the best of the two versions, you need to merge them.

There are many solutions and any that fits your needs is valid.

My problem was to merge up to 50 commits and I hate having to fix so many conflicts; it can take days. I also knew there was code that would need refactoring, so my idea was to avoid merging the whole changes in one batch. In this paradigm the am command is my best git option.

What is git am?

git am is a super useful command which can be used for many things as it’s reading and parsing the mailbox containing the commits you specify and allows you to use only parts of the extracted data such as the code diff, the autor, the message…

In this case, I used it to cherry pick a commit from a repo and add it to another, but this command is way more powerful than this.

You can find the whole documentation here https://git-scm.com/docs/git-am

knowledge required

UNIX shell basic usage

How do we use it?

For this use case, you need to have the two repos locally.

Let’s say we have this architecture:

$ demo ls
my-forked-repo   the-last-version-repo

Go to last version repo

$ cd the-last-version-repo/

Use the following line for each commit you want to import in the new version

$ git --git-dir=../<My-forked-repo>/.git \
format-patch -k -1 --stdout <your commit SHA> | \
git am -3 -k

Here is the full explanation of the command: https://stackoverflow.com/a/9507417

If there’s no conflicts, console will just display

$ Applying <your commit message>

And if you type in your console:

$ git status

You will see that you have a commit ahead of HEAD.

I’ve got a conflict

If for some reason the commit cannot be merged automatically, console will display a message with the conflicting files list and a small explanation of the failure. In this case, you can get more information with:

$ git am --show-current-patch

I want to solve it

To resolve the conflicts, go to the files, manually keep the code you want then go to your console and add them to git:

$ git add <path/to/file>

After adding the files to git, you will be able to do:

$ git am --continue

which will commit the files for you. A $ git status will produce the same result as before.

I do not want to solve it

In the case you want to handle the conflict without git, you can just

$ git am --abort

Which will abort all the commits you’re trying to import. Or you can use:

$ git am --skip

which will abort the import of the current commit having a conflict. In both cases, the files will be left as they were before you typed the command.

Notice that you can’t use these commands to resolve a conflict triggered by another git command (e.g git merge…)

I imported my commit, so what now?

Basically, nothing, you’re all good to go, push the commits to your branch whenever you want with

$ git push <your remote name> <your branch name>

And that’s it. If for some reason you need to update the message of the commit, use

$ git commit --amend

Careful, it only works with the last commit. There are ways to do the same with an older commit, but it involves rewriting the tree and this will change the commit SHA as well, read the git documentation for more details.

If you made a lot of changes on a commit that wasn’t originally yours, it may be accurate to change the author to yourself, for this use

$ git commit --amend --author="Author Name <email@address.com>"

On your last commit. For olders commit, it’s exactly as above.

Happy merging!

We have some news about our dear clever-tools, the Clever Cloud CLI for humans, robots and others.

Back in January, we released version 0.10.1 which contained some new commands and a few bug fixes. Today we're releasing version 1.0.0!

You can find all the details about the user oriented changes in the CHANGELOG. In this article, we'll cover the structural changes related to the project:

• Improved error handling = better usage in CI/CD pipelines
• ES2015 refactor + ESLint = better contributing experience
• Goodbye nodegit, hello isomorphic-git!
• Improved Jenkins build = more release methods and beta channel (npm, deb, rpm, arch, homebrew and chocolatey)

Now let me tell you the story of this release...

Improved error handling = better usage in CI/CD pipelines

It all started with issue 195. With the previous version, when a user tried to clever deploy a non existing branch, the error message was a bit too raw. Our colleague Philippe wanted emojis, but we had to decline 😭.

While investigating this error message and how to improve it, we realized that the command was not returning a proper error exit status. This is a very important rule to follow when you're writing a terminal command/script:

• If everything went well: return exit status 0
• If something went wrong: return exit status 1 (or another one more specific)

Our CLI is built for humans AND robots. We want to empower our terminal addicted users. Moreover, we want to empower our users to automate anything related to Clever Cloud in their CI/CD pipeline. In such cases, returning a proper error exit status is crucial to mark a CI/CD job as successful or failed.

We took a quick look at the rest of the codebase and noticed that other commands were concerned by this problem. We reworked the code of clever deploy and decided to apply the same error handling to all the other commands:

• For humans: Always have an [ERROR] prefix (colored in red) when we display an error message.
• For robots: Always log error messages to stderr.
• For robots: Always return 1 as exit status when something goes wrong.

This obviously required some significant work. We had to go through all the 28 commands. Nevertheless, we're sure this will help you to build more robust and automated jobs, interacting with the Clever Cloud platform.

ES2015 refactor + ESLint = better contributing experience

While we went through the codebase, we took the opportunity to do a bit of refactoring.

First, we decided to update all our code to ES2015+ syntax. We tried to use most of the modern good practices of the JavaScript ecosystem when it made sense (destructuring, rest parameters, fat arrows...). We also decided to setup an ESLint config file (based on StandardJS) to enforce our codestyle for the next iterations. This lint check is now part of our automated test suite on Travis CI which is triggered on each pull-request.

This big refactoring took some time but it allowed us to reduce the number of lines of code while improving the overall consistency and readability. We hope this will drive more users to customize the clever-tools and propose pull-requests for bug fixes and feature requests.

Looking at how easily the new clever console was added by an outside contributor, it seems like we're on the right path.

Goodbye nodegit

During this refactoring, we updated most of our dependencies but one of them required special attention: NodeGit.

NodeGit is a node.js module which provides bindings over libgit2, a portable and pure C implementation of the git core methods. This is commonly referred to as a "native module" in the node.js ecosystem. Each time you install NodeGit, npm will try to build libgit2 from source (in some situations, it will try to download pre-builds). Because we want our users to have a straightforward installation experience, this native module thing had many drawbacks:

• In our Travis CI config, we had to install some native dependencies so npm could build libgit2 from source.
• Some of our users tried to install the clever-tools via npm and had difficulties because of this.
• Our users don't all use the same version of node.js, this also multiplied the number of weird cases we had to support.
• Each time we wanted to update NodeGit, it was a puzzle to resolve.

All those problems pushed us to stop releasing the clever-tools via npm. ⚠️ SPOILER ALERT: it's back for version 1.0.0.

The second problem is that we use pkg. This tool is great. It lets you compile a node.js project to different portable binaries for MacOS, Windows and GNU/Linux. We've been using it for a while to provide various installation methods. When we dropped installation via npm for 0.10.1, the binaries built with pkg where the only official way to use the clever-tools.

This mix of NodeGit + pkg complicated the situation even more:

• The binaries built with pkg did not include native dependencies. We had to distribute the right nodegit.node file along with our main binary file and users had to have it in their PATH.
• On some GNU/Linux distributions, we had problems with native dependencies like libcurl-gnutls, libnghttp2-git or libssh2

We already looked for a replacement solution but a fairly new one came up to our attention in February...

Hello isomorphic-git

isomorphic-git is a "pure JavaScript implementation of git that works in node and browser environments". The project was created by William Hilton. This guy is awesome!

As I said, we heard about it in February but we only tried to use it for our use-case in July. It does not support all the git feature set (yet) but it has strong foundations and we only need a few things for the clever-tools:

• list and resolve branches
• list and resolve commits
• add/rm remotes
• push to repos (via HTTPS)

We were missing a few features so we reached out to William to see if we could help and/or contribute. In the end, we contributed a few features in the codebase and we sponsored the hosting of the CORS-proxy which is used to demo the project in browsers.

By replacing NodeGit with isomorphic-git, we can say goodbye to most of the problems we described earlier:

• no need to worry about different node.js versions
• no need to have native dependencies on our CI server to build/test the project
• no need to distribute nodegit.node with the pkg binary
• no need for a user to have the right native dependencies on his/her system to use it
• npm releases are back

Improved Jenkins build = more release methods and beta channel

Because we decided to ditch NodeGit, we were able to simplify the packaging (no more nodegit.node and no native dependencies). This required some rework on our Jenkins builds but it was a good occasion to migrate 5 different jobs to one multibranch pipeline project.

We'll discuss the details of this new build system and how we set it up in another article. For now though, here's what changed:

• All platforms:
  • Added: installation via npm is back
• GNU/Linux:
  • Added: rpm packages (stable & beta) with a yum repo on Bintray
  • Added: deb packages (stable & beta) with an apt repo on Bintray
  • Added: Exherbo exheres clever-tools-bin
  • Changed: dedicated AUR package for beta versions
• MacOS:
  • Changed: dedicated homebrew tap for beta versions
• Windows:
  • Changed: chocolatey packages are now automatically published (no more manual upload, no more manual review from chocolatey's team)
  • Changed: chocolatey packages are no longer published on chocolatey.org
  • Changed: chocolatey packages are published (beta & stable) on Bintray but you can still use the choco CLI to install them (see chocolatey installation docs for the specifics).

The structural changes of this release will help us to be more reactive on the project. We'll be able to publish new features and bug fixes more easily and more often. Hopefully, this will improve the quality of the clever-tools and push more customers who only use the Web console to try the CLI way of life.

We're looking forward to read your feedbacks on this release through are usual channels: support chat, email, Twitter, GitHub issues…

At Clever Cloud, git is core in the deployment process. With the years, our needs evolved, especially in terms of performance. This is how we dealt with it.

A bit of background first

When we started Clever Cloud, we chose to use gitolite to manage our git repositories. It appeared to be a complete and functional solution. We used it internally first, for managing our own internal repositories even when the product wasn't released yet.

After several months of testing, we were convinced that it was a really good solution, and chose to go with it.

Managing gitolite configuration

gitolite is primary designed to be configured manually by a sysadmin. You have to describe every user, group of users and repositories in configuration files. Gitolite will then use these files to generate its internal stuff for dealing with access rights. Gitolite ensures repositories are created and hooks propagated.

We were in the need of automatic reconfiguration, for quite obvious reasons. We therefore developed a tool called etilotig which aim was to keep gitolite's configuration up to date. On startup, this tool would load its initial configuration from the API and then listen to AMQP events to update its configuration cache and write the new gitolite configuration accordingly.

It worked actually quite well for longer than we expected, and we used it in production until May 5th 2015.

The drawbacks of gitolite

gitolite was great but had a few major drawbacks we weren't happy with.

• as previously said, it wasn't meant to be dynamically configurable, which required some non-trivial hacks
• it required duplicating the configuration in both the API and gitolite itself
• all the repositories were created in a same directory
• at each repository creation, it did a full pass on all the repositories to check whether the hooks were up to date or not
• rewriting only part of its configuration wasn't trivial at all so we ended up rewriting most of it for each change

Those problems were annoying but not critical at the beginning, but some of them became quite interesting to us.

The fact that all repositories are created in a same directory is a huge performance issue when the number of repositories become really high.

We actually dropped the part that checks for the hooks at each repository creation from the gitolite code a while back as it took half the time of the gitolite internal configuration regeneration on update.

The new etilotig

While gitolite used to be efficient, lately it had a lot of performance problems, and depending on the timing of the events, it could take up to five (!) minutes to create a new repository. This went far beyond our acceptable limits, so we had to find another solution.

The idea was simple: drop gitolite totally and improve our configuration management tool etilotig to do what was needed by itself.

The checklist we needed to accomplish was quite tiny:

• ssh keys management
• authorization management
• repositories creation
• hooks installation

Ssh keys management

When its goal was only to manage the gitolite configuration, etilotig only forwarded them to gitolite, which in turn handled them. Now, we have to manage the authorized_keys file by ourselves.

We basically write a new one each time an ssh key is added or removed and then replace the old one with the new one.

Each line is printed as such:

command="AUTH_SCRIPT USER_ID",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty PUBLIC_KEY

With AUTH_SCRIPT pointing to the authorization script that I'll speak of later, USER_ID being the user id of the key owner and PUBLIC_KEY being their public key.

The authorization script will thus be called with the user id as first parameter.

That solves the first point of our TODO list.

Repositories management

At startup, etilotig writes some bash configuration files defining basic things such as the directory in which repositories have to be created.

Then we have a tiny bash script in charge of the repository creation. We now create them in a deeper directory hierarchy to get as few repositories as possible per directory. Say we had /data/app_18c6021b-0860-4f97-a08d-0663f45cf3f0.git before, we now have /data/app_18/c6/02/app_18c6021b-0860-4f97-a08d-0663f45cf3f0.git which makes things waaaaay faster.

#!/bin/bash

create_repo() {
local repo_dir="${1}"

mkdir -p "${repo_dir}"
pushd "${repo_dir}" &>/dev/null
git init --bare
popd &>/dev/null
}

main() {
    local repo="${1}"
    local repo_dir
    
repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"

[[ -d "${repo_dir}"/hooks ]] || create_repo "${repo_dir}"
}

. "${HOME}"/.etilotig/.etilotigrc

main "${@}"

Then we have another one for hooks installation.

#!/bin/bash

shopt -s nullglob

main() {
    local repo="${1}"
    local repo_dir
    
repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"

for hook in ${HOME}/.etilotig/hooks/*; do
ln -sf "${hook}" "${repo_dir}"/hooks/
done
}

. "${HOME}"/.etilotig/.etilotigrc

main "${@}"

With those two simple scripts, we only have to call them for each repository at startup to ensure everything is OK, and at each repository creation, which solves two of the four points from our TODO list, only one left.

Authorization

Now, the goal was not to duplicate the configuration anymore, but rather use the configuration from the API, thus externalising the whole thing. Dropping all the configuration management from etilotig reduced its size by more than 50%.

The way etilotig manages authorization is quite simple: when it generates its internal configuration, it actually generates a perl script which is called on each ssh connection attempt. The script then authorises or not the transaction.

We made a similar script which prints the users some information about which repositories they have access to if they just run ssh git@push.par.clever-cloud.com or such, checks if they are authorized when they try to git push/pull or rejects any other request.

It looks like this (with some extra stuff added):

#!/bin/bash

sanity_check() {
    if [[ -z "${SSH_CONNECTION}" ]]; then
        echo "Who the hell are you?" >&2
        exit 1
    fi

if [[ -z "${SSH_ORIGINAL_COMMAND}" ]]; then
        export SSH_ORIGINAL_COMMAND="info"
fi
}

ask_for_info() {
    local userid="${1}"

# make the request to the API to retrieve user info message
echo "some info"
}

ask_for_authorization() {
    local userid="${1}"
    local appid="${2}"

# make the request and return the HTTP status code here. 200 means authorized.
echo 200
}

authorize() {
    local userid="${1}"
    local verb="${2}"
    local appid="${3}"
    local ret=1
    case "${verb}" in
        "git-receive-pack"|"git-upload-pack")
            local code
            code=$(ask_for_authorization "${userid}" "${appid}")
            [[ "${code}" == "200" ]] && ret=0
            ;;
    esac
    return "${ret}"
}

final_abort() {
    echo "What are you trying to achieve here?" >&2
    exit 2
}
main() {
    sanity_check
    local userid="${1}"
    local verb
    local repo
    local repo_dir
    verb=$(echo "${SSH_ORIGINAL_COMMAND}" | cut -d ' ' -f 1)
    repo=$(echo "${SSH_ORIGINAL_COMMAND}" | cut -d ' ' -f 2 | tr -d "'\"")
    [[ "${repo}" == /* ]] && repo=${repo:1}
    repo_dir="${REPOS_DIR}/${repo:0:6}/${repo:6:2}/${repo:8:2}/${repo}"
    if [[ "${verb}" == "info" ]]; then
        ask_for_info "${userid}"
    elif authorize "${userid}" "${verb}" "${repo}"; then
        export CC_USER="${userid}"
        export CC_NOTIFY_SCRIPT="${HOME}/.etilotig/send-push-event"
        exec "${verb}" "${repo_dir}"
    else
        final_abort
    fi
}
. "${HOME}"/.etilotig/.etilotigrc
main "${@}"

With this in place, nearly everything was ready. Two tiny hooks on top of that to only allow users to push on the master branch, and to trigger a deployment on git push:

hooks/update

#!/bin/bash

main() {
    local rev="${1}"
    if [[ "${rev}" == refs/tags/* ]]; then
        exit 0
    fi
    if [[ "${rev}" != "refs/heads/master" ]]; then
        echo "You tried to push to a custom branch."
        echo "Only master is allowed."
        exit 1
    fi
}

main "${@}"

hooks/post-update

#!/bin/bash

sanity_check() {
    local rev="${1}"
    if [[ "${rev}" == refs/tags/* ]]; then
        exit 0
    fi
}

main () {
    local rev="${1}"
    sanity_check "${rev}"
    local repo=$(basename $(pwd))
    local appId=${repo/.git/}
    local commitId=$(git rev-parse "${rev}")
    "${CC_NOTIFY_SCRIPT}" "${appId}" "${commitId}" "${CC_USER}"
    echo "[SUCCESS] The application has successfully been queued for redeploy."
}

main "${@}"

Conclusion

That's it, we have our new git server manager up and running, which works pretty well.

The performance gain? We went from between 3 and more than 5 minutes to less than 1 second per action, while dropping the whole gitolite codebase and reducing the size of etilotig by 50%, with an average performance gain of 30k%.

gitolite has been very useful both in its utilisation and its codebase to better comprehend the whole authentication mechanism, so a great thanks to this awesome tool.

Now gitolite is dead, long live etilotig!