cellar Archives | Clever Cloud

Migration Required for Buckets on Cellar C1 Cluster

Arnaud Lefebvre — Wed, 19 Jan 2022 09:41:48 +0000

tl;dr: On March 21th 2022 the Cellar C1 cluster will reach its end of life. So we ask you to perform a migration to a newer cluster Cellar, which will also improve performance. The C1 cluster did not accept new add-ons since the 20th January 2019.

In fact, data need to be migrated from this old add-on to a new Cellar add-on. Newly created Cellar C2 add-ons come with various improvements :

A better S3 protocol support (including the AWS V4 signature)
Improved upload and download performances (x3 for upload, x2 for download)
Improved resilience and availability of your data (Data is now replicated across two datacenters more than 20km apart)

How to migrate my data?

We can help you to migrate your data by syncing it directly on our side. To do that, reach the support and include in your message source and destination add-ons, including the buckets to migrate. The support team is here to help.

If you wish to migrate on your side, or at least keep your buckets synchronized, we built a simple tool to help you: Cellar migration tool.
You can run this tool as many times as you need. It will only synchronize objects that are different between the source and the destination buckets.
If you do not feel comfortable using it, you can also use the s3cmd sync command from the s3cmd tool.

If your application handle it, we encourage you to upload new objects as soon as possible to the new bucket. Then when you need to query that object, your application can query both buckets (old and new) to get it. You can also store the complete URL to make this easier. This should help you transition from one bucket to another without downtime.

Please note that we can't make sure that the buckets names you use will be available on the destination cluster. If you have a name conflict, please contact us and we will work something out.

A few days before the deadline, we will have short service brown-outs to remind customers that the service is going to be shut down. Those brown-outs will be announced on our status page a few days before. Make sure to subscribe to Status Updates to receive a notification.

What will happen if I can't do the migration?

If you can't do the migration or have any questions about it, we strongly recommend that you contact our support team via the console.
Before shutting down the cluster, we will backup your data and keep it for 6 months. Our support team will be able to provide you a download link or re-inject the data in a new bucket if you wish to recover it.

Setup a directory listing for an S3 bucket

Laurent Doguin — Thu, 08 Oct 2020 15:35:00 +0000

Yesterday I had a customer asking me how to make a simple directory listing like you have with apache server, but with the content stored in an S3 bucket. This way they can replace their git lfs + apache setup with something possibly simpler and more suited to binary storage. As it turns out it's not that complicated. You need to get the permissions right, and to be aware of a listing option in S3 API. Let's see how to make this work on Clever Cloud.

Create a Cellar Addon

Clever Cloud is a Platform as a Service. We provide many managed services and one of them is Cellar, our S3 compatible storage. Assuming you already have an account, here's how to create a Cellar add-on.

Using the CLI

Assuming you are already using clever-tools, all you have to do is run clever addon create cellar-addon --plan S myCellarService.

Using the Web Console

You can login to our console on https://console.clever-cloud.com/. From there click on Create, an add-on and choose Cellar. There is only one plan available since Cellar is billed based on consumption so click next. We have no need to link it to an existing application so click next again, give it a name and click on create.

Setup a Bucket

Now that you have your S3 add-on, the easiest way to manage buckets is probably to use s3cmd or Rclone. Here's how to create a bucket, make it public and store some files. First get your configuration file. It's waiting for you on the add-on dashboard. Browse your Cellar add-on from the console, you should see some details and a link to Download a pre-filled s3cfg file.. Download it and place it under your home directory. If you are a linux user it should be ~/.s3cfg. Now s3cmd will use this configuration. Nothing else to do. Typing s3cmd la list all resources available. If you just created the Cellar add-on, you should see nothing listed. To create your bucket run s3cmd mb s3://your-unique-bucket-name/. The chosen name should be lower case and unique. If I insist on the uniqueness of that name, it's because it can't be already taken by someone else. All buckets are made available by default on http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com. You can see why they have to be unique. You can also name it after your domain name, and put a CNAME redirection to cellar-c2.services.clever-cloud.com.. This way it will be available from your domain name directly. You will find more information on our documenation. Once you have created your bucket, you are ready to upload your files. Run s3cmd sync ./ s3://your-unique-bucket-name/ and the content of the current folder will be automatically updated to your bucket. By default everything is private so if you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com you should get an XML error saying access denied. To change this and make it readable by everyone, run s3cmd setacl --acl-public s3://your-unique-bucket-name/*. This will grant read access to all anonymous users. Each file should be accessible separately but you still cannot list the content of the bucket. This is because the bucket itself has to be set to public. Run s3cmd setacl --acl-public s3://your-unique-bucket-name and it will allow you to see the list of objects as XML. You can get a simpler list by adding type-list=2 as query param. So by browsing http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/?type-list=2, you should see something like:



ldoguinbucket

1000
false

    hello.txt
    2020-10-05T13:07:08.664Z
    "b1946ac92492d2347c6235b4d2611184"
    6
    STANDARD
    
        cellar_0e275449-c02a-4e07-b100-2e357b532e3b
        cellar_0e275449-c02a-4e07-b100-2e357b532e3b@user_51f7098f-604e-4d9e-8fb6-4026fd1767bb
    
    Normal

You can of course adjust the permissions to your liking. Not everything has to be public. You can make that listing look much better and be actually browsable. Here's how.

Add the Listing

As you have seen it's easy to get an XML list of your bucket content. You will find many examples on GitHub of single page HTML that fetch the XML list with an XHR request and then renders it nicely. All you have to do is usually upload that files at the root of your bucket and everything should work. I have tried with https://github.com/qoomon/aws-s3-bucket-browser and it works great. Simply clone this repo and upload all the files at the root of your bucket with the public ACL.

git clone https://github.com/qoomon/aws-s3-bucket-browser/ 
s3cmd put --acl-public aws-s3-bucket-browser/favicon.ico s3://your-unique-bucket-name/favicon.ico
s3cmd put --acl-public aws-s3-bucket-browser/index.html s3://your-unique-bucket-name/index.html
s3cmd put --acl-public aws-s3-bucket-browser/logo.png s3://your-unique-bucket-name/logo.png

Cellar directory listing

From there you should see a very nice listing when you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/index.html. Thanks to Bengt Brodersen for providing this :)

How to Deploy a Static Site on Cellar

Laurent Doguin — Wed, 24 Jun 2020 12:00:00 +0000

You don't need an application to deploy a static site on Clever Cloud anymore. Let's see how :) For years the only way to deploy a static website on Clever Cloud was to create an application. While the type of this application says static, it's in fact a PHP application in disguise. The good thing about this is it allows you to play with Apache's configuration to manage custom redirection and the likes. But sometimes you don't need any flexibility. And you don't want to pay for an application running when you don't need one. Enter Cellar, our S3 compatible object storage. You can use it to serve static assets. By default it uses the Cellar cluster URL, but with the right CNAME redirection and naming convention, it will serve your website behind your own custom domain. Cellar prices are available in our documentation. Here's how it works.

Domain Redirection

To redirect your domain name to Clever Cloud's server, go to your domain name registrar dashboard and select the domain you want to redirect. In my case it's ldoguin.name on OVHCloud.

You need to add a CNAME redirection to our Cellar's CNAME: cellar-c2.services.clever-cloud.com. CNAME redirection are not available for root domains. If you want to redirect a root domain you will need all the public IPs of our cluster. You can easily get them by typing dig cellar-c2.services.clever-cloud.com in a terminal.

Create the Bucket

Once you have redirected your domain name to the Cellar cluster, create the Cellar add-on and add a new bucket named after your domain name, in my case blog.ldoguin.name. An auto-renewed Let's Encrypt certificate will be setup automatically shortly after the bucket creation. Everything you will upload to this bucket will be served as a static file.

Upload your Site

If you already have your files ready, my favorite solution to upload them is to use rclone. It's great to sync your files to many different cloud storage from Cellar to S3, Google drive, Swift, Dropbox and more. Assuming all the files are in a folder called public and that you have a configured connection to Cellar called mys3, this is what you need to type: rclone sync ./public mys3:blog.ldoguin.name --progress --s3-acl=public-read This is pretty much rsync but for cloud storage :) To configure my connection I had the following environment variables:

export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=$CELLAR_ADDON_KEY_ID
export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=$CELLAR_ADDON_KEY_SECRET
export RCLONE_CONFIG_MYS3_ENDPOINT=$CELLAR_ADDON_HOST
export RCLONE_CONFIG_MYS3_TYPE="s3"

I copied them from the my Cellar add-on dashboard. We could stop here. You have a static website hosted on Clever Cloud without a running application. Let's take this a bit further by using an application to build the website.

Build and Upload your Static Site

I wrote about Hugo in a previous post. This is going to be almost similar so I'll be quick. The plan is to create a Clever Cloud application. Each time you will push new content to this application, it will start, generate the Hugo website, upload it to Cellar then stop itself. Assuming you have installed Hugo and clever-tools, this is everything you need to do to have a generated website:

mkdir hugo-cellar-static # Create the site root folder
cd hugo-cellar-static/
hugo new site . # Generate a new Hugo site
git init # initialize the git repository
git submodule add https://github.com/jakewies/hugo-theme-codex themes/codex # add a theme
cp themes/codex/exampleSite/config.toml config.toml # copy the default configuration
cp -r themes/codex/exampleSite/content/ . # copy default content for the exemple
gedit config.toml  # edit the configuration, like the theme or the baseUrl
clever create --type static-apache myapp # Create a new clever cloud application
clever service link-addon MyCellar # link the app to the Cellar add-on 
cat ~/.config/clever-cloud  # Get the clever-tools token

At that point you should have created a website structure and a Clever Cloud application. These are the environment variables needed to configure it:

CC_PHP_VERSION="7"
CC_PRE_RUN_HOOK="./hugo.sh" # A prerun hook to execute the build and sync it to Cellar
CC_RUN_SUCCEEDED_HOOK="clever stop" # use clever-tools to turn off the app when finished 
CLEVER_SECRET="xxxx" # Tokens for clever-tools
CLEVER_TOKEN="xxxxx" # Tokens for clever-tools
DL_RCLONE_VERSION="1.52.1" # which version of rclone to use
HUGO_VERSION="0.62.0" # which version of hugo to use
MY_DOMAIN="blog.ldoguin.name/" # The name of the bucket to sync
PORT="8080"
HUGO_ENV="production"

The clever stop trick is likely to be replaced by something cleaner in the coming weeks, with something like a task or job. The build and sync of the site is done when the pre run hook executes hugo.sh, which has the following content:

# Configure rclone Cellar connection
export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=$CELLAR_ADDON_KEY_ID
export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=$CELLAR_ADDON_KEY_SECRET
export RCLONE_CONFIG_MYS3_ENDPOINT=$CELLAR_ADDON_HOST
export RCLONE_CONFIG_MYS3_TYPE="s3"

# Download and unpack rclone and hugo
wget https://downloads.rclone.org/v$DL_RCLONE_VERSION/rclone-v$DL_RCLONE_VERSION-linux-amd64.zip
wget https://github.com/gohugoio/hugo/releases/download/v$HUGO_VERSION/hugo_extended_"$HUGO_VERSION"_Linux-64bit.tar.gz
unzip rclone-v$DL_RCLONE_VERSION-linux-amd64.zip
tar xvf hugo_extended_"$HUGO_VERSION"_Linux-64bit.tar.gz

# Execute the site generation with Hugo
chmod +x ./hugo
./hugo --gc --minify

# Sync the site to Cellar
./rclone-v$DL_RCLONE_VERSION-linux-amd64/rclone sync ./public mys3:$MY_DOMAIN --progress --s3-acl=public-read

Now what's left to do is add and commit all the files and deploy.

git add .  # add the current folder to git
git commit -m"init" # commit
clever deploy # Deploy your website

The application will start, generate the website and sync it to Cellar then stop itself. So there it is you will still have your website running, but without running an application. Please let us know what you think and what kind of features you would like to see supported for static sites :)

Cellar a S3-like service for Rails’ Active Storage

Valeriane Venance — Thu, 28 May 2020 14:20:00 +0000

Introduced with the release of Rails 5.2 , Active Storage replaces previous solutions for storing user videos, images, PDF, etc..

In the documentation only three implementations are demonstrated and they all rely on cloud services. Today I am going to show you how to use it to be Clever Cloud compatible.

To follow this tutorial you must already have a Rails >= 5.2 application running on Clever Cloud with a PostgreSQL database. You can read this if you need a basic setup.

Tell Rails you want to use Active Storage

In your console, at the root of your rails project repository, type in $ rails active_storage:install. This will create two migration files under ./db/migrate. These migrations will create two new tables: active_storage_blobs and active_storage_attachments, which is everything we need to start storing user attachments without adding new columns to our already existent models. Let's run them with $ rake db:migrate to have a local database up to date. You can now use has_one_attached : on any of your models to add an attachment on it.

Setting up the application for Clever Cloud

As we do provide AWS S3 like storage at Clever Cloud, we will use AWS gem and define custom endpoints. Add gem "aws-sdk-s3", require: false to your Gemfile. Run $ bundle install. Then in config/environments/production.yml add config.active_storage.service = :clevercloud. Now in config/storage.yml add the following lines:

clevercloud:
  service: S3
  access_key_id: <%= ENV['CELLAR_ADDON_KEY_ID'] %>
  secret_access_key: <%= ENV['CELLAR_ADDON_KEY_SECRET'] %>
  region: us-east-1
  bucket: <%= ENV['CELLAR_ADDON_BUCKET_NAME'] %>
  endpoint: <%= ENV['CELLAR_ADDON_ENDPOINT'] %>

I must make clear here that we do not use regions at Clever Cloud, but the aws-sdk-s3 gem needs it; otherwise it throws an AWS::Errors::MissingRegionError error. And we do not want a 500 in production. It is the usage of endpoint that will give us the right endpoint generated by the gem.

Creating our storage on Clever Cloud

Under the organization where your Rails application is deployed, select Create, an add-on, Cellar S3 storage. When prompted Select the applications that will use this add-on: select your Rails application. Name it and let it be deployed. Once this is done, in your addon dashboard, create a bucket. Keep its name. Now, under Environment variables in your Rails application, add the following:

CELLAR_ADDON_BUCKET_NAME=
CELLAR_ADDON_ENDPOINT=https://cellar-c2.services.clever-cloud.com

You can now commit the local files we have changed and push them to your Clever Cloud remote to enjoy Active Storage in production.

Do not forget to edit your safe params in your controllers, forms and permissions.

Happy storing!