s3 Archives | Clever Cloud

Avec Devoxx_ion, le panier fait les poches de Sudo

Horacio Gonzalez — Fri, 17 May 2024 06:23:45 +0000

Animé par Horacio Gonzalez - @LostInBrittany
avec la participation de :
- David Caussinus - @DCaussinus
- Steven Le Roux - @GwinizDu
- Antoine Blondeau - @P4ndaFR

Épisode enregistré le 5 avril 2024

👋 Venez discuter avec nous sur @clever_cloudFR pour nous dire ce que vous avez pensé de ce nouvel épisode.

➡️ Pour découvrir ou réécouter d’anciens épisodes c’est par ici !

Chapitrage et Liens

00:00:16 : Introduction et présentation des participants

00:01:50 : How an empty S3 bucket can make your AWS bill explode
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
https://twitter.com/QuinnyPig/status/1785311776386727993

00:08:56 : Run0 vs Sudo, drama in systemdland
https://mastodon.social/@pid_eins/112353324518585654
https://linux.developpez.com/actu/357237/Systemd-v256-presente-run0-Une-alternative-plus-sure-a-sudo-une-nouvelle-approche-de-l-elevation-de-privileges-securisee-visant-a-eliminer-progressivement-les-binaires-SUID-traditionnels/
https://opensourcesecurity.io/2024/05/05/episode-427-will-run0-replace-sudo/

00:25:24 : Enregistrements Devoxx: Faire simple, la clé de la durabilité
https://www.youtube.com/watch?v=VPV9tMHaaFE

00:35:02 : Enregistrements Devoxx : Les nouveautés de Java 21
https://www.youtube.com/watch?v=3Erjl1862Q4

00:37:55 : Scala capture checking available in experimental within scala 3.4
https://www.scmatters.io/post/capture-checking-in-scala-3-4

00:46:24 : Toxiproxy
https://github.com/Shopify/toxiproxy

00:53:30 : Antithesis
https://antithesis.com/blog/deterministic_hypervisor/

Setup a directory listing for an S3 bucket

Laurent Doguin — Thu, 08 Oct 2020 15:35:00 +0000

Yesterday I had a customer asking me how to make a simple directory listing like you have with apache server, but with the content stored in an S3 bucket. This way they can replace their git lfs + apache setup with something possibly simpler and more suited to binary storage. As it turns out it's not that complicated. You need to get the permissions right, and to be aware of a listing option in S3 API. Let's see how to make this work on Clever Cloud.

Create a Cellar Addon

Clever Cloud is a Platform as a Service. We provide many managed services and one of them is Cellar, our S3 compatible storage. Assuming you already have an account, here's how to create a Cellar add-on.

Using the CLI

Assuming you are already using clever-tools, all you have to do is run clever addon create cellar-addon --plan S myCellarService.

Using the Web Console

You can login to our console on https://console.clever-cloud.com/. From there click on Create, an add-on and choose Cellar. There is only one plan available since Cellar is billed based on consumption so click next. We have no need to link it to an existing application so click next again, give it a name and click on create.

Setup a Bucket

Now that you have your S3 add-on, the easiest way to manage buckets is probably to use s3cmd or Rclone. Here's how to create a bucket, make it public and store some files. First get your configuration file. It's waiting for you on the add-on dashboard. Browse your Cellar add-on from the console, you should see some details and a link to Download a pre-filled s3cfg file.. Download it and place it under your home directory. If you are a linux user it should be ~/.s3cfg. Now s3cmd will use this configuration. Nothing else to do. Typing s3cmd la list all resources available. If you just created the Cellar add-on, you should see nothing listed. To create your bucket run s3cmd mb s3://your-unique-bucket-name/. The chosen name should be lower case and unique. If I insist on the uniqueness of that name, it's because it can't be already taken by someone else. All buckets are made available by default on http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com. You can see why they have to be unique. You can also name it after your domain name, and put a CNAME redirection to cellar-c2.services.clever-cloud.com.. This way it will be available from your domain name directly. You will find more information on our documenation. Once you have created your bucket, you are ready to upload your files. Run s3cmd sync ./ s3://your-unique-bucket-name/ and the content of the current folder will be automatically updated to your bucket. By default everything is private so if you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com you should get an XML error saying access denied. To change this and make it readable by everyone, run s3cmd setacl --acl-public s3://your-unique-bucket-name/*. This will grant read access to all anonymous users. Each file should be accessible separately but you still cannot list the content of the bucket. This is because the bucket itself has to be set to public. Run s3cmd setacl --acl-public s3://your-unique-bucket-name and it will allow you to see the list of objects as XML. You can get a simpler list by adding type-list=2 as query param. So by browsing http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/?type-list=2, you should see something like:



ldoguinbucket

1000
false

    hello.txt
    2020-10-05T13:07:08.664Z
    "b1946ac92492d2347c6235b4d2611184"
    6
    STANDARD
    
        cellar_0e275449-c02a-4e07-b100-2e357b532e3b
        cellar_0e275449-c02a-4e07-b100-2e357b532e3b@user_51f7098f-604e-4d9e-8fb6-4026fd1767bb
    
    Normal

You can of course adjust the permissions to your liking. Not everything has to be public. You can make that listing look much better and be actually browsable. Here's how.

Add the Listing

As you have seen it's easy to get an XML list of your bucket content. You will find many examples on GitHub of single page HTML that fetch the XML list with an XHR request and then renders it nicely. All you have to do is usually upload that files at the root of your bucket and everything should work. I have tried with https://github.com/qoomon/aws-s3-bucket-browser and it works great. Simply clone this repo and upload all the files at the root of your bucket with the public ACL.

git clone https://github.com/qoomon/aws-s3-bucket-browser/ 
s3cmd put --acl-public aws-s3-bucket-browser/favicon.ico s3://your-unique-bucket-name/favicon.ico
s3cmd put --acl-public aws-s3-bucket-browser/index.html s3://your-unique-bucket-name/index.html
s3cmd put --acl-public aws-s3-bucket-browser/logo.png s3://your-unique-bucket-name/logo.png

Cellar directory listing

From there you should see a very nice listing when you browse http://your-unique-bucket-name.cellar-c2.services.clever-cloud.com/index.html. Thanks to Bengt Brodersen for providing this :)

S3 Storage “Clever Cloud Cellar” Emerges from Beta

Clément Nivolle — Wed, 02 Mar 2016 15:32:00 +0000

Storing files and assets on Clever Cloud just got easier. After a year of beta testing, we’re proud to release Cellar: a S3-compatible file storage API.

Cellar is faster, cheaper and more reliable thanks to its distributed architecture.

If you’re using FS buckets or storing files in your database, switching to Cellar will make your life easier: files can be directly uploaded to and downloaded from Cellar, without ever going through your application. This way you can handle large files without a hassle.

Cellar is perfect for:

user avatars
large files (images, videos)

Pricing

Storage

First 100MB Free
First 1TB € 20.48 / TB / mo
First 25TB € 15.36 / TB / mo
First 50TB € 10.24 / TB / mo

Traffic (outbound)

First 100MB Free
First 10TB € 0.09 / GB / mo
First 40TB € 0.07 / GB / mo

Getting started with Cellar and s3cmd

Create a new Cellar addon in the Clever Cloud console. From the addon dashboard, you will be able to get your access credentials, as well as to download a s3cmd configuration file to manage your files on Cellar.

Create a bucket

In cellar, files are stored in buckets, accessible under bucket-name.cellar.services.clever-cloud.com. To start using cellar, you first have to create a bucket for your files.

You can create a bucket with s3cmd (make sure you have saved the s3cmd configuration file under ~/.s3cfg).

s3cmd mb s3://my-bucket

Upload a file

# private file
s3cmd put my-private-file.jpg s3://my-bucket/my-private-file.jpg

# publicly accessible file
s3cmd put -P my-public-file.jpg s3://my-bucket/my-public-file.jpg

Using Cellar from Node JS

You can use Cellar from any S3-compatible SDK (including the official AWS SDKs) or directly through its REST API.

require fs from 'fs';
require AWS from 'aws-sdk';

// Initialize Cellar Credentials
AWS.config.update({accessKeyId: '', secretAccessKey: ''});
const ep = new AWS.Endpoint('cellar.services.clever-cloud.com');
const s3 = new AWS.S3({ endpoint: ep });


// Upload a file to Cellar
const body = fs.createReadStream('./my-file.jpg');
s3.putObject({Bucket: 'my-bucket', Key: 'my-file.jpg', Body: body}, (err, data) => {
  if (err) console.error(err);
  else console.log("Successfully uploaded data to my-bucket/my-file.jpg");
});

// Get an access URL for a private file
s3.getSignedUrl('getObject', {Bucket: 'my-bucket', Key: 'my-private-file'}, (err, url) => {
  if(err) console.log(err);
  else console.log("The URL is", url);
});

More information

Clever Cloud Cellar is backed by the rock solid Riak Simple Storage (aka Riak S2) from Basho.

Talk about cloud file storage: slides from our CTO Documentation: www.clever.cloud/developers/addons/cellar/

Using Cellar from Node.js with AWS SDK: SDK documentation

Cellar's S3 compatible API: Riak S2 API documentation