Getting started with AWS s3cmd command by examples

The following article will provide you with some basic examples on how to use AWS s3cmd command:

List all

The first s3cmd command we are going to cover will list all available data ( objects ) under our AWS s3 account. This will list all, buckets, directories and files:

$ s3cmd la

Create a Bucket

Now, it is time to create a new bucket. You can think of a bucket as a top-level directory where you can elegantly store and sort your data. The bucket name must be unique across entire system, otherwise you will receive an error: ERROR: S3 error: 409 (BucketAlreadyExists). The first bucket we are going to create will be called linuxconfigorg:

$ s3cmd mb s3://linuxconfigorg
Bucket 's3://linuxconfigorg/' created

List all available buckets

Now, that we have created our first bucket, it is time to confirm its existence by listing all existing S3 buckets:

$ s3cmd ls
2016-01-18 02:09  s3://linuxconfigorg

Copy file into bucket

Our bucket s3://linuxconfigorg is available and ready to receive our data. First, we need to create some sample file eg. foobar and then copy it to our new bucket:

$ echo linuxconfig.org > foobar
$ cat foobar 
linuxconfig.org

Let’s copy this new file into our bucket linuxconfigorg:

$ s3cmd put foobar s3://linuxconfigorg/

Confirm that the file has been copied over:

$ s3cmd ls s3://linuxconfigorg/
2016-01-18 02:21        16   s3://linuxconfigorg/foobar


Copy files between buckets

The s3cmd command also allows you to copy files between two S3 buckets. In the following example we will first create a new bucket named linuxconfigorg2 and then we will copy our foobar file located in linuxconfigorg bucket to linuxconfigorg2 bucket:

$ s3cmd mb s3://linuxconfigorg2
Bucket 's3://linuxconfigorg2/' created
$ s3cmd cp s3://linuxconfigorg/foobar s3://linuxconfigorg2/
remote copy: 's3://linuxconfigorg/foobar' -> 's3://linuxconfigorg2/foobar'

Move files between buckets

Same as with the copy command we can also move data between buckets. In the following example we are going to move file fooobar from linuxconfigorg2 bucket to linuxconfigorg and renaming it in the process to foobar2:

$ s3cmd mv s3://linuxconfigorg2/foobar s3://linuxconfigorg/foobar2
move: 's3://linuxconfigorg2/foobar' -> 's3://linuxconfigorg/foobar2'

As a result of the move process our linuxconfigorg2 bucket is now empty:

$ s3cmd ls s3://linuxconfigorg2/
$

Synchronize a locale directory

s3cmd command also allows for a file synchronization. Please note that s3cmd’s sync command does not synchronize empty directories. It uses md5sum and filesize to synchronize files. If you directory is empty it will not be created within a destination bucket.
Example:

$ mkdir -p dir1/dir2/dir3
$ touch dir1/dir2/file2
$ s3cmd sync dir1/ s3://linuxconfigorg/

In the above example we have created three directories containing only a single file within dir2 directory. As a result, s3cmd command omitted empty directories, while only dir2 has been copied over to our S3 bucket.

Creating a directory/folder

There is no need to create a directory before you upload your data. For example the following linux command will copy a file to s3://linuxconfigorg/ bucket while creating a new directory during the process:

$ s3cmd put foobar s3://linuxconfigorg/new_directory/
$ s3cmd la
                      DIR   s3://linuxconfigorg/dir2/
                       DIR   s3://linuxconfigorg/new_directory/
2016-01-18 02:21        16   s3://linuxconfigorg/foobar
2016-01-18 02:33        16   s3://linuxconfigorg/foobar2


Removing file

The following linux command will remove both foobar and foobar2 files currently located within linuxconfigorg bucket:

$ s3cmd rm s3://linuxconfigorg/foobar2 s3://linuxconfigorg/foobar
delete: 's3://linuxconfigorg/foobar2'
delete: 's3://linuxconfigorg/foobar'

As you can see s3cmd command accepts multiple arguments and therefore, we are able to remove multiple files using a single rm command.

Empty Entire Bucket

Next, will show you how to remove all files and directories within a bucket. For example, the following linux command will remove all available files and directories located within linuxconfigorg S3 bucket:

$ s3cmd del -r --force s3://linuxconfigorg/
delete: 's3://linuxconfigorg/dir2/file2'
delete: 's3://linuxconfigorg/new_directory/foobar'

Remove Bucket

Before you remove S3 bucket the bucket must be empty. Using the previous command we have removed all objects from linuxconfigorg S3 bucket and thus it can now be removed:

$ s3cmd rb s3://linuxconfigorg/
Bucket 's3://linuxconfigorg/' removed

Compress File Before Upload

The following linux command is more advanced as it uses pipes and STDOUT to first compress file with a maximum gzip compression and stream it directly to S3 bucket:

$ cat foobar | gzip -9 | s3cmd put - s3://linuxconfigorg/foobar.gz
'' -> 's3://linuxconfigorg/foobar.gz'  [part 1, 36B]
 36 of 36   100% in    1s    32.90 B/s  done

Compress Directory Before Upload

Similarly to the above command we can also compress entire directory before it gets uploaded to S3 bucket:

$ tar cz dir1/ | gzip -9 | s3cmd put - s3://linuxconfigorg/dir1.tar.gz
'' -> 's3://linuxconfigorg/dir1.tar.gz'  [part 1, 196B]
 196 of 196   100% in    0s   336.18 B/s  done


Comments and Discussions
Linux Forum