The following article will provide you with some basic examples on how to use AWS
s3cmd command we are going to cover will list all available data ( objects ) under our AWS s3 account. This will list all, buckets, directories and files:
$ s3cmd la
Create a Bucket
Now, it is time to create a new bucket. You can think of a bucket as a top-level directory where you can elegantly store and sort your data. The bucket name must be unique across entire system, otherwise you will receive an error:
ERROR: S3 error: 409 (BucketAlreadyExists). The first bucket we are going to create will be called
$ s3cmd mb s3://linuxconfigorg Bucket 's3://linuxconfigorg/' created
List all available buckets
Now, that we have created our first bucket, it is time to confirm its existence by listing all existing S3 buckets:
$ s3cmd ls 2016-01-18 02:09 s3://linuxconfigorg
Copy file into bucket
s3://linuxconfigorg is available and ready to receive our data. First, we need to create some sample file eg.
foobar and then copy it to our new bucket:
$ echo linuxconfig.org > foobar $ cat foobar linuxconfig.org
Let’s copy this new file into our bucket
$ s3cmd put foobar s3://linuxconfigorg/
Confirm that the file has been copied over:
$ s3cmd ls s3://linuxconfigorg/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar
Copy files between buckets
s3cmd command also allows you to copy files between two S3 buckets. In the following example we will first create a new bucket named
linuxconfigorg2 and then we will copy our
foobar file located in
linuxconfigorg bucket to
$ s3cmd mb s3://linuxconfigorg2 Bucket 's3://linuxconfigorg2/' created $ s3cmd cp s3://linuxconfigorg/foobar s3://linuxconfigorg2/ remote copy: 's3://linuxconfigorg/foobar' -> 's3://linuxconfigorg2/foobar'
Move files between buckets
Same as with the copy command we can also move data between buckets. In the following example we are going to move file
linuxconfigorg2 bucket to
linuxconfigorg and renaming it in the process to
$ s3cmd mv s3://linuxconfigorg2/foobar s3://linuxconfigorg/foobar2 move: 's3://linuxconfigorg2/foobar' -> 's3://linuxconfigorg/foobar2'
As a result of the move process our
linuxconfigorg2 bucket is now empty:
$ s3cmd ls s3://linuxconfigorg2/ $
Synchronize a locale directory
s3cmd command also allows for a file synchronization. Please note that s3cmd’s sync command does not synchronize empty directories. It uses md5sum and filesize to synchronize files. If you directory is empty it will not be created within a destination bucket.
$ mkdir -p dir1/dir2/dir3 $ touch dir1/dir2/file2 $ s3cmd sync dir1/ s3://linuxconfigorg/
In the above example we have created three directories containing only a single file within
dir2 directory. As a result,
s3cmd command omitted empty directories, while only
dir2 has been copied over to our S3 bucket.
Creating a directory/folder
There is no need to create a directory before you upload your data. For example the following linux command will copy a file to s3://linuxconfigorg/ bucket while creating a new directory during the process:
$ s3cmd put foobar s3://linuxconfigorg/new_directory/ $ s3cmd la DIR s3://linuxconfigorg/dir2/ DIR s3://linuxconfigorg/new_directory/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar 2016-01-18 02:33 16 s3://linuxconfigorg/foobar2
The following linux command will remove both
foobar2 files currently located within
$ s3cmd rm s3://linuxconfigorg/foobar2 s3://linuxconfigorg/foobar delete: 's3://linuxconfigorg/foobar2' delete: 's3://linuxconfigorg/foobar'
As you can see
s3cmd command accepts multiple arguments and therefore, we are able to remove multiple files using a single rm command.
Empty Entire Bucket
Next, will show you how to remove all files and directories within a bucket. For example, the following linux command will remove all available files and directories located within
linuxconfigorg S3 bucket:
$ s3cmd del -r --force s3://linuxconfigorg/ delete: 's3://linuxconfigorg/dir2/file2' delete: 's3://linuxconfigorg/new_directory/foobar'
Before you remove S3 bucket the bucket must be empty. Using the previous command we have removed all objects from
linuxconfigorg S3 bucket and thus it can now be removed:
$ s3cmd rb s3://linuxconfigorg/ Bucket 's3://linuxconfigorg/' removed
Compress File Before Upload
The following linux command is more advanced as it uses pipes and STDOUT to first compress file with a maximum gzip compression and stream it directly to S3 bucket:
$ cat foobar | gzip -9 | s3cmd put - s3://linuxconfigorg/foobar.gz '
' -> 's3://linuxconfigorg/foobar.gz' [part 1, 36B] 36 of 36 100% in 1s 32.90 B/s done
Compress Directory Before Upload
Similarly to the above command we can also compress entire directory before it gets uploaded to S3 bucket:
$ tar cz dir1/ | gzip -9 | s3cmd put - s3://linuxconfigorg/dir1.tar.gz '
' -> 's3://linuxconfigorg/dir1.tar.gz' [part 1, 196B] 196 of 196 100% in 0s 336.18 B/s done