The following article will provide you with some basic examples on how to use AWS s3cmd
command:
List all
The first s3cmd
command we are going to cover will list all available data ( objects ) under our AWS s3 account. This will list all, buckets, directories and files:
$ s3cmd la
Create a Bucket
Now, it is time to create a new bucket. You can think of a bucket as a top-level directory where you can elegantly store and sort your data. The bucket name must be unique across entire system, otherwise you will receive an error: ERROR: S3 error: 409 (BucketAlreadyExists)
. The first bucket we are going to create will be called linuxconfigorg
:
$ s3cmd mb s3://linuxconfigorg Bucket 's3://linuxconfigorg/' created
List all available buckets
Now, that we have created our first bucket, it is time to confirm its existence by listing all existing S3 buckets:
$ s3cmd ls 2016-01-18 02:09 s3://linuxconfigorg
Copy file into bucket
Our bucket s3://linuxconfigorg
is available and ready to receive our data. First, we need to create some sample file eg. foobar
and then copy it to our new bucket:
$ echo linuxconfig.org > foobar $ cat foobar linuxconfig.org
Let’s copy this new file into our bucket linuxconfigorg
:
$ s3cmd put foobar s3://linuxconfigorg/
Confirm that the file has been copied over:
$ s3cmd ls s3://linuxconfigorg/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar
Copy files between buckets
The s3cmd
command also allows you to copy files between two S3 buckets. In the following example we will first create a new bucket named linuxconfigorg2
and then we will copy our foobar
file located in linuxconfigorg
bucket to linuxconfigorg2
bucket:
$ s3cmd mb s3://linuxconfigorg2 Bucket 's3://linuxconfigorg2/' created $ s3cmd cp s3://linuxconfigorg/foobar s3://linuxconfigorg2/ remote copy: 's3://linuxconfigorg/foobar' -> 's3://linuxconfigorg2/foobar'
Move files between buckets
Same as with the copy command we can also move data between buckets. In the following example we are going to move file fooobar
from linuxconfigorg2
bucket to linuxconfigorg
and renaming it in the process to foobar2
:
$ s3cmd mv s3://linuxconfigorg2/foobar s3://linuxconfigorg/foobar2 move: 's3://linuxconfigorg2/foobar' -> 's3://linuxconfigorg/foobar2'
As a result of the move process our linuxconfigorg2
bucket is now empty:
$ s3cmd ls s3://linuxconfigorg2/ $
Synchronize a locale directory
s3cmd
command also allows for a file synchronization. Please note that s3cmd’s sync command does not synchronize empty directories. It uses md5sum and filesize to synchronize files. If you directory is empty it will not be created within a destination bucket.
Example:
$ mkdir -p dir1/dir2/dir3 $ touch dir1/dir2/file2 $ s3cmd sync dir1/ s3://linuxconfigorg/
In the above example we have created three directories containing only a single file within dir2
directory. As a result, s3cmd
command omitted empty directories, while only dir2
has been copied over to our S3 bucket.
Creating a directory/folder
There is no need to create a directory before you upload your data. For example the following linux command will copy a file to s3://linuxconfigorg/ bucket while creating a new directory during the process:
$ s3cmd put foobar s3://linuxconfigorg/new_directory/ $ s3cmd la DIR s3://linuxconfigorg/dir2/ DIR s3://linuxconfigorg/new_directory/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar 2016-01-18 02:33 16 s3://linuxconfigorg/foobar2
Removing file
The following linux command will remove both foobar
and foobar2
files currently located within linuxconfigorg
bucket:
$ s3cmd rm s3://linuxconfigorg/foobar2 s3://linuxconfigorg/foobar delete: 's3://linuxconfigorg/foobar2' delete: 's3://linuxconfigorg/foobar'
As you can see s3cmd
command accepts multiple arguments and therefore, we are able to remove multiple files using a single rm command.
Empty Entire Bucket
Next, will show you how to remove all files and directories within a bucket. For example, the following linux command will remove all available files and directories located within linuxconfigorg
S3 bucket:
$ s3cmd del -r --force s3://linuxconfigorg/ delete: 's3://linuxconfigorg/dir2/file2' delete: 's3://linuxconfigorg/new_directory/foobar'
Remove Bucket
Before you remove S3 bucket the bucket must be empty. Using the previous command we have removed all objects from linuxconfigorg
S3 bucket and thus it can now be removed:
$ s3cmd rb s3://linuxconfigorg/ Bucket 's3://linuxconfigorg/' removed
Compress File Before Upload
The following linux command is more advanced as it uses pipes and STDOUT to first compress file with a maximum gzip compression and stream it directly to S3 bucket:
$ cat foobar | gzip -9 | s3cmd put - s3://linuxconfigorg/foobar.gz '' -> 's3://linuxconfigorg/foobar.gz' [part 1, 36B] 36 of 36 100% in 1s 32.90 B/s done
Compress Directory Before Upload
Similarly to the above command we can also compress entire directory before it gets uploaded to S3 bucket:
$ tar cz dir1/ | gzip -9 | s3cmd put - s3://linuxconfigorg/dir1.tar.gz '' -> 's3://linuxconfigorg/dir1.tar.gz' [part 1, 196B] 196 of 196 100% in 0s 336.18 B/s done