List allThe first
s3cmdcommand we are going to cover will list all available data ( objects ) under our AWS s3 account. This will list all, buckets, directories and files:
$ s3cmd la
Create a BucketNow, it is time to create a new bucket. You can think of a bucket as a top-level directory where you can elegantly store and sort your data. The bucket name must be unique across entire system, otherwise you will receive an error:
ERROR: S3 error: 409 (BucketAlreadyExists). The first bucket we are going to create will be called
$ s3cmd mb s3://linuxconfigorg Bucket 's3://linuxconfigorg/' created
List all available bucketsNow, that we have created our first bucket, it is time to confirm its existence by listing all existing S3 buckets:
$ s3cmd ls 2016-01-18 02:09 s3://linuxconfigorg
Copy file into bucketOur bucket
s3://linuxconfigorgis available and ready to receive our data. First, we need to create some sample file eg.
foobarand then copy it to our new bucket:
$ echo linuxconfig.org > foobar $ cat foobar linuxconfig.orgLet's copy this new file into our bucket
$ s3cmd put foobar s3://linuxconfigorg/Confirm that the file has been copied over:
$ s3cmd ls s3://linuxconfigorg/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar
BLUE SKY STUDIOS are looking for Linux Administrator to maintain and support the Studio's 450+ production Linux workstations, including daily interactions with the Studio’s digital animation artists.
LOCATION: Greenwich, Connecticut, USA
Copy files between bucketsThe
s3cmdcommand also allows you to copy files between two S3 buckets. In the following example we will first create a new bucket named
linuxconfigorg2and then we will copy our
foobarfile located in
$ s3cmd mb s3://linuxconfigorg2 Bucket 's3://linuxconfigorg2/' created $ s3cmd cp s3://linuxconfigorg/foobar s3://linuxconfigorg2/ remote copy: 's3://linuxconfigorg/foobar' -> 's3://linuxconfigorg2/foobar'
Move files between bucketsSame as with the copy command we can also move data between buckets. In the following example we are going to move file
linuxconfigorgand renaming it in the process to
$ s3cmd mv s3://linuxconfigorg2/foobar s3://linuxconfigorg/foobar2 move: 's3://linuxconfigorg2/foobar' -> 's3://linuxconfigorg/foobar2'As a result of the move process our
linuxconfigorg2bucket is now empty:
$ s3cmd ls s3://linuxconfigorg2/ $
Synchronize a locale directory
s3cmdcommand also allows for a file synchronization. Please note that s3cmd's sync command does not synchronize empty directories. It uses md5sum and filesize to synchronize files. If you directory is empty it will not be created within a destination bucket. Example:
$ mkdir -p dir1/dir2/dir3 $ touch dir1/dir2/file2 $ s3cmd sync dir1/ s3://linuxconfigorg/In the above example we have created three directories containing only a single file within
dir2directory. As a result,
s3cmdcommand omitted empty directories, while only
dir2has been copied over to our S3 bucket.
Creating a directory/folderThere is no need to create a directory before you upload your data. For example the following linux command will copy a file to s3://linuxconfigorg/ bucket while creating a new directory during the process:
$ s3cmd put foobar s3://linuxconfigorg/new_directory/ $ s3cmd la DIR s3://linuxconfigorg/dir2/ DIR s3://linuxconfigorg/new_directory/ 2016-01-18 02:21 16 s3://linuxconfigorg/foobar 2016-01-18 02:33 16 s3://linuxconfigorg/foobar2
Removing fileThe following linux command will remove both
foobar2files currently located within
$ s3cmd rm s3://linuxconfigorg/foobar2 s3://linuxconfigorg/foobar delete: 's3://linuxconfigorg/foobar2' delete: 's3://linuxconfigorg/foobar'As you can see
s3cmdcommand accepts multiple arguments and therefore, we are able to remove multiple files using a single rm command.
Empty Entire BucketNext, will show you how to remove all files and directories within a bucket. For example, the following linux command will remove all available files and directories located within
$ s3cmd del -r --force s3://linuxconfigorg/ delete: 's3://linuxconfigorg/dir2/file2' delete: 's3://linuxconfigorg/new_directory/foobar'
Remove BucketBefore you remove S3 bucket the bucket must be empty. Using the previous command we have removed all objects from
linuxconfigorgS3 bucket and thus it can now be removed:
$ s3cmd rb s3://linuxconfigorg/ Bucket 's3://linuxconfigorg/' removed
Compress File Before UploadThe following linux command is more advanced as it uses pipes and STDOUT to first compress file with a maximum gzip compression and stream it directly to S3 bucket:
$ cat foobar | gzip -9 | s3cmd put - s3://linuxconfigorg/foobar.gz '
' -> 's3://linuxconfigorg/foobar.gz' [part 1, 36B] 36 of 36 100% in 1s 32.90 B/s done
Compress Directory Before UploadSimilarly to the above command we can also compress entire directory before it gets uploaded to S3 bucket:
$ tar cz dir1/ | gzip -9 | s3cmd put - s3://linuxconfigorg/dir1.tar.gz '
' -> 's3://linuxconfigorg/dir1.tar.gz' [part 1, 196B] 196 of 196 100% in 0s 336.18 B/s done