Using AWS CLI to manage S3 buckets

Some common AWS S3 CLI commands to manage files on S3 buckets. I will explain how to install the AWS CLI, set up your credentials, sync files, delete, upload and download.

Install CLI

Download AWS CLI from Amazon. Install to your system.

Set up your credentials

Once AWS CLI is installed you will need to configure it with your credentials to be able to work with your S3 buckets. Run the following command.

aws configure

You will need to enter the following:

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

To get these credentials login to AWS Console and go to Security Credentials and create them according to IAM Best Practices. Once completed this will allow AWS CLI to communicate with your S3 buckets.

Managing multiple AWS credentials

If you manage several AWS accounts, its best to setup Named profiles to manage your various AWS CLI credientails. Open your AWS credentials files

~/.aws/credentials (Linux & Mac) or %USERPROFILE%\.aws\credentials (Windows)

Then add the various accounts:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[user1]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

Save and close the file. You will now be able to run AWS CLI commands with specific named profiles. For emxaple:

aws s3 ls --profile user1

See the official docs for Named Profiles

Being safe when running AWS CLI commands

It is recommended to add --dryrun parameter flag before running AWS CLI commands which will display the operations without actually running them. This is good for debugging and being sure of the commands you want to execute.

Bucket wide commands

List AWS S3 buckets

To list AWS S3 buckets run:

aws s3 ls

Creating an S3 bucket

To create an S3 bucket, run:

aws s3 mb s3://bucket-name

Keep in mind that bucket names are unique across AWS.

Successful creation will return make_bucket: bucket-name

Deleting an S3 bucket

To delete an S3 bucket, run:

aws s3 rb s3://bucket-name

Successful creation will return remove_bucket: bucket-name

Download S3 buckets contents

To download S3 bucket contents to local system use the following command

aws s3 sync s3://bucket-name .

The following command will download the contents to the current local folder.

Upload local contents to S3 bucket

To upload local file contents to the S3 bucket run the following command

aws s3 sync . s3://bucket-name

Deleting files and updating S3 bucket

If you delete files locally and want the S3 bucket to also delete files you have to add the --delete parameter flag.

aws s3 sync . s3://bucket-name --delete

I recommend adding the --dryrun flag and running the command to make sure you are deleting exactly what you want.

Deleting files and updating S3 bucket

File commands

Copy one file to bucket

To copy a single file to an S3 bucket use the following command

aws s3 cp filename s3://bucket-name

Delete one file from bucket

To delete a single file from an S3 bucket

aws s3 rm s3://bucket-name/filename

Note: if you delete a file locally, and attempt to sync, the file will not be deleted on the S3 bucket without a --delete flag added to the sync command.

Setting permissions

When uploading files especially for static public sites use the --acl parameter. e.g.

aws s3 cp filename s3://bucket-name --acl public-read

e.g.

aws s3 sync . s3://my-bucket/path --acl public-read

Further commands

See the full documentation for high level CLI commands here Using High-Level (s3) Commands with the AWS CLI. For full documentation for sync on the AWS Docs page

Written by Farhad Agzamov who lives and works in London building things. You can follow him on Twitter and check out his github here