In this AWS Tutorial, we’ll explain the Steps to Install s3cmd on Linux and Manage AWS S3 Bucket. Nowadays most of the cloud/sysadmin use the s3cmd tool to manage s3 bucket.
S3cmd is an Open Source tool that is free for both commercial and private use, but you need to pay only for Amazon resources. so most of the users are using this tool for managing AWS S3 Buckets. S3cmd is written in Python. S3cmd is a free command-line tool.
S3cmd tool comes with many features and option like:
- more than 60+ command-line options
- encryption
- incremental backup
- s3 sync
- include multipart uploads
- ACL and metadata management
- Checking s3 bucket size
- and more
In this article, we explain simple steps to install s3cmd on Linux and Manage AWS S3 Bucket.
We can also mount s3 buckets as local storage on Linux systems using S3FS with FUSE. Read the article on how to mount Amazon s3 to the Linux system.
Steps to Install s3cmd on Linux and Manage AWS S3 Bucket
s3cmd repositories is by default installed for CentOS, RHEL, and Ubuntu System.
So you can install s3cmd with a simple command as mentioned below.
Command to install s3cmd on CentOS/RHEL:
yum install s3cmd
Command to install s3cmd on Ubuntu/Debian:
sudo apt-get install s3cmd
Command to install s3cmd on SUSE Linux Enterprise Server 11:
zypper addrepo http://s3tools.org/repo/SLE_11/s3tools.repo zypper install s3cmd
Another method to install s3cmd using the source in Linux
For the latest version visit the link https://sourceforge.net/projects/s3tools/files/s3cmd/ or download from the below command.
Step 1: Download s3cmd using source
wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.0.2/s3cmd-2.0.2.tar.gz
Step 2: Untar the download files
tar xzf s3cmd-2.0.1.tar.gz
Step 3: Install the s3cmd tool
cd s3cmd-2.0.1 sudo python setup.py install
At the time of this blog, the latest s3cmd version is s3cmd 2.0.2. These are other steps to install s3cmd on Linux and Manage AWS S3 Bucket.
Steps to configure s3cmd environment to manage Amazon S3 Bucket
We will be needing Amazon credentials like Access Key and Secret Key. You need to login to AWS IAM Console and goto IAM user.
– https://console.aws.amazon.com/iam/home?#security_credential
Once you generate IAM user Access Key and Secret Key then configure these credentials in the Linux system with s3cmd command as shown below:
# s3cmd --configure
Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key: ---enter-your-access-key---- Secret Key: ------enter-your-secret-key---- Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: xxxxxxxxxx Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: Yes New settings: Access Key: xxxxxxxxxxxxxxxxxxxxxx Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Encryption password: xxxxxxxxxx Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine 🙂 Now verifying that encryption works... Success. Encryption and decryption worked fine 🙂 Save settings? [y/N] y Configuration saved to '/root/.s3cfg'
The configuration is completed.
Let’s see how to use the s3cmd command line to manage the Amazon s3 bucket in Linux.
s3cmd command to list s3 bucket
# s3cmd ls
This command will list s3bucket from Amazon S3, If you are unable to get the list of the bucket then there will be the following reasons.
1. There is no bucket created in Amazon s3.
2. As you have used IAM user Access Key and Secret Key while configuring the s3cmd tool, please verify IAM user has required permission to access the s3 bucket.
3. If you are using an IAM role then check whether the s3 bucket policy is attached to the EC2 instance or not.
s3cmd command to create a new bucket on Amazon s3
# s3cmd mb s3://sysadminxpert Bucket 's3://sysadminxpert/' created
s3cmd command to upload file in Amazon s3 bucket
# s3cmd put anyfile.txt s3://sysadminxpert/ anyfile.txt -> s3://sysadminxpert/anyfile.txt [1 of 1] 10316 of 10316 100% in 0s 1268.35 kB/s done
s3cmd command to upload directory in Amazon s3 bucket
# s3cmd put -r backupdir s3://sysadminxpert/ backupdir/samplefile1.txt -> s3://sysadminxpert/backupdir/samplefile1.txt [1 of 2] 8201 of 8201 100% in 0s 15.18 kB/s done backupdir/samplefile2.txt -> s3://sysadminxpert/backupdir/samplefile2.txt [2 of 2] 210 of 210 100% in 0s 0.00 B/s done
s3cmd command to list data from Amazon s3 bucket
# s3cmd ls s3://sysadminxpert/ DIR s3://sysadminxpert/backupdir/ 2019-09-03 03:18 8201 s3://sysadminxpert/backupdir/samplefile1.txt 2019-09-03 03:19 201 s3://sysadminxpert/backupdir/samplefile2.txt
s3cmd command to download a file from Amazon s3 bucket
# s3cmd get s3://sysadminxpert/backupdir/samplefile1.txt s3://sysadminxpert/backupdir/samplefile1.txt -> ./samplefile1.txt [1 of 1] 4 of 4 100% in 0s 110.80 B/s done
s3cmd command to Delete/Remove file from Amazon s3 bucket
Removing file from s3 bucket
# s3cmd del s3://sysadminxpert/backupdir/samplefile1.txt File s3://sysadminxpert/backupdir/samplefile1.txt deleted
Removing directory from s3 bucket
# s3cmd del s3://sysadminxpert/backupdir/ File s3://sysadminxpert/backupdir/ deleted
s3cmd command to remove S3 Bucket
s3cmd rb s3://sysadminxpert ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty
Above s3cmd remove command failed because the s3 bucket contains some file or data which needs to be deleted first then this command will work.
# s3cmd rb s3://sysadminxpert Bucket 's3://sysadminxpert/' removed
This is the end of tutorials, we explained the Steps to Install s3cmd on Linux and Manage AWS S3 Bucket.
Don’t miss to check out how to mount s3bucket to Linux.
Thanks for reading this blog, you’ll also like the below articles.
AWS Backup Strategies at rest and in transit
Amazon S3 and Glacier Server-Side Security
Traceroute command to count hop in Linux
Steps To Install and Configure Prometheus On a Linux Server
