Skip to content

S3

S3 is the secure, durable, and scalable object storage infrastructure of AWS. Often used for serving static website content or holding backups or data.

Commands⚑

Bucket management⚑

List buckets⚑

aws s3 ls

Create bucket⚑

aws s3api create-bucket \
    --bucket {{ bucket_name }} \
    --create-bucket-configuration LocationConstraint=us-east-1

Enable versioning⚑

aws s3api put-bucket-versioning --bucket {{ bucket_name }} --versioning-configuration Status=Enabled

Enable encryption⚑

aws s3api put-bucket-encryption --bucket {{ bucket_name }} \
    --server-side-encryption-configuration='{
  "Rules":
  [
    {
      "ApplyServerSideEncryptionByDefault":
      {
        "SSEAlgorithm": "AES256"
      }
    }
  ]
}'

Download bucket⚑

aws s3 cp --recursive s3://{{ bucket_name }} .

Audit the S3 bucket policy⚑

IFS=$(echo -en "\n\b")
for bucket in `aws s3 ls | awk '{ print $3 }'`
  do echo "Bucket $bucket:"
  aws s3api get-bucket-acl --bucket "$bucket"
done

Add cache header to all items in a bucket⚑

  • Log in to AWS Management Console.
  • Go into S3 bucket.
  • Select all files by route.
  • Choose "More" from the menu.
  • Select "Change metadata".
  • In the "Key" field, select "Cache-Control" from the drop down menu max-age=604800Enter (7 days in seconds) for Value.
  • Press "Save" button.

Object management⚑

Remove an object⚑

aws s3 rm s3://{{ bucket_name }}/{{ path_to_file }}

Upload⚑

Upload a local file with the cli⚑
aws s3 cp {{ path_to_file }} s3://{{ bucket_name }}/{{ upload_path }}
Upload a file unauthenticated⚑
curl --request PUT --upload-file test.txt https://{{ bucket_name }}.s3.amazonaws.com/uploads/

Restore an object⚑

First you need to get the version of the object

aws s3api list-object-versions \
    --bucket {{ bucket_name }} \
    --prefix {{ bucket_path_to_file }}

Fetch the VersionId and download the file

aws s3api get-object \
    --bucket {{ bucket_name }} \
    --key {{ bucket_path_to_file }} \
    --version-id {{ versionid }}

Once you have it, overwrite the same object in the same path

aws s3 cp \
    {{ local_path_to_restored_file }} \
    s3://{{ bucket_name }}/{{ upload_path }}

Copy objects between buckets⚑

aws s3 sync s3://SOURCE_BUCKET_NAME s3://NEW_BUCKET_NAME

Troubleshooting⚑

get_environ_proxies() missing 1 required positional argument: 'no_proxy'⚑

sudo pip3 install --upgrade boto3