How and when to use AWS S3 Intelligent-Tiering to automatically decrease your AWS S3 costs

What is AWS Intelligent-Tiering?

Managing your storage costs with AWS is an important part of architecting your AWS infrastructure. AWS offers several storage classes for your data and each offers a unique balance between cost and accessibility. For example, using Amazon S3 Glacier Storage Class for items that are needed less frequently helps reduce cost. However, what happens when you need to access those items suddenly and quickly? Are you prepared to pay the extra expense to retrieve it rapidly?

Choosing the right S3 storage tier can be challenging. Even with careful analysis it is difficult to predict access patterns, and even more difficult to manage your many S3 storage classes accordingly. AWS S3 Intelligent-Tiering is a fully automated service that solves this problem by monitoring data access and moving data to less expensive storage tiers if it has not been accessed for 30 or 90 days. If the object is accessed, it is moved back to the Frequent Access tier.

AWS S3 Intelligent-Tiering:

• Is completely automated.
• Monitors data access and automatically moves objects to cheaper storage tiers, if they have not been recently accessed.
• Provides instant retrieval – there is no performance impact or service degradation.
• Can be enabled on all buckets (pricing updates released in September 2021 eliminated the minimum storage duration and monitoring costs for small objects.)
• Works easily with existing lifecycle policies. When there are multiple policies for a bucket, AWS S3 Intelligent-Tiering will optimize for costs. 
• Requires no changes to the application. Data in the S3 is accessed as usual.
• Can provide between 18% and 37% cost savings!

Here are a few more things to note about AWS S3 Intelligent-Tiering. If there are multiple lifecycle policies AWS S3 will automatically optimize for cost. S3 Intelligent-Tiering also ignores small objects (< 128 KB); these will always stay in the Frequent Access tier. This removes the problem for buckets where otherwise the per-object monitoring would outweigh potential cost savings.

When should I use AWS Intelligent-Tiering?

How will you know when AWS S3 Intelligent-Tiering is right for you? Intelligent-Tiering can be used if your application uses AWS S3, it’s as simple as that. It can be safely enabled for all buckets, and AWS S3 Intelligent-Tiering offers the same performance as Standard S3 storage classes.

AWS S3 Intelligent-Tiering’s highly competitive pricing model

Pricing and Cost Comparison for Storage Tiers

Storage	Storage Tier
$0.023 per GB / month – First 50 TB$0.022 per GB / month – Next 450 TB$0.021 per GB / month – Over 500 TB	S3 Standard Frequent Access Tier
$0.0125 per GB / month – After 30 days	Infrequent Access Tier
$0.004 per GB / month – After 90 days	Archive Instant Access Tier
Monitoring and Automation	Service
$0.0025 per 1,000 objects (> 128KB)	S3 – Intelligent-Tiering

Cost Comparison: Storage Tiers

Making sense of it all

S3 Infrequent Access and Glacier Instant Retrieval are two storage classes that have noticeably high retrieval costs.   These costs exceed Standard S3 storage with as little as three retrievals per month, rendering these tiers suitable only for special use cases with S3 Infrequent Access. Now consider how many combinations of storage classes and access patterns your infrastructure is using across the board. You can see why keeping all of this information straight for even one storage class can be overwhelming. 

Thankfully AWS made a simple solution. AWS S3 Intelligent-Tiering has no retrieval costs, making it a safe and a cost-competitive choice for all buckets. If you have data which is accessed frequently, it behaves as a Standard S3 storage class. Otherwise, the price drops slowly to equal Glacier Instant Retrieval, without the risk of high costs if there are instances of unexpected access. 

There are other storage tiers that offer even lower monthly fees, but they do not offer instant retrieval. Retrieval times in minutes or hours can result in service degradation or failures, unless adjustments are made to how an application accesses S3.

How to Enable AWS S3 Intelligent-Tiering:

1. Copy the following script into a text file and save it as “enable-intelligent-tiering.sh”:

Note: The script is in dry-run mode, see the comments in row 15-17.

#!/bin/bash

# this will list all buckets in the account
# adapt this script to filter the list according to your needs
for bucket in $(aws s3api list-buckets --query "Buckets[].Name"); do
:
# exclude opening and closing brackets
if [ $bucket = "[" ] || [ $bucket = "]" ]; then
  continue
fi

# remove double quotes and commas
bucket=$(echo "$bucket" | sed 's/"//g' | sed 's/,//g')

# Dry run
# Uncomment the following line to upload the lifecycle
# $(aws s3api put-bucket-lifecycle-configuration --bucket $bucket --lifecycle-configuration file://policy.json)
echo "Updating lifecycle configuration for bucket $bucket"

done

2. Copy the following script and save it as “policy.json” in the same folder:

"Rules": [
    {
        "ID": "enable_intelligent_tiering",
        "Filter": {},
        "Status": "Enabled",
        "Transitions": [
            {
                "Days": 1,
                "StorageClass": "INTELLIGENT_TIERING"
            }
        ],
        "NoncurrentVersionTransitions": [
            {
                "NoncurrentDays": 1,
                "StorageClass": "INTELLIGENT_TIERING"
            }
        ]
    }
]

3. On Unix-based systems, execute the following command:

bash enable-intelligent-tiering.sh

Windows users can run the command using tools such as cygwin or GIT Bash.

Note: You must either be the owner of the buckets or have the s3:PutLifecycleConfiguration access right to make these changes.

Run this command from time to time to ensure AWS S3 Intelligent Tiering is enabled for new buckets.

For more ways to optimize your AWS services and environments, check out CloudFix.