Optimizing AWS CloudTrail Costs

AWS CloudTrail gives AWS customers the ability to enable governance, compliance, operational, and risk auditing of their AWS account. CloudTrail records activity from users and API usage across a customer’s AWS services. Without any monitoring or optimization, CloudTrail can incur unnecessary costs. In this article, we will dive into the source of the additional costs and the AWS recommended fix.

Duplicate logging may be responsible for surprise CloudWatch costs

To better understand the source of costs, we need to look at the type of events AWS CloudTrail logs. AWS CloudTrail can log three different types of events, with each type charged separately:

1. Data events
2. Insight events
3. Management events

For this article, we are only considering management events, as they are usually responsible for the bulk of CloudTrail costs. Management events provide visibility into management operations performed on resources, including most API calls to AWS resources.

It’s this kind of request — for business intelligence — that can cause surprises. AWS does not have a way to prevent you from duplicating efforts, but they do charge a hefty fee for duplicate events sent via CloudWatch.

AWS CloudTrail pricing

The source of the unnecessary cost stems from the $2 per 100,000 event price. Two dollars per 100,000 events may sound like a small amount, but that cost can easily balloon with scale and demands from the business. These additional charges can accumulate with surprising results, causing duplicate management events to be responsible for as much as 90% of CloudWatch costs.

Management events and CloudTrail pricing: A closer look

AWS delivers the first copy of management events in the CloudTrail free tier. Additional copies are charged at $2 per 100,000 management events delivered. At $2 per copy, it’s easy to see how CloudTrail pricing can quickly become a big headache, especially after enjoying free tier in production.

Because there is no clear value in configuring two CloudTrail trails to deliver the exact copy of management events to different S3 buckets, we want to disable duplicate CloudTrail trails that deliver the same copy of management events. 

By removing duplicate CloudWatch trails, all CloudTrail Management events will be logged using the free tier and avoid the $2 per 100,000 cost.

Identify and remove duplicate CloudWatch trails

Identify duplicate trails

Now that we know about the two types of duplicate CloudWatch trails that can incur costs, we can use the following methods to identify our duplicate trails:

1. AWS Billing and Cost Management Console → choose Bills.
   1. Go to the Bill details by service tab.
   2. In AWS Services Charges, expand CloudTrail.
   3. Expand the AWS Region to view the event cost record details. Then, review the PaidEventsRecorded metric to identify duplicate event records.
2. AWS Athena query
   1. Create a table in an S3 bucket for logging CloudWatch events
   2. Run a SQL-style query to the Athena table using the PaidEventsRecorded method to search records e.g.,

      SELECT eventName, count(eventName) AS NumberOfChanges, eventSource
      FROM your_athena_tablename
      WHERE eventtime >= '2019-01-01T00:00:00Z'and eventtime < '2019-01-31T00:00:00Z'
      GROUP BY eventName, eventSource
      ORDER BY NumberOfChanges DESC

In order to compare if two trails are duplicates, the following dimensions must be completely identical: 

• IsOrganizationTrail
• IsMultiRegionTrail
• ReadWriteType
• ExcludeManagementEventSource
• DataResources
• HasInsightSelectors
• InsightSelectors
• IncludeGlobalServiceEvents
• HasCustomEventSelectors
• AdvancedEventSelectors

Once you verify the trails are identical, then you have a target list of duplicate trails. However, there is one more scenario that you should check for to maximize optimization: trail supersets.

Identifying trail supersets

To maximize this optimization, we must also check for supersets. For this scenario, we must identify where a CloudWatch superset trail will be tracking all the events of a subset trail. Let’s review the two scenarios below:

1. Scenario 1: A = B
2. Scenario 2: A ⊃ B

#	SCENARIO
1	Trail A & Trail B are identical. They track exactly the same events. A = B
2	Trail A is a superset of Trail B. Trail A tracks all the events that Trail B tracks, plus others. A ⊃ B

As you can see in the second scenario, customers are again charged for duplicate copies of management events. This can happen frequently when an organization has multiple projects that create a report with similar management metrics.

We can identify supersets by comparing the configuration settings for two trails along the following dimensions and criteria:

DIMENSION	CONDITION
IsOrganizationTrail	Both trails have the same configuration, or the superset trail has this set to “True,” while the subset trail has this set to “False.”
IsMultiRegionTrail	Both trails have the same configuration, or the superset trail has this set to “True,” while the subset trail has this set to “False.”
ReadWriteType	Both trails have the same configuration, or the superset trail has an “All” configuration, while the subset trail has a “ReadOnly” or “WriteOnly” configuration.
ExcludeManagementEventSource	Both trails have the same configuration, or the superset trail excludes fewer events than the subset.
DataResources	Both trails have the same configuration, or the superset trail tracks all the data sources of the subset trail.
HasInsightSelectors	Both trails have the same configuration, or the superset trail has this set to “True,” while the subset trail has this set to “False.”
InsightSelectors	Both trails have the same configuration, or the superset trail contains all the types included in the subset trail.
IncludeGlobalServiceEvents	Both trails have the same configuration, or the superset trail has this set to “True,” while the subset trail has this set to “False.”
HasCustomEventSelectors	Both trails have the same configuration, or the superset trail has this set to “False,” while the subset trail has this set to “True.”
AdvancedEventSelectors	Both trails have the same configuration, or the superset trail has an “empty” list, while the subset trail has a non “empty” list.

Which trail should I remove?

Now that we have identified the optimization opportunities, we can safely remove the duplicate trails. The following table summarizes the actions that should be taken to remove duplicate trails:

#	SCENARIO	ACTION
1	Trail A & Trail B are identical. They track exactly the same events. A = B	Disable the most recent trail
2	Trail A is a superset of Trail B. Trail A tracks all the events that Trail B tracks, plus others. A ⊃ B	Disable Trail B

How to remove duplicate CloudWatch trails

To manually remove a trail, you’ll need to use the updateTrail method in the AWS Management Console or in the AWS CLI. Below is a quick summary of how to disable these trails:

1. The AWS Management Console and open the CloudTrail console. From there, select Trails, then navigate to CloudWatch Logs and deselect “Enabled” from the appropriate trail.

2. Use the StopLogging method AWS CLI for any trail.

Create a billing alarm

One last thing to cover is the ability to be proactive and create billing alarms. Your AWS environment is constantly changing, and without a doubt, duplicates will occur again resulting in surprising costs. Consider adding a CloudWatch billing alarm to alert your team before the surprise bills arrive so that your team can take immediate action.