The elasticity in modern cloud computing and the wide variety in resource options necessitate the use of a monitoring tool for resources and applications.
AWS CloudWatch addresses this need by providing a repository of metrics and custom logs for each AWS region, along with basic alarm functionality and the capability to assess and optimize resource utilization to reduce costs.
Since CloudWatch can provide this sort of insight into resources, the features can be leveraged in multiple ways to enhance cost optimization. These include:
- Store AWS CloudWatch data in your FinOps data lake to conduct advanced queries and correlate usage and costs
- Set up key usage alarms with AWS CloudWatch and more granular alarms with AWS QuickSight
- Use CloudWatch logs for application log analytics
Store AWS CloudWatch data in FinOps data lakes
CloudWatch was created with a UI that provides all sorts of actionable data and insights allowing proper monitoring of applications. However, the introduction of AWS CloudWatch Metrics Insights in 2021 took metric monitoring to another level.
With AWS CloudWatch Metrics Insights, it’s now possible to use SQL-like querying functionality to browse through the relevant information.
Limitations on AWS CloudWatch Metrics Insights
Although the Metrics Insights can help meet special requirements, the feature is constrained with the following limitations:
- It can only query the last 3 hours of data, limiting its ability to identify the long-term patterns required for AWS architecture optimization.
- The maximum number of metrics that can be queried per process is 10,000 with a limit on time series being less than 500.
- It cannot query linked accounts for AWS Organizations, making it unsuitable for enterprise architecture reviews.
How to overcome AWS CloudWatch Metrics Insights limitations
The limitations affecting AWS CloudWatch Metrics Insights can be worked around by data laking. The process includes the restoration of all metrics of interest in a single S3 data lake for all AWS Organization member accounts. Then, by joining the data set with Cost and Usage Reports (CUR) data sets from AWS Athena or AWS QuickSight, you can associate usage and costs together and answer important questions that are critical for selecting the correct AWS architecture and not paying for unneeded services.
Specifically, sample useful queries include:
- Queries for monthly idle costs
- Example: EC2 Instance – % CPU not used * the instance hourly charge * a number of days per month the instance is up
- Queries to identify ways to complement the AWS Compute Optimizer
- Examples:
- EC2/RDS instances with low consistent utilization for a downscale review, or enabling Instance Scheduler
- Note: An under-utilized instance is usually defined as one having average CPU utilization less than 30% and having memory utilization less than 30% over the last 7 days.
- EC2/RDS instances with no utilization to shut down or to terminate
- EC2/RDS instances with too high utilization for an upscale review
- EC2/RDS instances with low consistent utilization for a downscale review, or enabling Instance Scheduler
- Examples:
- EBS queries for:
- Unattached EBS volumes
- EBS volumes over or under-provisioned for throughput
- EBS volumes over or under-provisioned for size
See additional examples here.
Set up key usage alarms with AWS CloudWatch and AWS QuickSight
Monitoring the usage and threshold insights throughout any cloud environment is essential to the cost optimization process. It ensures that no limits are surpassed and resource management is always under control. AWS CloudWatch and AWS QuickSight provide a gate to set up triggers for demand and usage metrics which have proven successful in eliminating the need for expensive third-party tools.
How AWS CloudWatch is priced
Using AWS CloudWatch alarms, however, can be expensive. With prices reaching up to $0.50 per alarm with only 10 alarms offered as part of the free tier, the alarm feature may only be suitable for mission-critical applications and showstoppers:
| Standard Resolution (60 sec) | $0.10 per alarm metric |
| High Resolution (10 sec) | $0.30 per alarm metric |
| Composite | $0.50 per alarm |
More information on AWS CloudWatch’s features pricing here.
Since large organizations may need to use hundreds to thousands of alarms for thorough coverage across their cloud infrastructure, AWS CloudWatch wouldn’t be an efficient option.
Why AWS QuickSight
Instead, for architecture-wide alarms, we recommend setting up AWS QuickSight alarms. This is a much cheaper option with the same functionality. It can provide alerts covering all types of changes in data and even thresholds which can be integrated with email notification systems. These insights can be manageable and viewable over QuickSight-supported web browsers.
Differences between CloudWatch and QuickSight
The key differences between AWS CloudWatch alarms and AWS QuickSight alarms based upon a FinOps data lake are:
- In the SQL query for the AWS QuickSight alarm, you do not need to refer to any particular resource ID.
- The same AWS QuickSight alarm can cover all resources in an AWS organization, but this would require multiple AWS CloudWatch alarms.
- The QuickSight pricing model, which is based on monthly fixed fees, makes it cost-efficient compared to AWS CloudWatch alarms, with alarms being free as part of the AWS QuickSight dashboard running on top of a FinOps data lake.
AWS CloudWatch logs for application log analytics
CloudWatch also provides a user-friendly UI for custom application log analysis eliminating any need to use special cloud monitoring services. The central storing approach of the metrics and logs allows for simple querying without the need to provision any additional compute or storage resources.
Other services already stream logs to AWS CloudWatch
Depending on CloudWatch for log analysis becomes feasible when you think of the dashboard as a centralized place for all logs. Multiple AWS services, including Amazon CloudTrail, stream their logs into AWS CloudWatch, AWS S3, or Kinesis Data Firehose. A list of all the services streaming logs into CloudWatch can be found here with a detailed guide.
General guidelines to reach an optimal logging strategy
- Specify a retention period: For most logs, there is no need to keep more than a few months of data.
- The right data: Pick the proper composition of the logging granularity, for example, logging actionable information only.
- The proper format: This is essential to ensure easy querying.
- The proper tool: Choose the proper tool that fits the required syntax for queries. For example, CloudWatch is optimal in terms of cost-effectiveness when it comes to simple queries while AWS OpenSearch is recommended for advanced queries. Besides being well integrated with CloudWatch, logs can be streamed between the two services.
As you can see, there are a number of ways to optimize your AWS costs with CloudWatch. For automated AWS cost optimizations, check out CloudFix.