CHAN-BLOG.DEV
Back to blog
AWS8 min read2025년 12월 14일

AWS CloudWatch basics

Course notes on CloudWatch metrics, alarms, logs, and the querying tools around them, taken while studying for DOP-C02.

Annyeong!

This post is a summary of the course – AWS Certified DevOps Engineer Professional 2026 - DOP-C02

https://www.udemy.com/share/101WpU3@e1p3aMEiqftLR1EWCFu5C0VftyOyal–IWWAwvmG-VvpKpCJ-knnVg_rGkEMQw1c1Q==/

CloudWatch Metrics

Metric is a variable to monitor (CPUUtilization, NetworkIn, … )

  • Metric belong to namespaces
  • Dimension is an attribute of a metric, so it’s simillar to tags; up to 30 dimensions per metric are accepted
  • Metrics have timestamps
  • Can also create custom metrics

CloudWatch Metric Streams

Provides near-real-time delivery and low latency

  • Metrics → Amazon Kinesis Data Firehose → Destinations
  • Can send data to 3rd party service provider
  • Can send them to S3, Redshift, OpenSearch, …
  • Option to filter metrics to only stream a subset of them

CloudWatch Custom Metrics

  • Example: RAM usage, disk usage, …
  • Use API call PutMetricData
  • Can specify metric resolution (StorageResolution API parameter - two possible value):
    • Standard: 1 minute
    • High resolution: 1 / 5 / 10 / 30 seconds (high cost)

? Accepts metric data points two weeks in the past and two hours in the future (make sure to configure your EC2 instance time correctly)

CloudWatch Anomaly Detection

  • Continuously analyze metrics to determine normal baselines and surface anomalies using ML algorithms
  • Allows you to create Alarms based on metric’s expected value (instead of static threshold)
  • Ability to exclude specified time periods or events from being trained

CloudWatch Logs

  • Log groups: arbitrary name, usually representing an application
  • Log stream: instances within application / log files / containers
  • Can define log expiration policies (never expire / 1 day to 10 years)
  • CloudWatch Logs can send logs to:
    • S3 (export them into batch), Kinesis Data Streams, Kinesis Data Firehose, Lambda, OpenSearch
  • Logs are encrypted by default
  • Can setup KMS-based encryption with your own keys

CloudWatch Logs - Sources

  • SDK, CloudWatch Log Agent, CloudWatch Unified Agent
    • Since the unified agent can send logs, using CloudWatch Log Agent is kinda deprecated
  • Elastic Beanstalk: collection of logs from application
  • ECS: collection from containers
  • AWS Lambda: collection from function logs
  • VPC Flow Logs: VPC specific logs
  • API Gateway
  • CloudTrail based on filter
  • Route53: Log DNS queries

CloudWatch Logs Insights

  • Search and analyze log data stored in CloudWatch Logs
  • Provides a purpose-built query language
    • Automatically discovers fields from AWS services and JSON log events
  • Can query mutiple Log Groups in different AWS accounts
  • It’s a qurey engine, not a real-time engine

CloudWatch Logs - S3 Export

  • Log data can take up to 12 hours to become available for export
  • The API call is CreateExportTask
  • Not near-real time or real-time

CloudWatch Logs Subscriptions

  • Get a real-time log events from CloudWatch Logs
  • Send to Kinesis Data Streams, Kinesis Data Firehose, or Lambda
  • Subscription Filter — filter which logs are events delivered to your destination

CloudWatch Logs → Subscription Filter → …

CloudWatch Logs Aggregation — Multi-Account & Multi Region

Logs from different sources → different subscription filters → Kinesis Data Streams → Kinesis Data Firehose —(near-real time)→ S3

  • Cross-Account Subscription — send log events to resources in a different AWS account (KDS, KDF)
    • Subscription Filter and Subscription Destination
    • Then you need a Destination Access Policy and IAM Role to access the destination (Kinesis Data Streams, allow PutRecord)
    • So the Subscription Filter assumes the IAM Role

CloudWatch Logs Metric Filter

? Filters do not retroactively filter data, Filters only publish the metric data points for events that happen after the filter was created.

  • Ability to specify up to 3 dimensions for the metric filter (optional)

CloudWatch Logs Agent → CW logs → Metric Filters → CW alarm

All kind of logs

  • Application logs
    • Logs generated from the application
  • Operating System Logs (event logs, system logs)
    • Logs that are generated by your operating system (host)
    • Informing you of system behavior (ex: /var/log/messages or /var/log/auth.log)
  • Access Logs
    • List of all the requests for individual files that people have requested from a website
    • Example for httpd: /var/log/apache/access.log
    • Usually for load balancers, proxies, web servers, etc
    • AWS provides some access logs
  • AWS Managed Logs
    • Load Balancer Access Logs (ALB, NLB, CLB) → S3
      • Access logs
    • CloudTrail Logs → S3 / CloudWatch Logs
      • Logs for API calls made within your account
    • VPC Flow Logs → S3 / CloudWatch Logs
      • Information about IP traffic going to and from ENI in your VPC
    • Route 53 Access Logs → CloudWatch Logs
      • Log information about the queries that Route 53 receives
    • S3 Access Logs → S3
      • Server access logging provides detailed records for the requests that are made to a bucket
    • CloudFront Access Logs → S3
      • Detailed information about every user request that CloudFront receives

CloudWatch Logs for EC2

  • By default, no logs from EC2 will go to CloudWatch
  • You need to run a CloudWatch agent on EC2 to push the logs files you want
  • Make sure IAM permissions are correct
  • The CloudWatch log agent can be setup on-premise too

CloudWatch Logs Agent (legacy) & Unified Agent

  • CloudWatch Logs Agent
    • Old version of the agent
    • Can only send to CloudWatch logs
  • CloudWatch Unified Agent
    • Collect additional system-level metrics such as RAM, processes, ..
    • Collect logs to send to CloudWatch Logs
    • Centralized configuration using SSM Parameter Store

CloudWatch Unified Agent — Metrics

  • Collected directly on your Linux server / EC2 instance
  • CPU (active / guest / idle / system / user / steal)
  • Disk metrics (free, used, total), Disk IO (writes, reads, bytes, iops)
  • RAM (free / inactive / used / total / cached)
  • Netstat (number of TCP and UDP connections, net packets, bytes)
  • Processes (total, dead, blocked, idle, running, sleep)
  • Swap Space (free, used, used %)

? OOTB metrics for EC2 are disk, CPU, network (high level). If you need more, consider using CloudWatch Unified Agent

CloudWatch Alarms

  • Alarms are used to trigger notifications for any metric
  • Various options (sampling, %, max, min, etc)
  • Alarm States:
    • OK
    • INSUFFICIENT_DATA
    • ALARM
  • Period:
    • Length of time in seconds to evaluate the metric (evaluation window in Datadog)
    • Can have a short length for high resolution custom metrics (like 10 sec, 30 sec or multiple of 60 sec)

CloudWatch Alarm Targets

  • Stop, Terminate, Reboot, or Recover an EC2 instance
  • Trigger Auto Scaling Action
  • Send notification to SNS (from which you can do pretty much anything)

CloudWatch Alarms — Composite Alarms

  • CloudWatch Alarms are on a single metric
  • Composite Alarms are monitoring the states of multiple other alarms
  • AND and OR conditions
  • Helpful to reduce “alarm noise” by creating complex composite alarms

EC2 Instance Recovery

  • Status Check:
    • Instance status = check the EC2 VM
    • System status = check the underlying hardware
    • Attached EBS status = check attached EBS volumes
  • You can create CloudWatch Alarms based on these checks (e.g. StatusCheckFailed_System) and do EC2 Instance Recovery when triggered
    • Recovery will launch another EC2 instance with: same private / public / elastic IP, metadata, placement group

CloudWatch Alarm — good to know

  • Alarms can be created based on CloudWatch Logs Metrics Filters
  • To test alarms and notifications, set the alarm state to Alarm using CLI
 aws cloudwatch set-alarm-state \
   --alarm-name "<alarm-name>" \
   --state-value ALARM \
   --state-reason "<message>"

CloudWatch Synthetics Canary

  • Configurable script that monitor your APIs, URLs, Websites, …
  • Reproduce what your customers do programmatically to find issues before customers are impacted
  • Checks the availability and latency of your endpoints and can store load time data and screenshots of the UI
  • Integration with CloudWatch Alarms
  • Scripts written in Node.js or Python
  • Programmatic access to a headless Google Chrome browser
  • Can run once or on a regular schedule

CloudWatch Synthetics Canary Blueprints (templates)

  • Heartbeat Monitor — load URL, store screenshot and an HTTP archive file
  • API Canary — test basic read and write functions of REST APIs
  • Broken Link Checker — check all links inside the URL that you are testing
  • Visual Monitoring — compare a screenshot taken during a canary run with a baseline screenshot
  • Canary Recorder — used with CloudWatch Synthetics Recorder (record your actions on a website and automatically generates a script for that)
  • GUI Workflow Builder — verifies that actions can be taken on your webpage (e.g. test a webpage with a login form)

Amazon Athena

  • Serverless query service to analyze data stored in Amazon S3
  • Uses standard SQL language to query the files (built on Presto engine which is SQL language)
  • Supports CSV,JSON, ORC, Avro, and Parquet
  • Pricing: $5.00 per TB of data scanned
  • Commonly used with Amazon Quicksight for reporting / dashboards
  • Use cases: business intelligence / analyics / reporting, analyze & query VPC Flow Logs, ELB Logs, CloudWatch trails, …
  • Exam Tip: analyze data in S3 using serverless SQL, use Athena

Amazon Athena — Performance Improvement

Use columnar data for cost-savings (for less scan)

  • Apache Parquet or ORC is recommended
  • Huge performance improvement
  • Use Glue to convert your data to Parquet or ORC
  • Row-based format example:
user_id, user_name, country, purchase_amount, created_at
1, John, US, 53, 2024-01-01
2, Jane, KR, 20, 2024-01-02
...
  • Columnar-based format example:
user_id:
  1, 2, 3, ...
country:
  US, KR, JP, ...
purchase_amount:
  53, 20, 10, ...

Additional considerations:

  • Compress data for smaller retrievals (bzip2, gzip, lz4, snappy, zlip, zstd, …)
  • Partition datasets in S3 for easy querying on virtual columns
    • example: s3://athena-examples/flight/parquet/year=1991/month=1/day=1/
  • Use larger files (> 128 MB) to minimize overhead

Amazon Athena — Federated Query

  • Allows you to run SQL queries across data stored in relational, non-relational, object, and custom data sources (AWS or on-premises)
  • Uses Data Source Connectors that run on AWS Lambda to run Federated Queries (e.g. CloudWatch Logs, DynamoDB, RDS, ..)
  • Store the results back in Amazon S3

Topics

awscloudwatchobservability