How do I retrieve log data from CloudWatch Logs?

Short description

To retrieve your log data from CloudWatch Logs, use the following best practices based on your use case:

• Stream log data with subscription filters.
• Run a CloudWatch Logs Insights query.
• Export log data to Amazon Simple Storage Service (Amazon S3).
• Use CloudWatch APIs.
• Download logs into a .csv file.

Resolution

Stream log data with subscription filters

To retrieve your log data from CloudWatch Logs in real time, use subscription filters. Log data subscription filters prevent CloudWatch API throttling. For more information, see Real-time processing of log data with subscriptions and Log group-level subscription filters. For Amazon Kinesis Data Streams, throttled service APIs are automatically retried.

Run a CloudWatch Logs Insights query

To search and analyze your log data, run a query in CloudWatch Logs Insights. Queries help you respond to issues, and identify causes for further troubleshooting. For more information, see Analyzing log data with CloudWatch Logs Insights.

Export log data to Amazon S3

You can export CloudWatch Logs data to Amazon S3 in batch use cases. The log data export can take up to 12 hours to become available from CloudWatch Logs. For real-time analysis and processing, use subscription filters.

Note: An AWS account can have only one export task for log data in the PENDING, PENDING_CANCEL, or RUNNING state.

Use CloudWatch APIs

To manually locate your log data, use the GetLogEvents or FilterLogEvents CloudWatch APIs. These APIs aren't scalable due to the transactions per second (TPS) quota. For current quotas, see CloudWatch Logs quotas. If you experience API throttling, then use subscription filters.

Download logs into a .csv file

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To retrieve your log data in .csv file format, use the following methods:

• Use the CloudWatch console.
• Use the AWS CLI.
• Use CloudWatch Logs Insights.

Use the CloudWatch console

Complete the following steps:

1. Open the CloudWatch console, and then select your log group.
2. Choose the log stream that contains the log events that you want to download.
3. Choose Actions, and then choose Download Search Results.

Use the AWS CLI

To retrieve log events in .csv file format, run the get-log-events command:

Note: Replace example-loggroup-name with your log group name and example-logstream-name with your log stream name.

aws logs get-log-events --log-group-name example-loggroup-name  --log-stream-name example-logstream-name  --start-time 1705147529 --end-time 1705233929 > log_events.csv

Note: The .csv file is stored in the same directory that the command was initiated from. Also, the AWS CLI uses the default AWS Region and profile that's configured for the workstation. Specify the appropriate Region. The GetLogEvents API might return empty results when there are more log events available through the token.

Use CloudWatch Logs Insights

Complete the following steps:

1. Run a query to retrieve your log data.
2. Depending on the number of logs, use on of the following methods to export the data:
   For fewer than 10,000 logs, choose Export results.
   For more than 10,000 logs, export the logs to an Amazon S3 bucket. Then, download the data into a .csv file.