How can I collect custom metrics from Amazon EMR cluster instances and monitor them in CloudWatch?

Resolution

You can configure and collect metrics from an Amazon EMR cluster by installing the CloudWatch agent on Amazon Elastic Compute Cloud (Amazon EC2). After installing the CloudWatch agent, you can use the metrics collected in addition to the default published metrics for Amazon EC2 instances.

You can store and view the metrics that you collect with the CloudWatch agent in CloudWatch similar to any other CloudWatch metrics. The default namespace for metrics collected by the CloudWatch agent is CWAgent. However, you can specify a different namespace when you configure the agent.

To configure the CloudWatch agent and publish custom metrics from the Amazon EMR cluster nodes, follow these steps:

1.    Create an example CloudWatch agent configuration file (config.json) with the following example configuration. The agent configuration file is a JSON file that specifies the metrics and logs that the agent must collect, including custom metrics. In the following example for collecting the metrics for a Linux server, four CPU metrics, two disk metrics, and one memory metric are collecting. The agent is set up to receive these metrics from a collectd client.

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "aggregation_dimensions": [
      [
        "InstanceId"
      ]
    ],
    "append_dimensions": {
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "collectd": {
        "metrics_aggregation_interval": 60
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ],
        "totalcpu": false
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "statsd": {
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      }
    }
  }
}

2.    Copy the following bootstrap script (example: emr-cloudwatchagent-bootstrap.sh) to your Amazon Simple Storage Service (Amazon S3) location.

------emr-cloudwatchagent-bootstrap.sh---------
#!/bin/bash

sudo yum install amazon-cloudwatch-agent -y
sudo amazon-linux-extras install collectd -y

aws s3 cp <s3 path for config.json> /home/hadoop/config.json

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:///home/hadoop/config.json
------------------------------

The script does the following:

• Installs the CloudWatch agent and collectd client.
• Copies the config.json file from Amazon S3 to the local path /home/hadoop.
• Runs the CloudWatch agent with the config.json file.

3.    Add the bootstrap action to the Amazon EMR cluster.

After importing the collectd metrics into CloudWatch, you can view these metrics as time series graphs. You can also set alarms so that you are notified when the metrics exceed a certain threshold that you specify. To view the collectd metrics in the CloudWatch console,. choose the namespace for the metrics that are collected by the agent. By default, this namespace is called CWagent. For more information, see Viewing collectd metrics imported by the CloudWatch agent.