Collect AWS CloudWatch metrics by using the Prometheus CloudWatch exporter

This document describes how to use the open source Prometheus CloudWatch exporter and the Ops Agent running on a Compute Engine instance to collect AWS CloudWatch metrics and store them in a Google Cloud project.

This document is intended for developers and system administrators who need to collect AWS CloudWatch metrics. This document describes how to set up the Prometheus CloudWatch exporter to collect AWS CloudWatch metrics.

With Cloud Monitoring, you can view your AWS metrics in the same context as your Google Cloud metrics. For example, you can create a dashboard with charts that display CPU utilization for your Amazon EC2 instances and for your Compute Engine instances. You can also create alerting policies to monitor your AWS metrics. For more information, see the following sections:

Before you begin

To collect AWS CloudWatch metrics by using the Prometheus CloudWatch exporter, you need the following:

  • A Google Cloud project with permissions to do the following:
    • Create a VM
    • Write logs to Cloud Logging
    • Write metrics to Cloud Monitoring
  • An AWS account with AWS credentials that can be used by the Prometheus exporter to fetch metrics. For more information, see Run the Prometheus exporter.

Create a Compute Engine VM

We recommend creating a Linux Compute Engine VM to use specifically for running the Ops Agent and the Prometheus CloudWatch exporter. This VM acts as the collection site for all AWS metrics.

  1. To create a Debian Linux VM named aws-exporter-test in a zone that you specify, run the following command:

    gcloud compute instances create \
  --image-project debian-cloud \
  --image-family debian-11 \
  --zone ZONE \
  aws-exporter-test

    Configure the command as follows:

    • Replace ZONE with the zone for your new VM
    • Optional. Replace aws-exporter-test with different name for your VM.

    For more information about this command, see the gcloud compute instances create reference.

  2. To access your VM so that you can install the Prometheus CloudWatch exporter and the Ops Agent, you can use the following command:

    gcloud compute ssh --zone ZONE  --project PROJECT_ID  aws-exporter-test

    Configure the command as follows:

    • Replace ZONE with the zone in which you created your VM
    • Replace PROJECT_ID with ID of your Google Cloud project
    • Replace aws-exporter-test if you created your VM with a different name

    For more information about this command, see the gcloud compute ssh reference.

Set up the Prometheus CloudWatch exporter

The following sections describe the procedure for downloading, installing, and configuring the Prometheus CloudWatch exporter on your Compute Engine VM.

Download the Prometheus exporter and the JRE

To run the Prometheus CloudWatch exporter, you need to download the exporter and the Java Runtime Environment (JRE), version 11 or newer.

  1. To download the JAR file containing the Prometheus CloudWatch exporter, run the following command on your Compute Engine instance:

    curl -sSO https://github.com/prometheus/cloudwatch_exporter/releases/download/v0.15.1/cloudwatch_exporter-0.15.1-jar-with-dependencies.jar

  2. To install the JRE, you can use a command like the following:

    sudo apt install default-jre

Configure the Prometheus exporter

To configure the Prometheus CloudWatch exporter, you create a configuration file for the AWS service or services from which you want to collect metrics. For general information, see the Prometheus CloudWatch exporter configuration documentation.

  • Migrating users: If you are migrating your existing AWS CloudWatch metrics to the Prometheus CloudWatch exporter, then you can use the configuration files in Prometheus CloudWatch exporter configurations for migration. These configuration files are designed to replicate the existing metrics as closely as possible, but they do not collect all the metrics available by using the Prometheus CloudWatch exporter for the AWS services.

  • New users: If you are not migrating existing metrics, we recommend that you don't use the migration configurations. See the AWS CloudWatch service documentation for information about how to define exporter configurations for other services. You can also find additional samples in the Prometheus CloudWatch exporter GitHub repository.

You can combine configuration for multiple AWS services into one configuration file. The examples in this document assume that your configuration file is named config.yml.

Run the Prometheus exporter

Before you can run the Prometheus CloudWatch exporter, you must provide the exporter with credentials and authorization. The Prometheus CloudWatch exporter uses the the AWS Java SDK, which offers ways to provide credentials by using the following environment variables:

For more information about providing credentials to the SDK, see AWS SDK for Java 2.x.

You must also have permission to use the CloudWatch API to retrieve metrics, You need the following AWS IAM CloudWatch permissions:

  • cloudwatch:ListMetrics
  • cloudwatch:GetMetricStatistics
  • cloudwatch:GetMetricData

Using the aws_tag_select feature also requires the tag:GetResources AWS IAM permission.

For more information about authorizing access to AWS services, see AWS Identity and Access Management.

To run the Prometheus CloudWatch exporter, do the following:

  1. To provide credentials for the exporter, set the access-key environment variables:

    export AWS_ACCESS_KEY=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN=YOUR_SESSION_TOKEN

    Replace the YOUR_KEY variables with your access keys. You need to set the AWS_SESSION_TOKEN environment variable only if you are using temporary credentials.

  2. To test your configuration, start the exporter and load your configuration file, run the following command:

    java -jar cloudwatch_exporter-0.15.1-jar-with-dependencies.jar 9106 config.yml

    Change the port (9106) and configuration-file (config.yml) values if necessary.

    If you modify your config.yml file while the exporter is running, then you can reload the exporter by running the following command:

    curl -X POST localhost:9106/-/reload

    For use in a production environment, you can configure the exporter to restart if the VM restarts. For example, on Debian systems, you can use the system and service manager, systemd.

Set up the Ops Agent

The following sections describe how to install configure, and start the Ops Agent. These sections provide minimal set-up information for the Ops Agent for use with the Prometheus CloudWatch exporter. For more information about these topics, see Ops Agent overview.

Install the Ops Agent

To install the Ops Agent, use the following commands to download and run the agent's installation script:

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install

Configure the Ops Agent

To configure the Ops Agent, you add configuration elements to the agent's user-configuration file. On Linux, the user-configuration file is /etc/google-cloud-ops-agent/config.yaml.

When you configure the Ops Agent to collect AWS metrics from the Prometheus CloudWatch exporter, you use the agent's Prometheus receiver. This document describes two basic configurations for the Ops Agent. Select one of the configurations and add it to the user-configuration file:

sudo vim /etc/google-cloud-ops-agent/config.yaml

Basic configuration for the Ops Agent

The following configuration provides a minimal configuration for the Ops Agent. This configuration does the following:

  • Creates a receiver named aws of type prometheus. The receiver is configured to scrape metrics from the aws_exporter job. The port specified must match the port on which the Prometheus CloudWatch exporter is exporting metrics; see Run the Prometheus exporter.

  • Creates a pipeline named aws_pipeline that uses the aws metrics recevier.

metrics:
  receivers:
    aws:
      type: prometheus
      config:
        scrape_configs:
          - job_name: 'aws_exporter'
            scrape_interval: 10s
            static_configs:
              - targets: ['localhost:9106']
  service:
    pipelines:
      aws_pipeline:
        receivers:
          - aws

Configuration that drops JVM metrics

The following configuration does everything that the basic configuration does, but it also adds a relabeling config that drops the JVM metrics generated by the exporter. Dropping these metrics reduces the ingested metric data, but it can make problems with the exporter more difficult to debug, because you aren't getting the exporter's own metrics:

metrics:
  receivers:
    aws:
      type: prometheus
      config:
        scrape_configs:
          - job_name: 'aws_exporter'
            scrape_interval: 10s
            static_configs:
              - targets: ['localhost:9106']
            # Drop the exporter's own JVM metrics to reduce noise.
            metric_relabel_configs:
              - source_labels: [ __name__ ]
                regex: 'jvm_.*'
                action: drop
  service:
    pipelines:
      aws_pipeline:
        receivers:
          - aws

You can create much more complex configurations for the Ops Agent.

Restart the Ops Agent

To apply configuration changes to the Ops Agent, you must restart the agent.

  1. To restart the agent, run the following command on your instance: 
    sudo service google-cloud-ops-agent restart
  2. To confirm that the agent restarted, run the following command and verify that the components "Metrics Agent" and "Logging Agent" started: 
    sudo systemctl status google-cloud-ops-agent"*"

View metrics in Cloud Monitoring

In Cloud Monitoring, you can query your AWS CloudWatch metrics and create charts like you do for any other metrics. From the Metrics Explorer interface, you can use PromQL, Monitoring Query Language (MQL), or a query-builder interface. For more information, see Create charts with Metrics Explorer.

If you have created charts you that want to keep, then you can save them to custom dashboards. For more information, see Dashboards overview.

The following chart shows a PromQL query for the aws_ec2_cpuutilization_sum metric for AWS VMs:

The chart shows the result of fetching the aws_ec2_cpuutilization_sum statistic for AWS VMs by using PromQL.

You can query any metric in Cloud Monitoring by using PromQL. For information, see Mapping Cloud Monitoring metrics to PromQL.

You can query Prometheus metrics by using PromQL or by using Cloud Monitoring tools like Monitoring Query Language (MQL). When Prometheus metrics are ingested into Cloud Monitoring, each metric is transformed by using the standard OpenTelemetry-to-Prometheus transformation and mapped to the Cloud Monitoring prometheus_target monitored-resource-type. The transformation includes the following changes:

  • The metric name is prefixed with the string prometheus.googleapis.com/.
  • Any non-alphanumeric characters, such as periods (.), in the metric name are replaced by underscores (_).
  • The metric name is postfixed with a string that indicates the metric kind, like /gauge or /counter.

To query the Prometheus aws_ec2_cpuutilization_sum metric by using MQL, refer to the metric as prometheus.googleapis.com/aws_ec2_cpuutilization_sum/gauge, and specify the prometheus_target monitored-resource type: 

fetch prometheus_target :: 'prometheus.googleapis.com/aws_ec2_cpuutilization_sum/gauge'

The following chart shows the result of the MQL query for the Prometheus aws_ec2_cpuutilization_sum metric:

The chart shows the result of fetching the aws_ec2_cpuutilization_sum statistic for AWS VMs by using MQL.

Alert on metric behavior

In Cloud Monitoring, you can create [alerting policies][alert-intro] to monitor your AWS CloudWatch metrics and notify you of spikes, dips, or trends in metric values.

Monitor multiple regions

The configuration of the Prometheus CloudWatch exporter supports the use of only one region per configuration file. If you need to monitor multiple regions, then we recommend that you run multiple instances of the Prometheus exporter, one configured for each region you want to monitor. You can run multiple exporters on a single VM, or you can distribute them across VMs. The Prometheus exporter Docker images might be useful in this situation.

You can configure the Ops Agent running on the Compute Engine VM to scrape multiple Prometheus endpoints. We recommend that, when you configure multiple instances of the Prometheus exporter, you use a different job name the scrape config for each, so you can distinguish the exporter instances if you need to troubleshoot them.

For information about configuring the Ops Agent and the Prometheus receiver, see Configure the Ops Agent.

Cost

Amazon charges for every CloudWatch API request or for every CloudWatch metric you request; for current pricing, see Amazon CloudWatch Pricing. The Prometheus CloudWatch exporter has the following query characteristics, which can affect your Amazon costs:

  • When using the GetMetricStatistics method (default), every metric requires one API request. Each request can include multiple statistics.
  • When using aws_dimensions, the exporter must perform API requests to determine which metrics to request. The number of dimensions requests is typically negligible in comparison to the number of metric requests.

Cloud Monitoring charges for AWS CloudWatch metrics from the Prometheus exporter by the number of samples ingested. For current pricing, see Monitoring pricing summary.

Prometheus CloudWatch exporter configurations for migration

This section provides configurations that replicate, as closely as possible, the AWS service metrics documented in the AWS metrics list. These configuration files are intended for use by customers who are migrating to the Prometheus CloudWatch exporter from the legacy solution. If you are setting up the Prometheus CloudWatch exporter as a new user rather than a migrating user and you use these configurations, then you are not collecting all the AWS metrics that the Prometheus CloudWatch exporter makes available.

To view a sample configuration file for AWS CloudWatch metrics, expand one of the following sections.