Collect metrics from AWS accounts

This document describes how to collect metrics from your Amazon EC2 instances, and store those metrics in a Google Cloud project. You can create charts to display those metrics and you can create alerting policies to monitor those metrics. This page is intended for developers and system administrators who need to view and manage metrics for services and resources that are associated with AWS accounts.

Cloud Monitoring lets you import metrics from your Amazon Elastic Compute Cloud (Amazon EC2) instances and view them in the same context as your Google Cloud metrics. For example, you can create a dashboard with charts that display CPU utilization for your Amazon EC2 instances and your Compute Engine instances.

AWS Connector projects

An AWS Connector project is a Google Cloud project that lets Cloud Monitoring read metrics for a specific AWS account. The following diagram shows a Google Cloud project that has an AWS Connector project as a monitored project. That AWS Connector project reads the metrics from an AWS account and then stores those metrics:

An AWS Connector project lets you read metrics from an AWS account.

The AWS Connector project is created when you connect your AWS account to Google Cloud. For information about these steps, see Connect your AWS account to Google Cloud.

To display your AWS account metrics in multiple Google Cloud projects, connect your AWS account to Google Cloud, and then follow the steps in Add AWS Connector projects to a metrics scope.

By default, the AWS Connector project stores only Amazon EC2 metrics. To store your Amazon EC2 logs and system metrics, authorize and install the Logging agent and the Monitoring agent on those instances.

Before you begin

  • You can't collect metrics from an Amazon EC2 instance when your scoping project is in a folder.

    When your scoping project is in an organization but not a folder, ensure that you have permission to create a Google Cloud project at the organization level.

  • You must have an AWS account.

  • Ensure that your Identity and Access Management (IAM) role on the scoping project includes all permissions in the Monitoring Admin (roles/monitoring.admin) role. This role lets you modify a metrics scope. For information about IAM roles for Cloud Monitoring, see Control access with Identity and Access Management.

  • Define a naming convention for your AWS Connector projects to make them identifiable. We recommend that an AWS Connector project name includes identifying information about the AWS account it monitors. You can't change the AWS account monitored by an AWS Connector project.

  • Determine if your AWS account is connected to Google Cloud.

    • If it isn't connected to Google Cloud, then follow the instructions in Connect your AWS account to a Google Cloud project.

      In these steps, an AWS Connector project is created. Don't create multiple AWS Connector projects for the same AWS account. AWS CloudWatch might throttle your metric collection when you create multiple AWS Connector projects for the same AWS account.

    • If it is connected to Google Cloud, then you can add the AWS Connector project to multiple metrics scope. For information about these steps, see View AWS metrics in multiple metrics scopes.

  • To understand the costs associated with ingesting your AWS account metrics into Cloud Monitoring, see Pricing.

Connect your AWS account to Google Cloud

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Monitoring Settings:

    Go to Monitoring Settings

  2. Use the Google Cloud console project picker to select the scoping project for the metrics scope.
  3. In the Settings page, click Create AWS connector project.

    Create AWS Connector project button.

  4. In the Create a connector project step and click Select a project.

  5. In the dialog, select New project and complete the new project dialog.

  6. Click Next to advance to the Authorize AWS for Monitoring step.

    In this step, you create an Amazon IAM role that grants Google Cloud read-only access to your AWS account, and then you provide that role's ARN to your Google Cloud project:

  7. Create an Amazon IAM role:

    1. Open a new window and log in to your AWS account, select the IAM page and then click Roles.
    2. Select Create Role.
    3. Select Another AWS account.
    4. In the Account ID text box, enter the account ID displayed in the Authorize AWS for Monitoring page of the Google Cloud console.
    5. Select Require external ID.
    6. In the External ID text box, enter the external ID displayed in the Authorize AWS for Monitoring page of the Google Cloud console.
    7. Leave Require MFA clear and then click Next: Permissions.
    8. In the permissions search bar, enter ReadOnlyAccess and then select ReadOnlyAccess.
    9. Expand Set Permission Boundary and ensure Create role without a permissions boundary is checked.
    10. Click Next: Tags.
    11. Click Next:Review.
    12. Enter a role name and description, and then click Create Role.
    13. Select the role you created to open its Summary page. Copy the Role ARN into your clipboard.
  8. In the Google Cloud console, paste the AWS Role ARN in the Role ARN text box and then click Add AWS Account.

    After you complete these steps, the AWS Connector project is a monitored project for the current metrics scope. You can now view these AWS metrics from this metrics scope.

    After you add projects to a metrics scope, it takes about 60 seconds for changes to propagate through all Monitoring systems. Before you create a chart or alerting policy, wait at least 60 seconds. You might need to refresh the Google Cloud console page for the new metrics to be visible.

  9. (Optional) To view your AWS account metrics in multiple metrics scopes, add the AWS Connector project to those metrics scopes. For information, see Add AWS Connector projects to a metrics scope.

  10. (Optional) To collect AWS EC2 logs and system and application metrics and send them to your AWS Connector project, authorize and install the Logging and Monitoring agents on your Amazon EC2 instances:

View AWS metrics in multiple metrics scopes

To add an existing AWS Connector project to a metrics scope, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Monitoring Settings:

    Go to Monitoring Settings

  2. For each metrics scope that you want to use to view your AWS metrics, do the following:

    1. Use the Google Cloud console project picker to select the scoping project for the metrics scope.
    2. In the AWS Accounts in scope pane, click Add AWS connector project.
    3. Select the AWS Connector projects that you want to add and then click Add projects.

      After you add projects to a metrics scope, it takes about 60 seconds for changes to propagate through all Monitoring systems. Before you create a chart or alerting policy, wait at least 60 seconds. You might need to refresh the Google Cloud console page for the new metrics to be visible.

Remove AWS Connector projects from a metrics scope

When you remove a project from a metrics scope, the metrics stored in that project aren't accessible to the metrics scope. Removing a project from a metrics scope doesn't change the configuration of charts, dashboards, alerting policies, uptime checks, or groups that you defined. However, the time series displayed on charts and the time series monitored by alerting policies might change.

To remove AWS Connector projects from a metrics scope, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Monitoring Settings:

    Go to Monitoring Settings

  2. Use the Google Cloud console project picker to select the scoping project for the metrics scope.
  3. In the AWS Accounts in scope pane, select the AWS Connector projects that you want to remove and then click Remove project.

  4. Delete any AWS Connector projects that you removed and that are no longer monitored projects.

List monitored projects in a metrics scope

To display a list of projects in the current metrics scope, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Monitoring Settings:

    Go to Monitoring Settings

  2. View the tables on this page. The tables list the projects in the current metrics scope.

You can also see the list of projects for the current metrics scope by going to the navigation pane, and then clicking Expand on the Metrics scope field. The expanded pane displays the following information:

  • A list of the projects whose metrics are accessible to the current metrics scope.
  • A list of projects whose metrics scope includes the selected project.

Select a different metrics scope

The project selected in the Google Cloud console project picker is the scoping project of the current metrics scope. There is a one-to-one relationship between a scoping project and a metrics scope.

To select a different metrics scope, select a different project with the Google Cloud console project picker.

View AWS account metrics

To view the metrics stored in your AWS Connector project, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring:

    Go to Monitoring

  2. Select a metrics scope where the AWS Connector project is a monitored project.

    To view your AWS account metrics, create a chart with Metrics Explorer or create a chart on a custom dashboard.

Stop ingestion of AWS account metrics

To stop ingestion of AWS account metrics and logs, delete the AWS Connector project for that account.

Pricing

In general, Cloud Monitoring system metrics are free, and metrics from external systems, agents, or applications are not. Billable metrics are billed by either the number of bytes or the number of samples ingested.

For more information about Cloud Monitoring pricing, see the following documents:

What's next