Quickstart for AWS

This quickstart shows you how to collect system metrics from an EC2 instance and how to view those metrics in Cloud Monitoring. In this quickstart, you do all of the following:

  1. Install the Monitoring and Logging agents on your EC2 instances.
  2. Configure your Amazon Web Services (AWS) account to enable sending the data collected by the agents to Monitoring.
  3. View and display that data.

On this page, the term connected means that an AWS account is configured to send the data collected by the Monitoring and Logging agents to Cloud Monitoring.

Before you begin

You must have an AWS account that isn't currently monitored by a Workspace. You cannot monitor an AWS account from more than one Workspace.

To disconnect an AWS account from a Workspace, go to Removing a project from a Workspace.

Overview of steps

Adding an AWS account to a Workspace requires that you create a Google Cloud project to serve as the host project for the workspace. After the Workspace is created, you add the AWS account to the Workspace.

The following steps connect your AWS account to Monitoring:

  1. Create a new Google Cloud project.

  2. Create a new Workspace (recommended) or Connect an AWS account (if you want to use an existing Workspace).

  3. Identify your trusted account ID and external ID.

  4. Create an AWS role using the Account ID and External ID.

  5. Connect your Workspace and AWS account using the AWS Role by using an AWS connector project.

  6. Create a service account in the AWS connector project to authorize access to Google Cloud.

Each of the preceding steps is described in detail in the following sections.

Configuring your Workspace

It is recommended that you Create a new Workspace for this quickstart. However, if you want to use an existing Workspace, skip ahead to Connect an AWS account.

In either case, be sure to get the Account ID and External ID that you need for your AWS account. For more information, go to Getting your account and external IDs.

Create a Google Cloud project

To create a Google Cloud project:

  1. In the Cloud Console, go to New Project.

    Create a New Project

  2. In the Project Name field, enter Quickstart.

  3. Click Create.

Get your account and external IDs

To identify the trusted account ID and external ID required by AWS:

  1. Go to Monitoring

    Go to Monitoring

    The first time you access Monitoring for a Google Cloud project, Monitoring creates a Workspace and associates it with your project. This process is automatic unless you have a multi-project Workspace. In this case, a dialog appears that asks you to select between creating a Workspace and adding the project to an existing Workspace. Select the option to create a Workspace.

  2. Click Settings and select the Summary tab.

  3. Click Add AWS account.

  4. Record the Account ID and External ID. You need this data to create your AWS Role.

  5. Click Cancel. You add your AWS account after you create your AWS role.

Creating an AWS role

To create your AWS role needed to authorize Cloud Monitoring, you must have the Account ID and External ID for your Workspace. If you don't have them, follow the instructions in Getting your account and external IDs.

To create the AWS role, do the following:

  1. Log into your AWS IAM console and click Roles in the left-side menu.
  2. Click Create role and do the following:

    • For the Role type, select Another AWS account.
    • In the Account ID field, enter the account ID provided by Monitoring.
    • Select the Require external ID checkbox.
    • In the External ID field, enter the external ID provided by Monitoring.
    • Don't select Require MFA.
  3. Click Next: Permissions.

  4. From the Policy name drop-down list, select ReadOnlyAccess:

  5. Click Next: Tags.

  6. (Optional) Add metadata to the role by attaching tags as key–value pairs.

  7. Click Next: Review and fill in or verify the following information:

    • In the Role name field, enter a name such as GoogleStackdriver.
    • (Optional) In the Role description field, enter anything you wish.
    • In the Trusted entities field, verify it's the Account ID you entered earlier.
    • In the Policies field, verify the value is ReadOnlyAccess.
  8. In the AWS IAM page, click Create Role.

  9. On the Summary page, copy the Role ARN string so that you can give it to Monitoring. If you don't see the summary, click the name of your role (for example, GoogleStackdriver) in the list of AWS roles.

Connecting an AWS account

To add an AWS account to an existing Workspace, do the following:

  1. In the Cloud Console, go to Monitoring:

    Go to Monitoring

  2. Click Settings and select the Summary tab.

    The pane in the following screenshot shows that you are monitoring a single Google Cloud project—the Workspace's hosting project. You aren't yet monitoring any AWS accounts.

    Monitoring monitored accounts.

  3. Click Add AWS account.

  4. Click Select a project and create a Google Cloud project or select an existing project to be used as the AWS connector project.

    A Google Cloud project can be a connector project for only one AWS account.

    Don't use connector projects for any other purpose, and don't delete them while your Workspace is still connected to your AWS account.

  5. Enter the following information in the form:

    • In the Role ARN field, enter your Role ARN from Creating an AWS role or follow the instructions on the Add AWS account page to create the role.
    • In the Description of account field, enter a short description of your AWS account. The first word or two is used to create a new project ID.

      Monitoring AWS monitored accounts.

  6. Click Add AWS account. In a moment, the connection is confirmed.

AWS connector projects

The Monitored accounts page in your Workspace settings now includes the ID for the AWS connector project:

Your AWS account description [YOUR_AWS_ACCOUNT_NUMBER]
Connected to [CONNECTOR_PROJECT_ID]

Where:

  • [YOUR_AWS_ACCOUNT_NUMBER] represents the account number for your AWS account.
  • [CONNECTOR_PROJECT_ID] represents the connector project where you receive logs and metrics from your AWS account and where you set up authorization for agents and other AWS applications that need to access Google Cloud.

Next step: Authorizing AWS applications

Troubleshooting

If you are told that your AWS account is already being monitored, do the following:

  • If another Workspace is monitoring your AWS account, then you must remove your AWS account from it. You cannot monitor an AWS account from more than one Workspace. To disconnect an AWS account from a Workspace, go to Removing a project from a Workspace.

  • This message can also appear if you didn't use the correct Account ID and External ID from your present Workspace when you created your AWS Role. The External ID is unique for each Workspace.

Authorizing AWS applications

You must perform the following steps if you do any of the following:

  • Run the Monitoring or Logging agents on your AWS VM instances.
  • Use any Google Cloud services from AWS applications.

To authorize applications running on AWS to access Google Cloud services, you give them access to a Google Cloud service account that has suitable Google Cloud IAM roles.

A single service account can authorize multiple AWS VM instances and applications in the same AWS account, or you can create multiple service accounts.

Create a service account

To create the service account, do the following:

  1. Go to the IAM & Admin > Service accounts page for your connector project:

    Go to Service accounts

  2. Select the AWS connector project for your AWS account.

  3. Your connector project likely has no service accounts, so you are asked to create one. Click Create service account and enter the following information:

    • In the Service account name field, enter Monitoring agent authorization and click Create.

      Create service account.

    • In the Service account permissions dialog, select the following roles and then click Continue:

      • Monitoring > Monitoring Metric Writer
      • Logging > Logs Writer

      Create service account.

    • In the Create key dialog, click Create key and select JSON.

  4. Click Done. The service account's private-key file is downloaded to your workstation with a name such as Downloads/[PROJECT_NAME]-[KEY_ID].json.

    Where:

    • [PROJECT_NAME] represents the name of your Google Cloud project.
    • [KEY_ID] represents the generated private key.

    To make the following instructions simpler, save the location of the credentials file in the variable CREDS on your workstation:

    CREDS="Downloads/[PROJECT_NAME]-[KEY_ID].json"
    

Add a service account to a VM instance

To add a service account, do the following:

  1. From your workstation, copy the private-key credentials file to your AWS EC2 instance and save it in a file named temp.json. In the scp command, specify the path to key.pem, your AWS SSH key pair file, and provide your AWS credentials:

    KEY="/path/to/key.pem"
    scp -i "$KEY" "$CREDS" AWS_USERNAME@AWS_HOSTNAME:temp.json
    
  2. On your EC2 instance, move the credentials to /etc/google/auth/application_default_credentials.json:

    GOOGLE_APPLICATION_CREDENTIALS="/etc/google/auth/application_default_credentials.json"
    sudo mkdir -p $(dirname "$GOOGLE_APPLICATION_CREDENTIALS")
    sudo mv "$HOME/temp.json" "$GOOGLE_APPLICATION_CREDENTIALS"
    
  3. (Optional): Restrict access to the private-key credentials for the service account. For example:

    sudo chown root:root "$GOOGLE_APPLICATION_CREDENTIALS"
    sudo chmod 0400 "$GOOGLE_APPLICATION_CREDENTIALS"
    
  4. Make sure the environment variable GOOGLE_APPLICATION_CREDENTIALS is visible to the agents and other applications that are authorized to use Google Cloud. The environment variable name is understood by the standard Google Cloud client libraries.

Install the agents

  1. (Optional): Install the Cloud Monitoring agent on your EC2 instance. While Cloud Monitoring can access some instance metrics without the Monitoring agent, including CPU utilization and uptime information, the Monitoring agent provides metrics on a broader set of system resources and application services.

    To install the Cloud Monitoring agent, see Installing the Cloud Monitoring agent and select the instructions based on your operating system. These instructions include information on how to verify that the agent is running and a link to a troubleshooting guide.

  2. (Optional): Install the Cloud Logging agent, which streams logs to to Cloud Logging, on your EC2 instance. See Installing the Cloud Logging agent and select the instructions based on your operating system. These instructions include information on how to verify that the agent is running and a link to a troubleshooting guide.

Using Monitoring services with AWS

This section shows you how to use Monitoring services with your AWS account.

Create an uptime check and an alerting policy

To create an uptime check, do the following:

  1. In the Cloud Console, select Monitoring:

    Go to Monitoring

    The first time you access Monitoring for a Google Cloud project, Monitoring creates a Workspace and associates it with your project. This process is automatic unless you have a multi-project Workspace. In this case, a dialog appears that asks you to select between creating a Workspace and adding the project to an existing Workspace. Select the option to create a Workspace.

  2. Click Uptime checks.

  3. Click Create Uptime check.

    Create an uptime check dialog.

  4. For the title, enter My Uptime Check and then click Next.

  5. Target:

    1. Ensure HTTP is selected for the protocol.
    2. Select any available resource for the Resource Type.
    3. Complete additional fields as necessary.
    4. Click Next.
  6. Response Validation: Leave these fields at their default values and click Next

  7. Alert & Notification:

    • Ensure that the toggle's label is Alerting is enabled.
    • Leave the name and duration fields at their default values.
    • To add a notification channel to the alerting policy, in the text box labeled Notification channels, click Menu . Select the channels to add and click OK. The notifications are grouped alphabetically for each channel type.

      To add an entry to the checkbox list, click Manage notification channels and follow the instructions. When you return to this dialog, click Refresh .

  8. To verify your uptime check configuration, click Test. If you see a "Connection error - refused" message, you might have not installed the Apache HTTP Server or you might have specified the HTTPS check type rather than HTTP. For other errors, see Verify your uptime check.

  9. Click Create. When the create action is successful, a banner displays Check and alert created. The Uptime checks window lists the new check and contains a link to the uptime checks' dashboard. The Alerting window lists the new alerting policy and contains a link to the policies dashboard. If required data is missing, the create action fails and a list of fields that require data is displayed next to the dialog buttons.

Create a dashboard and chart

To display the metrics collected by Monitoring, complete the following steps:

  1. Go to Monitoring:

    Go to Monitoring

  2. Select Dashboards and then select Create dashboard.

  3. Enter Quickstart dashboard as the name for the dashboard and click Confirm.

  4. Click Add Chart.

  5. Ensure the Metric tab is selected:

    Display the add chart dialog with default settings.

  6. Under the heading Find resource type and metric, click the textbox and select an AWS metric.

  7. Click Save.

View your logs

Monitoring and Logging are closely integrated.

  1. In the Cloud Console , go to Logging and then select the AWS connector project.
  2. The Logs Explorer for your AWS connector project, contains your AWS logs. To change the Logs Explorer focus to see the logs you want:

    • Go to Google Project > All project_id You should see at least one audit log from setting up your AWS connector project:

      AWS Logs Viewer.

    • If you installed the Cloud Monitoring agent on your supported AWS VM instances, you might see other log options.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this quickstart, follow these steps.

  1. Remove your Monitoring charts and alerts. Go to Monitoring:

    1. Delete your alerting policy from Alerting.
    2. Delete your uptime check from Alerting.
    3. Delete your charts from Dashboards.
  2. Click Settings and select the Summary tab.

  3. In the section titled AWS Accounts, identify the AWS account that you used for this Quickstart, click More , and select Remove from workspace.

  4. In your Amazon account, delete the AWS IAM role that you created for the quickstart.

  5. In the Google Cloud Console, delete your AWS connector project and—if you created it for this quickstart—your Google Cloud project, aws-quickstart. To delete a project, you select the project, go to IAM & Admin and select Settings, and then click Shut down.

What's next

  • Go to Supported metrics for a list of all the built-in metrics. There are over 500 metrics for Amazon AWS. If you want to create your own Monitoring metrics, go to Custom metrics.

  • To use the Monitoring API, go to the API reference.

  • For more information on logging and its relation to monitoring, go to Logging.

  • Read our resources about DevOps and explore our research program.