Quickstart for AWS

This Quickstart shows you how to connect Stackdriver Monitoring to your Amazon Web Services (AWS) account, and how to install the Stackdriver Monitoring and Logging agents on your EC2 instances.

Recent changes:

  • You no longer create an EC2 instance in your AWS account as part of the Quickstart.
  • You create your AWS Role at a different stage of the Quickstart.
  • You use the environment variable GOOGLE_APPLICATION_CREDENTIALS to describe where GCP service account credentials are stored on AWS.

Before you begin

  • You must have an AWS account that is not currently monitored by Stackdriver. You cannot monitor an AWS account from more than one Stackdriver account.

    To disconnect an AWS account from a Stackdriver account, go to the Account settings > Monitored accounts page of the Stackdriver Monitoring Console. Choose a connected AWS account and click Remove from account from its menu.

  • Only Stackdriver accounts in the Premium service tier can connect to AWS accounts. This Quickstart begins by creating a new Stackdriver account that has a 30-day free trial of the Premium Tier, so you do not incur any charges. For more information, see Stackdriver Pricing.

Overview of steps

Use the following steps to connect your AWS account to Stackdriver Monitoring:

  1. Create a new Stackdriver account in Cloud Platform. Get the Account ID and External ID for AWS.

  2. Create an AWS Role using the Account ID and External ID.

  3. Connect your Stackdriver account and AWS account using the AWS Role. This creates a new AWS connector project.

  4. Create a service account in the AWS connector project to authorize access to GCP.

Each of the preceding steps is described in detail in the following sections.

Next step: One of the following:

Creating a new Stackdriver account

To create a new Stackdriver account to use with your AWS account, do the following:

  1. Go to the Stackdriver Monitoring Console:

    Go to the Stackdriver Monitoring Console

  2. If you are not asked to create a Stackdriver account immediately, then select Create Stackdriver account from the drop-down list of Stackdriver accounts at the top of the page. You see the Create your free Stackdriver account page:

    Create Stackdriver account

  3. In the text box Google Cloud Platform Project, select New Project and enter a project name, such as stackdriver-aws-quickstart.

  4. Click Create account. There is a pause while Stackdriver creates the new GCP project, which is also your Stackdriver account.

  5. You see the page Add Google Cloud Platform projects to monitor. Click Continue, because you are not adding any GCP projects.

  6. You see the page Monitor AWS accounts. In the instructions appearing on that page, look for an Account ID and an External ID, which are specific to your new Stackdriver account. For example, you see:

    Enter the following:
      Account ID: a number
      External ID: an ID string

    Save these numbers, which you will need in the following section.

  7. Click Continue at the bottom of the Monitor AWS accounts page to finish creating your Stackdriver account.

  8. You can skip through the following pages until you see Gathering information... and then Finished initial collection!

Next step: Creating an AWS role.

Creating an AWS role

To create the AWS role needed to authorize Stackdriver, do the following:

  1. Log into your AWS IAM console and click Roles in the left-side menu.
  2. Click Create New Role and do the following:

    • Select the role type Another AWS account.
    • For Account ID, enter the account ID provided by Stackdriver.
    • Check the box Require external ID.
    • For External ID, enter the external ID provided by Stackdriver.
    • Do not check Require MFA.
    • Click Next: Permissions.
  3. From the Policy name list, select ReadOnlyAccess, which is near the bottom of the long list:

    ReadOnlyAccess policy

  4. Click Next: Review and fill in or verify the following information:

    • For Role name, enter a name such as GoogleStackdriver.
    • For Role description, enter anything you wish.
    • The value of Trusted entities should be the Account ID entered earlier.
    • The value of Policies should be ReadOnlyAccess.
  5. Click Create Role in the AWS IAM page.

  6. On the Summary page for your AWS role, copy the Role ARN string so that you can give it to Stackdriver. If you do not see the summary, click the name of your role (for example, GoogleStackdriver) in the list of AWS roles.

Next step: Connect your AWS account

Connecting an AWS account

To add an AWS account to an existing Premium Tier Stackdriver account, do the following:

  1. Go to the Stackdriver Monitoring Console:

    Go to the Stackdriver Monitoring Console

  2. From the project menu at the top of the page, select your Stackdriver account. If you created a new Stackdriver account as part of this Quickstart, then select that Stackdriver account.

  3. Click Account settings at the bottom of the same project menu: You should see a page like the following:

    Stackdriver account settings

  4. Click Monitored accounts under Settings. The panel in the following screenshot shows that you are monitoring a single GCP project—the Stackdriver account project itself. There are not yet any AWS accounts being monitored.

    Stackdriver monitored accounts

  5. Click Add AWS account. You see instructions for creating an AWS Role, and in those instructions are two important numbers: an Account ID and an External ID:

    Enter the following:
      Account ID: a number
      External ID: an ID string

    Below the instructions for creating an AWS Role, you see the following form:

    Stackdriver monitored accounts

    If you followed the previous Quickstart instructions, then you should already have a Role ARN. If not, see Creating an AWS role or follow the instructions on the Add AWS account page to create the role.

  6. Enter the following information into the preceding form:

    • Role ARN: Enter your Role ARN from Creating an AWS role.
    • Description of account: Enter a short description of your AWS account. The first word or two will be used to create a new project ID.
  7. Click Add AWS account at the bottom of the form. In a moment, the connection is confirmed.

You can now leave the Account settings page.

Next step: Authorizing AWS applications

Troubleshooting

If you are told that your AWS account is already being monitored, do the following:

  • If another Stackdriver account is monitoring your AWS account, then you must remove your AWS account from it. Go to the Account settings > Monitored accounts page of the Stackdriver account to remove the AWS account.

  • This message can also appear if you did not use the correct Account ID and External ID from your present Stackdriver account when you created your AWS Role. The External ID is different for each Stackdriver account.

AWS connector projects

When you connect to an AWS account, Stackdriver Monitoring creates an AWS connector project for you. The Monitored accounts page in your Stackdriver account settings now includes the ID for this project:

Your AWS account description [YOUR_AWS_ACCOUNT_NUMBER]
Connected to [CONNECTOR_PROJECT_ID]

The connector project's ID always begins with aws-, and the project's name always begins with AWS Link.

The connector project is where you receive logs and metrics from your AWS account and where you set up authorization for agents and other AWS applications that need to access GCP.

Authorizing AWS applications

You must perform the following steps if you do any of the following:

  • Run any of the Stackdriver agents on AWS VM instances.
  • Use any Cloud Platform services from AWS applications.

Overview

To authorize applications running on AWS to access GCP services, give them access to a GCP service account that has suitable GCP IAM roles.

Create the service account in the AWS connector project for your AWS account. To find the connector project, see the Account settings > Monitored accounts page of your Stackdriver account.

A single service account can authorize multiple AWS VM instances and applications in the same AWS account, or you can create multiple service accounts.

Creating a service account

Service accounts are managed in the GCP Console, not in the Stackdriver Monitoring Console.

To create the service account, go to the IAM & Admin > Service accounts page for your connector project:

Go to the Service Accounts page

  1. Choose the AWS connector project (named AWS Link...) for your AWS account.

  2. Your connector project likely has no service accounts, so you are asked to create one. Click Create service account and enter the following information:

    • Service account name: Stackdriver agent authorization
    • Role: Add both of the following:
      • Monitoring > Monitoring Metric Writer
      • Logging > Logs Writer
    • Furnish a new private key: (checked)
    • Key type: JSON
    • Enable G Suite Domain-wide Delegation: (leave unchecked)

    Create service account

  3. Click Create. The service account's private-key file is downloaded to your workstation, with a name like Downloads/[PROJECT_NAME]-[KEY_ID].json.

    To make some of the following instructions simpler, save the location of the credentials file in variable CREDS on your workstation:

    CREDS="Downloads/[PROJECT_NAME]-[KEY_ID].json"
    

Adding a service account to a VM instance

Copy the Stackdriver private-key credentials file on your workstation to a location on your AWS EC2 instance. You must define the environment variable GOOGLE_APPLICATION_CREDENTIALS to hold its path name:

  1. From your workstation, copy the credentials file to a temporary file on AWS. The file key.pem is your AWS scp key pair file.

    KEY="/path/to/key.pem"
    scp -i "$KEY" "$CREDS" "ec2-user@$NAME:temp.json"
    
  2. On your EC2 instance, move $HOME/temp.json, to its final location. The following name and location is arbitrary:

    GOOGLE_APPLICATION_CREDENTIALS="/etc/google/auth/application_default_credentials.json"
    sudo mkdir -p $(dirname $GOOGLE_APPLICATION_CREDENTIALS)
    sudo mv "$HOME/temp.json" "$GOOGLE_APPLICATION_CREDENTIALS"
    
  3. (Optional) You might want to restrict access to the private-key credentials for the service account:

    sudo chown root:root "$GOOGLE_APPLICATION_CREDENTIALS"
    sudo chmod 0400 "$GOOGLE_APPLICATION_CREDENTIALS"
    
  4. Make sure the environment variable GOOGLE_APPLICATION_CREDENTIALS is visible to the agents and other applications that should be authorized to use GCP. The environment variable name is understood by the standard GCP client libraries.

Installing the agents

(Optional) Install the Stackdriver Monitoring and Logging agents by running the following commands on your EC2 instance:

curl -O https://repo.stackdriver.com/stack-install.sh
sudo bash stack-install.sh --write-gcm

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh

To verify that the agents are running, use the following two commands:

ps ax | grep fluentd
ps ax | grep collectd

Expected output:

[PROCESS_ID] ?    Sl   0:00 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd ...
[PROCESS_ID] ?    Ssl  0:00 /opt/stackdriver/collectd/sbin/stackdriver-collectd ...

Using Stackdriver services with AWS

This section shows you how to use Stackdriver services with your AWS account.

Creating uptime checks and alerting policies

Uptime checks verify that your web server is always accessible. The alerting policy controls who is notified if the uptime checks should fail. There are more details at Using Uptime Checks and Introduction to Alerting. Following are abbreviated instructions:

  1. Go back to the Stackdriver Monitoring Console.

  2. If you see the invitation Create an Uptime Check on the dashboard, then click it. Otherwise, select Uptime Checks > Uptime Check Overview from the left menu and then click Add Uptime Check or Create an Uptime Check. You see the New Uptime Check panel:

    Create an uptime check

  3. Fill in the following fields for the uptime check:

    • Resource Type: Choose from the menu of available resources
    • Depending on the resource type, you might have other options.
  4. Click Test to verify your uptime check is working.

  5. Click Save. You see the following panel:

    Do you want an alerting policy

  6. Click Create Alerting Policy in the preceding panel.

  7. The Conditions section is already set up with your uptime check. You don't have to change it.

  8. In the Notifications section, click Add Notification and fill in your email address.

  9. In the Documentation section, click Add Documentation and enter: Stackdriver AWS Quickstart example.

  10. In the Name this policy section, you can accept the default Uptime Check Policy.

  11. Click Save Policy.

Creating dashboards

Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:

  1. In the top menu of Stackdriver Monitoring Console, select Dashboards > Create....

    Go to the Create Dashboard page

  2. Click Add Chart. In the Metric Type menu, select an AWS metric.

  3. Click Save.

  4. In the new dashboard, change Untitled Dashboard to Stackdriver AWS Quickstart dashboard.

Viewing your logs

Stackdriver Monitoring and Stackdriver Logging are closely integrated. In the Stackdriver Monitoring Console left-side menu, choose Logging > AWS Link.... You see the Logs Viewer for your AWS connector project, which also holds your AWS logs. Change the Logs Viewer focus to see the logs you want:

  • Select Google Project > All project_id in the first drop-down menu. You should see at least one audit log from setting up your AWS connector project:

    AWS Logs Viewer

  • If you installed the Stackdriver Monitoring agent on your supported AWS VM instances, you might see other log options.

Clean up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this quickstart:

  1. Remove your Stackdriver charts and alerts, so that you won't get errors when you shut down your VM instance. In the Stackdriver Monitoring Console:

    1. Delete your alerting policy from Alerting > Policy Overview.
    2. Delete your uptime check from Alerting > Uptime Checks.
    3. Delete your charts from Dashboards > Stackdriver AWS Quickstart example
  2. In the Stackdriver Monitoring Console, go to the Account Settings page for your Stackdriver account, stackdriver-aws-quickstart. In the Monitored projects section, remove your AWS account.

  3. In your Amazon account, delete the AWS IAM role that you created for the Quickstart.

  4. In the Google Cloud Platform Console, delete your AWS connector project and—if you created it for this Quickstart—your Stackdriver account project, stackdriver-aws-quickstart. You delete a project by selecting the project, going to the IAM & Admin > Settings page, and clicking Delete Project at the top of the page.

What's next

  • See Supported Metrics for a list of all the built-in metrics. There are over 500 metrics for Amazon AWS. If you want to create your own Stackdriver Monitoring metrics, see Custom metrics.

  • To use the Stackdriver Monitoring API, see the API reference.

  • For more information on logging and its relation to monitoring, see Stackdriver Logging.

Send feedback about...

Stackdriver Monitoring