This quickstart shows you how to connect Stackdriver Monitoring to your Amazon Web Services (AWS) account. It also covers how to install the Monitoring and Logging agents on your EC2 instances.
Before you begin
You must have an AWS account that isn't currently monitored by a Workspace. You cannot monitor an AWS account from more than one Workspace.
To disconnect an AWS account from a Workspace, see Removing a project from a Workspace.
Overview of steps
The following steps connect your AWS account to Monitoring:
Create an AWS role using the Account ID and External ID.
Connect your Workspace and AWS account using the AWS Role to create a new AWS connector project.
Create a service account in the AWS connector project to authorize access to GCP.
Each of the preceding steps is described in detail in the following sections.
Configuring your Workspace
In either case, be sure to get the Account ID and External ID that you need for your AWS account.
Creating a Workspace
To create a Workspace to use with your AWS account, do the following:
Go to the Monitoring console:
If you aren't asked to create a Workspace immediately, from the Workspace drop-down list, select Create Workspace:
On the Create your free Workspace page, complete the following steps:
In the Google Cloud Platform Project drop-down list, select New Project and enter a project name, such as
Click Create Workspace. There is a pause while Stackdriver creates the new GCP project.
On the Add Google Cloud Platform projects to monitor page, click Continue, because you aren't adding any GCP projects.
On the Monitor AWS accounts page, save the values for Account ID and External ID that are unique to your new Workspace.
To finish creating your Workspace, click Skip AWS Setup.
You can skip through the following pages until you see Gathering information and then Finished initial collection!
Creating an AWS role
To create your AWS role needed to authorize Stackdriver, you must have the Account ID and External ID for your Workspace. If you don't have them, follow the instructions in Connecting an AWS account.
To create the AWS role, do the following:
- Log into your AWS IAM console and click Roles in the left-side menu.
Click Create New Role and do the following:
- For the Role type, select Another AWS account.
- In the Account ID field, enter the account ID provided by Stackdriver.
- Select the Require external ID checkbox.
- In the External ID field, enter the external ID provided by Stackdriver.
- Don't select Require MFA.
- Click Next: Permissions.
From the Policy name drop-down list, select ReadOnlyAccess:
Click Next: Review and fill in or verify the following information:
- In the Role name field, enter a name such as
- In the Role description field, enter anything you wish.
- In the Trusted entities field, verify it's the Account ID you entered earlier.
- In the Policies field, verify the value is ReadOnlyAccess.
- In the Role name field, enter a name such as
In the AWS IAM page, click Create Role.
On the Summary page, copy the Role ARN string so that you can give it to Stackdriver. If you don't see the summary, click the name of your role (for example, GoogleStackdriver) in the list of AWS roles.
Connecting an AWS account
To add an AWS account to an existing Workspace, do the following:
Go to the Stackdriver Monitoring console.
From the Workspace menu at the top of the page, select your Workspace. If you created a new Workspace as part of this Quickstart, then select that Workspace.
At the bottom of the Workspace menu, click Workspace Settings.
Under Settings, click Monitored accounts. The pane in the following screenshot shows that you are monitoring a single GCP project—the Workspace's hosting project. You aren't yet monitoring any AWS accounts.
Click Add AWS account. Enter the Account ID and External ID from when you created a Workspace.
Enter the following information in the form:
- In the Role ARN field, enter your Role ARN from Creating an AWS role or follow the instructions on the Add AWS account page to create the role.
In the Description of account field, enter a short description of your AWS account. The first word or two is used to create a new project ID.
Click Add AWS account. In a moment, the connection is confirmed.
AWS connector projects
When you connect to an AWS account, Monitoring creates an AWS connector project for you. The Monitored accounts page in your Workspace settings now includes the ID for this project:
Your AWS account description [YOUR_AWS_ACCOUNT_NUMBER]
Connected to [CONNECTOR_PROJECT_ID]
[YOUR_AWS_ACCOUNT_NUMBER]represents the account number for your AWS account.
[CONNECTOR_PROJECT_ID]represents the connector project where you receive logs and metrics from your AWS account and where you set up authorization for agents and other AWS applications that need to access GCP.
The connector project's ID always begins with
aws-, and the project's name always begins with
You can now close the Workspace Settings page.
Next step: Authorizing AWS applications
If you are told that your AWS account is already being monitored, do the following:
If another Workspace is monitoring your AWS account, then you must remove your AWS account from it. You cannot monitor an AWS account from more than one Workspace. To disconnect an AWS account from a Workspace, see Removing a project from a Workspace.
This message can also appear if you didn't use the correct Account ID and External ID from your present Workspace when you created your AWS Role. The External ID is unique for each Workspace.
Authorizing AWS applications
You must perform the following steps if you do any of the following:
- Run any of the Stackdriver agents on AWS VM instances.
- Use any GCP services from AWS applications.
To authorize applications running on AWS to access GCP services, you give them access to a GCP service account that has suitable GCP IAM roles.
A single service account can authorize multiple AWS VM instances and applications in the same AWS account, or you can create multiple service accounts.
Create a service account
Service accounts are managed in the GCP Console, not in the Stackdriver Monitoring console.
To create the service account, go to the IAM & Admin > Service accounts page for your connector project:
Select the AWS connector project (named
AWS Link...) for your AWS account.
Your connector project likely has no service accounts, so you are asked to create one. Click Create service account and enter the following information:
- In the Service account name field, enter
Stackdriver agent authorization.
In the Role field, add both of the following values:
- Monitoring > Monitoring Metric Writer
- Logging > Logs Writer
Select the Furnish a new private key checkbox.
For Key type, click JSON.
Clear the Enable G Suite Domain-wide Delegation checkbox.
- In the Service account name field, enter
Click Create. The service account's private-key file is downloaded to your workstation with a name such as
[PROJECT_NAME]represents the name of your GCP project.
[KEY_ID]represents the generated private key.
To make the following instructions simpler, save the location of the credentials file in the variable
CREDSon your workstation:
Add a service account to a VM instance
Copy the Stackdriver private-key credentials file on your workstation
to a location on your AWS EC2 instance. You must define the environment variable
GOOGLE_APPLICATION_CREDENTIALS to hold its path name:
From your workstation, copy the credentials file to your AWS EC2 instance and save in a file named
temp.json. In the
scpcommand, specify the path to
key.pem, your AWS scp key pair file, and provide your AWS credentials.
KEY="/path/to/key.pem" scp -i "$KEY" "$CREDS" AWS_USERNAME@AWS_HOSTNAME:temp.json
On your EC2 instance, move
$HOME/temp.json, to its final location. The following name and location is arbitrary:
GOOGLE_APPLICATION_CREDENTIALS="/etc/google/auth/application_default_credentials.json" sudo mkdir -p $(dirname $GOOGLE_APPLICATION_CREDENTIALS) sudo mv "$HOME/temp.json" "$GOOGLE_APPLICATION_CREDENTIALS"
(Optional): Restrict access to the private-key credentials for the service account.
sudo chown root:root "$GOOGLE_APPLICATION_CREDENTIALS" sudo chmod 0400 "$GOOGLE_APPLICATION_CREDENTIALS"
Make sure the environment variable
GOOGLE_APPLICATION_CREDENTIALSis visible to the agents and other applications that are authorized to use GCP. The environment variable name is understood by the standard GCP client libraries.
Install the agents
(Optional): Install the Stackdriver Monitoring and Logging agents by running the following commands on your EC2 instance:
curl -sSO https://dl.google.com/cloudagents/install-monitoring-agent.sh sudo bash install-monitoring-agent.sh curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh sudo bash install-logging-agent.sh --structured
--structuredflag lets the Logging agent send structured data to Stackdriver Logging. For more information, see Structured logging operations.
Verify that the agents are running.
ps ax | grep fluentd ps ax | grep collectd
The expected output should be similar to the following:
[PROCESS_ID] ? Sl 0:00 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd ... [PROCESS_ID] ? Ssl 0:00 /opt/stackdriver/collectd/sbin/stackdriver-collectd ...
Using Stackdriver services with AWS
This section shows you how to use Stackdriver services with your AWS account.
Create an uptime check
Uptime checks verify that your web server is accessible from locations around the world. The alerting policy controls who is notified if the uptime checks should fail.
To create an alerting policy using that check, follow these steps:
Go back to the Stackdriver Monitoring console:
If you see the invitation Create an Uptime Check, click it. Otherwise, from the left menu, go to Uptime Checks > Uptime Checks Overview and then click Add Uptime Check or Create an Uptime Check.
In the New Uptime Check window, fill in the following fields:
- In the Title field, enter
My Uptime Check.
- In the Check type drop-down list, select HTTP.
- In the Resource Type drop-down list,choose an available resource
Depending on the select resource type, you might have other additional fields.
- In the Title field, enter
To verify that your uptime check is working, click Test. If you see a
Connection error - refusedmessage, you either didn't install the Apache HTTP Server or you might have specified the HTTPS check type rather than HTTP. For other errors, see Verify your uptime check.
Create an alerting policy
In the Uptime Check Created pane, click Create Alerting Policy:
In the Untitled Condition field, enter a title for the alert policy condition. All other fields in the conditions pane are automatically populated from the uptime check you created.
In the Notification Channel Type drop-down list, select Email.
Enter your email address and then click Add Notification Channel.
In the Name this policy pane, enter
My Uptime Check Policy.
Click Save. You see a summary of the policy.
Create a dashboard and chart
To display the metrics collected by Monitoring in your own charts and dashboards, complete the following steps:
In the Stackdriver Monitoring console, go to Dashboards > Create Dashboard.
In the upper right-hand corner, click Add Chart.
Click the Metric tab:
Under the heading Find resource type and metric, click the textbox and select an AWS metric.
In the new dashboard, change
AWS Quickstart dashboard.
View your logs
Monitoring and Logging are closely integrated.
- In the Stackdriver Monitoring console left-side menu, go to Logging > AWS Link.
The Logs Viewer for your AWS connector project, containes your AWS logs. To change the Logs Viewer focus to see the logs you want:
Go to Google Project > All project_id You should see at least one audit log from setting up your AWS connector project:
If you installed the Stackdriver Monitoring agent on your supported AWS VM instances, you might see other log options.
To avoid incurring charges to your GCP account for the resources used in this quickstart:
Remove your Stackdriver charts and alerts. In the Stackdriver Monitoring console:
- Delete your alerting policy from Alerting > Policy Overview.
- Delete your uptime check from Alerting > Uptime Checks.
- Delete your charts from Dashboards > AWS Quickstart example
In the Stackdriver Monitoring console, go to the Workspace Settings page for your Workspace,
aws-quickstart. In the Monitored accounts section, remove your AWS account.
In your Amazon account, delete the AWS IAM role that you created for the quickstart.
In the Google Cloud Platform Console, delete your AWS connector project and—if you created it for this quickstart—your GCP project,
aws-quickstart. To delete a project, you select the project, go to IAM & Admin >Settings, and then at the top of the page click Delete Project.
To use the Monitoring API, see the API reference.
For more information on logging and its relation to monitoring, see Logging.