Quickstart for Amazon EC2

This Quickstart shows you how to monitor an Amazon EC2 virtual machine (VM) instance with Stackdriver. If instead you want to monitor a Google Compute Engine VM instance, then see Quickstart for Compute Engine.

In this Quickstart, you do the following:

  1. Create an Amazon EC2 VM instance.
  2. Install an Apache web server.
  3. Create a Stackdriver account.
  4. Install the Stackdriver monitoring and logging agents.
  5. Create a Stackdriver Monitoring uptime check with an alert policy.
  6. Create a custom dashboard.

Before you begin

  1. You must have an AWS account that is not monitored by Stackdriver. You cannot monitor an AWS account from more than one Stackdriver account.
  2. You should have some experience using EC2 instances. In this Quickstart, you create a new EC2 instance in your AWS account to monitor.

Creating the EC2 instance

The following instructions explain how to launch a t2.micro Amazon EC2 instance running Amazon Linux, edit the security group to allow HTTP connections, connect a terminal to the instance, and install an Apache web server.

Launching the instance

Create a new t2.micro EC2 instance running Amazon Linux.

Go to the Amazon EC2 console

  1. Select Launch Instance.

  2. Choose Amazon Linux AMI as your instance's OS by clicking Select.

  3. Check the box for the t2.micro instance family. Click Review and Launch.

  4. Click Launch.

  5. You see the Select an existing key pair or create a new key pair dialog:

    1. In the menu, choose Create a new key pair.

    2. In the text box, enter a key pair name. For example, my-ec2-key-pair. The dialog should look like this:

      Create a new key pair

    3. Click Download Key Pair. The key file is typically stored in your downloads directory: Downloads/my-ec2-key-pair.pem.

    4. For your convenience in this Quickstart, save your key file's location in the variable KEY. For example, on your workstation enter

      KEY="~/Downloads/aws-ec2-{some-id}.pem"
      
    5. Click Launch Instances.

  6. Click View Instances.

  7. Your new instance is listed in your Amazon EC2 console. It takes a few minutes to complete initialization. While you're waiting, give your new instance a name by filling in the Name field. Stackdriver displays this name to identify your instance.

  8. Make a note of the public DNS name for your instance, which is listed in the console. For convenience in this Quickstart, save the DNS name in the variable NAME on your workstation. For example, run

    NAME="ec2-{your-ip}.{your-zone}.compute.amazonaws.com"
    

Allowing HTTP connections

Change your instance's security group to allow HTTP connections. This is needed for Stackdriver Monitoring's uptime checks.

  1. In the Amazon EC2 console, select your new instance.

  2. Click on the security group for your new instance. The security group opens at the bottom of the page.

  3. Select the Inbound tab and click Edit.

  4. Click Add Rule.

  5. In the Type drop-down list, choose HTTP. Leave the other settings with their default values.

  6. Choose Save.

Opening a terminal to your instance

In the Amazon EC2 console, select your new instance and click the Connect button at the top of the page. You are given instructions for different methods to connect to your instance. For convenience in this Quickstart, we assume you use an ssh command already installed on your workstation. This also prepares you to use the scp command later to copy files to your EC2 instance. If you use a different method, change the instructions accordingly.

  1. On your workstation, verify you have the following:

    • Your EC2 instance's public DNS name in variable NAME.
    • Your EC2 instance's key pair file location in variable KEY.
  2. Run the following commands on your workstation:

    chmod 400 "$KEY"
    ssh -i "$KEY" "ec2-user@$NAME"
    

    You see a message from your instance:

           __|  __|_  )
           _|  (     /   Amazon Linux AMI
          ___|\___|___|
    
    https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
    10 package(s) needed for security, out of 22 available
    Run "sudo yum update" to apply all updates.
    [ec2-user@ip-{your-ip} ~]$
    

Installing the Apache web server

In this Quickstart, you install the Apache web server for the following reasons:

  • It is as an endpoint needed by Stackdriver Monitoring's uptime check facility.
  • It demonstrates Stackdriver's ability to monitor third-party applications.
  • By browsing to your instance, you cause activity on your EC2 instance that you can monitor in Stackdriver.

Use the following instructions to install and configure the web server. For more information, see LAMP tutorial for Amazon Linux.

  1. Update and install software by running the following commands on your EC2 instance:

    sudo yum update -y
    sudo yum install -y httpd24 php56
    sudo service httpd start
    sudo chkconfig httpd on
    
  2. Test your web server by browsing to your instance's public DNS name. The URL has the following format, but use your instance's actual name:

    http://ec2-{your-ip}.{your-zone}.compute.amazonaws.com
    

    You should see the following web page. If you do not, check that you changed your instance's security group.

    Apache2 default page

  3. Configure the Apache web server, because the default web page causes the error 403 (Forbidden) in Stackdriver Monitoring uptime checks:

    sudo groupadd www
    sudo usermod -a -G www ec2-user
    exit   # Exit and reconnect to get ec2-user into the new group.
    
    sudo chown -R root:www /var/www
    sudo chmod 2775 /var/www
    find /var/www -type d -exec sudo chmod 2775 {} \;
    find /var/www -type f -exec sudo chmod 0664 {} \;
    echo "Hello from EC2" > /var/www/html/index.html
    

    Browse to your instance; you should now see a page that says:

    Hello from EC2
    

Create a Stackdriver account

Stackdriver accounts are Google Cloud Platform (GCP) projects that include Stackdriver information. Go to the Stackdriver Monitoring Console at Stackdriver > Monitoring in the Google Cloud Platform Console to create a new Stackdriver account that includes a 30-day free trial of Premium service:

Go to the Stackdriver Monitoring Console

  1. If you are not asked to create a Stackdriver account immediately, then select Create Stackdriver account from the drop-down list of Stackdriver accounts at the top of the page. You see the Create a Stackdriver account page:

    Create Stackdriver account

  2. In the text box, Google Cloud Platform Project, select New Project and enter stackdriver-aws-quickstart as the project name.

    Click Create account. There is a pause while Stackdriver creates the new GCP project, which is also your Stackdriver account.

  3. When you see the page Add Google Cloud Platform projects to monitor, click Continue because you are not adding any GCP projects.

  4. When you see the page Monitor AWS accounts, follow the instructions to add an AWS account. The beginning of the page is shown below:

    Authorize AWS

  5. After filling out the information in the Monitor AWS accounts page, click Add AWS account.

    Stackdriver creates a second GCP project, named AWS LINK..., that is the "GCP project associated with your EC2 instance." For more information, see AWS Connector projects.

  6. Click Done at the bottom of the page. Wait until you see the following message:

    Finished initial collection

  7. Click Launch monitoring.

  8. In the Get Reports by Email page, select No reports and click Continue.

  9. You are now looking at the dashboard for your Stackdriver account, stackdriver-aws-quickstart.

Using the Stackdriver agents

For more information about the Stackdriver Monitoring agent, see Monitoring Agent Overview. For more details and options when installing the agent, see Installing the Monitoring Agent.

Creating a service account

You use service accounts to authorize your EC2 instance for the agent. Start in the IAM & Admin > Service accounts page of the Cloud Platform Console.

Go to the Service Accounts page

  1. Click Select a project. Choose the AWS Connector project (AWS Link...) created in the previous section. Click Open:

    Service Accounts

  2. In Service Accounts, click Create service account and enter the following information:

    Service account name: Stackdriver agent authorization
    Role: Add both Project > Editor and Logging > Logs writer
    Furnish a new private key: (checked)
    Key type: JSON
    Enable G Suite Domain-wide Delegation: (leave unchecked)

    Create service account

  3. Click Create. The service account's private-key file is downloaded to your workstation, with a name like Downloads/{project_name}-{key_id}.json. Save the location of the credentials file in variable CREDS on your workstation:

    CREDS="Downloads/{project_name}-{key_id}.json"
    

Adding credentials to your instance

Copy the Stackdriver credentials file to /etc/google/auth/application_default_credentials.json on your EC2 instance:

  1. From your workstation, copy the credentials file to a temporary file:

    scp -i "$KEY" "$CREDS" "ec2-user@$NAME:temp.json"
    
  2. On your EC2 instance, move temp.json, to its final location:

    PRIVATE_KEY_FILE="/etc/google/auth/application_default_credentials.json"
    sudo mkdir -p /etc/google/auth
    sudo mv "$HOME/temp.json" "$PRIVATE_KEY_FILE"
    sudo chown root:root "$PRIVATE_KEY_FILE"
    sudo chmod 0400 "$PRIVATE_KEY_FILE"
    

(Optional) To verify your credentials, see Verifying private-key credentials.

Installing the agents

Install the Stackdriver Monitoring and Logging agents by running the following commands on your EC2 instance:

curl -O https://repo.stackdriver.com/stack-install.sh
sudo bash stack-install.sh --write-gcm

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh

To verify that the agents are running, use the following two commands:

ps ax | grep fluentd
ps ax | grep collectd

Expected output:

{process} ?    Sl   0:00 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd ...
{process} ?    Ssl  0:00 /opt/stackdriver/collectd/sbin/stackdriver-collectd ...

Creating an uptime check and an alerting policy

Uptime checks verify that your web server is always accessible. The alerting policy controls who is notified if the uptime checks should fail:

  1. Go back to the Stackdriver Monitoring Console.

  2. If you see the invitation Create an Uptime Check on the dashboard, then click it. Otherwise, go to Alerting > Uptime Checks in the top menu and then click Add Uptime Check. You see the New Uptime Check panel:

    Create an uptime check

  3. Fill in the following fields for the uptime check:

    • Check type: HTTP
    • Resource Type: Instance
    • Applies To: Single, {your-ec2-instance-name}
    • Leave the other fields with their default values.
  4. Click Test to verify your uptime check is working. If you see a 403 (Forbidden) error, you might have to configure the Apache web server. For other errors, see Uptime checks.

  5. Click Save. You see the following panel:

    Do you want an alerting policy

  6. Click Create Alerting Policy in the preceding panel. You see the following:

    Create an alerting policy

  7. The Conditions section is already set up. You don't have to change it.

  8. In the Notifications section, click Add notification and fill in your email address.

  9. In the Documentation section, click Add Documentation and enter: Stackdriver AWS quickstart example.

  10. In the Name this policy section, you can accept the default Uptime Check Policy.

  11. Click Save Policy.

Testing the check and alert

It can take up to 15 minutes for you to see the initial successful uptime checks for your instance.

To test the alert for uptime check failures, stop your instance:

  1. Go to the Amazon EC2 console.

    Go to Amazon EC2 console.

  2. Select your instance.

  3. In the Actions menu click Instance state > Stop. It might take up to 10 minutes before Stackdriver Monitoring issues the alert.

To correct the "problem," select Instance state > Start for your instance.

Creating a dashboard and chart

Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:

  1. In the top menu of Stackdriver Monitoring Console, select Dashboards > Create....

    Go to the Create Dashboard page

  2. Click Add Chart. You see the Add Chart page:

    Add chart

  3. In the Metric Type menu, select CPU Load Average (past 5m). You see the chart data in the Preview section.

  4. Click Save.

  5. Create a second chart. Select Add Chart in the top-right of the new dashboard.

  6. In the Metric Type menu, select Network Inbound Traffic (agent).

  7. Click Save.

  8. In the new dashboard, change Untitled Dashboard to Stackdriver AWS Quickstart dashboard.

Viewing your logs

Stackdriver Monitoring and Stackdriver Logging are closely integrated. In the Stackdriver Monitoring Console left-side menu, choose Logging > AWS Link.... You see the Logs Viewer for your AWS connector project, which also holds your AWS logs. Change the Logs Viewer focus to see the logs you want:

  1. Select Amazon EC2 > All resource types in the first drop-down menu. You also notice audit logs from other services you used in connecting up Stackdriver to your AWS account.

  2. Select syslog in the second logs menu. You also notice Apache web server logs.

You see the syslog logs from your VM instance:

AWS Logs Viewer

If you scroll to the approximate time you installed the Stackdriver agents, you see the test message from fluentd, highlighted in the preceding screenshot.

Stackdriver Monitoring can also show you the logs that are relevant to a chart. Return to the Stackdriver Monitoring Console and in one of your charts open the settings menu and select View Logs.

Clean up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this quickstart:

  1. Remove your Stackdriver charts and alerts, so that you won't get errors when you shut down your VM instance. In the Stackdriver Monitoring Console:

    1. Delete your alerting policy from Alerting > Policy Overview.
    2. Delete your uptime check from Alerting > Uptime Checks.
    3. Delete your charts from Dashboards > Stackdriver AWS Quickstart example
  2. In the Stackdriver Monitoring Console, go to the Account Settings page for your Stackdriver account, stackdriver-aws-quickstart. In the Monitored projects section, remove your AWS account.

  3. In your Amazon account, do the following:

    • Terminate the AWS VM instance you created for this Quickstart.
    • Revoke the AWS IAM role, Stackdriver, that you created for the Quickstart. The role is specific to the Quickstart projects you created.
  4. In the Google Cloud Platform Console, delete your AWS connector project, AWS LINK..., and your Stackdriver account project, stackdriver-aws-quickstart. You delete a project by selecting the project, going to the IAM & Admin > Settings page, and clicking Delete Project at the top of the page.

What's next

  • See Supported Metrics for a list of all the built-in metrics. There are over 500 metrics for Amazon AWS. If you want to create your own Stackdriver Monitoring metrics, see Custom metrics.

  • To use the Stackdriver Monitoring API, see the API reference.

  • For more information on logging and its relation to monitoring, see Stackdriver Logging.

Send feedback about...

Stackdriver Monitoring