Quickstart: Cloud Logging for Compute Engine VMs

Quickstart: Cloud Logging for Compute Engine VMs

This document shows you how to view syslog logs collected from an Apache Web Server installed on a Compute Engine virtual machine (VM) instance. You can use a process similar to the one in this quickstart to monitor third-party applications.

In this quickstart, you do the following:

1. Create a Compute Engine VM instance.
2. Install an Apache Web Server.
3. Install and configure the Ops Agent for the Apache Web Server.
4. View your logs in the Logs Explorer.
5. Create a log-based alert.
6. Clean up.

---

To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:

Guide me

---

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

You must have a Google Cloud project with billing enabled to complete this quickstart. If you don't have a Google Cloud project, or if you don't have billing enabled for your Google Cloud project, then do the following:

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Google Cloud project.

6. If you have an email notification channel configured, then skip this step.

   To set up an email notification channel, do the following:

   1. In the Google Cloud console, select Monitoring, and then select  Alerting, or click the following button:

      Go to Alerting

   2. Click Edit notification channels.
   3. In the Email section, click Add new and complete the dialog.

Create a Compute Engine VM instance

We recommend that you create a new Compute Engine VM instance for this quickstart because you install and configure software.

1. In the Google Cloud console, select Compute Engine, and then select VM instances, or click the following button:

   Go to VM instances

2. Click Create instance.

3. Fill in the fields for your instance as follows:

   • In the Name field, enter quickstart-vm.
   • In the Machine type field, select Small.
   • Ensure the Boot disk is configured for Debian GNU/Linux.
   • In the Firewall field, select both Allow HTTP traffic and Allow HTTPS traffic.

   Leave the rest of the fields at their default values.

4. Click Create. When your VM is ready, it appears in the list of instances in the Instances tab.

Install an Apache Web Server

To deploy an Apache Web Server on your Compute Engine VM instance, do the following:

1. To open a terminal to your instance, in the Connect column, click SSH.

2. To update the package lists on your instance, run the following command:

   sudo apt-get update
   

3. To install an Apache2 HTTP Server, run the following command:

   sudo apt-get install apache2 php7.0
   

   Note: If the previous command fails, then use sudo apt-get install apache2 php. If asked to continue the installation, enter Y.

4. Open your browser and connect to your Apache2 HTTP server by using the URL http://EXTERNAL_IP, where EXTERNAL_IP is the external IP address of your VM. You can find this address in the External IP column of your VM instance.

   You see the Apache2 default page:

   

Install and configure the Ops Agent

To collect logs and metrics from your Apache Web Server, install the Ops Agent by using the terminal:

1. To open a terminal to your VM instance, in the Connect column, click SSH.

2. To install the Ops Agent, run the following command:

      curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
      sudo bash add-google-cloud-ops-agent-repo.sh --also-install
   

   You see google-cloud-ops-agent installation succeeded.

3. Copy the following command, then paste it into the terminal:

   # Configures Ops Agent to collect telemetry from the app and restart Ops Agent.
   
   set -e
   
   # Create a back up of the existing file so existing configurations are not lost.
   sudo cp /etc/google-cloud-ops-agent/config.yaml /etc/google-cloud-ops-agent/config.yaml.bak
   
   # Configure the Ops Agent.
   sudo tee /etc/google-cloud-ops-agent/config.yaml > /dev/null << EOF
   metrics:
     receivers:
       apache:
         type: apache
     service:
       pipelines:
         apache:
           receivers:
             - apache
   logging:
     receivers:
       apache_access:
         type: apache_access
       apache_error:
         type: apache_error
     service:
       pipelines:
         apache:
           receivers:
             - apache_access
             - apache_error
   EOF
   
   sudo service google-cloud-ops-agent restart
   sleep 60
   

   The previous command creates the configuration to collect and ingest logs and metrics from the Apache Web Server. For more information about ingesting logs from the Apache Web Server, see Configure the Ops Agent for Apache Web Server.

View your logs

You can use the Logs Explorer to view logs in the Google Cloud console. To select certain log entries, you can run a query.

To view your logs in the Google Cloud console, do the following:

1. In the Google Cloud console, select Logging, and then select Logs Explorer, or click the following button:

   Go to the Logs Explorer

2. In the Query pane, enable Show query. The query editor opens.

3. To locate your Google Cloud project ID, expand the list of Google Cloud projects from the Google Cloud project selector. In the ID column, copy the Google Cloud project ID into your clipboard.

4. In the following expression, paste the copied ID into the PROJECT_ID field, and then copy the expression into the query editor:

   resource.type="gce_instance"
   logName=("projects/PROJECT_ID/logs/apache_access" OR
   "projects/PROJECT_ID/logs/apache_error")
   

   When the previous query is run, only apache_access and apache_error log entries are shown.

5. Click Run query or press the Enter key. The results of the query are displayed in the Query results pane.

Create a log-based alert

To be notified when a specific message appears in your log entries, create a log-based alert. In this section, you create a log-based alert so that you are notified when a 404 Not Found message appears in your log entries.

To create a log-based alert, do the following:

1. To create a log-based alert, you need to configure a notification channel. For this quickstart, create an email notification channel. See Configure email notification in the Before you begin section of this quickstart.

2. In the Google Cloud console, select Logging, and then select Logs Explorer, or click the following button:

   Go to the Logs Explorer

3. In the Query results toolbar, click Create alert. The log-based alert policy pane opens. Fill in the fields as follows:

   1. In Alert details, enter 404 Not Found in the Alert Policy Name field.

   2. In Choose logs to include in this alert, copy the following query and paste it into the log filter text box:

       severity>=DEFAULT /help httpRequest.status=404
      

      The previous log filter searches for log entries with a severity of at least DEFAULT, that contain the text /help, and that contain an httpRequest status of 404.

   3. In Set notification frequency and autoclose duration, select the following values:

      1. Time between notifications: 5 min

      2. Incident autoclose duration: 30 min

   4. In Who should be notified?, select an email from Notification Channels, and then click Save to configure the alert policy.

To test the log-based alert you just created, do the following:

1. In the Google Cloud console, select Compute Engine, and then select VM instances, or click the following button:

   Go to VM instances

2. To open a terminal to your instance, in the Connect column, click SSH.

3. To search the server for the fake page localhost/help, run the following command:

   curl localhost/help
   

   After you see a 404 Not Found message in the terminal, an email notification is sent. It might take several minutes for this process to complete.

   The email notification you receive looks similar to the following::

   

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

• If you created a new Google Cloud project for this quickstart, then delete the Google Cloud project:

  Caution: Deleting a project has the following effects:
  • Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
  • Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
  1. In the Google Cloud console, go to the Manage resources page.

     Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

• If you created a new VM in an existing project, then delete the VM instance:

  1. In the Google Cloud console, go to the VM instances page.

     Go to VM instances

  2. Select the checkbox for the instance that you want to delete.
  3. To delete the instance, click More actions, click Delete, and then follow the instructions.

• If you used an existing VM instance for this quickstart, then uninstall the Ops Agent:

  1. To uninstall the Ops Agent, open the terminal and run the following command:

     sudo bash add-google-cloud-ops-agent-repo.sh --uninstall
     

What's next

• For information about viewing logs in the Logs Explorer, see Using the Logs Explorer.

• For information about querying in the Logs Explorer, see Build queries in the Logs Explorer.

• For information about how to monitor a Compute Engine VM instance, see the Monitor a Compute Engine virtual machine quickstart.

• For more information about Monitoring and its relation to Cloud Logging, see Monitoring.