Monitoring patch jobs

After you create a patch job, you can use Cloud Monitoring to set up alerts that you can use to monitor the patching process.

When a patch job is created, VM Manager posts the aggregate VM state to the Cloud Monitoring API. This state is posted every minute after the patch job is started. The posting of the state stops 15 minutes after the patch job completes.

To monitor your patch jobs by using alerts complete the following steps:

  1. Start a patch job. See Creating patch jobs.
  2. Set up metrics in Cloud Monitoring for your patch job.
  3. Create patch alerts.

Before you begin

  • If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init
    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init

Set up metrics

In this section, you set up tracking metrics for your patch job using Metrics Explorer. For more detailed information about using the Metrics Explorer, see Selecting metrics when using Metrics Explorer.

You can use Cloud Monitoring to track resources across multiple projects. To see metrics for patch jobs that are in separate projects, add your projects to the same monitoring workspace. For more information, see View metrics for multiple projects

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. In the Monitoring navigation pane, click Metrics Explorer.

  3. In the Resource type drop-down, select Patch Job.

  4. In the Metric drop-down, select VM instance patch state.

  5. Review the metrics.

    Cloud Monitoring running.

  6. Optional. Create a chart. For more information, see Creating a chart with Metrics Explorer.

Create patch alerts

In Cloud Monitoring, you can set up an alerting policy that provides notifications of the patch state so that you can resolve these issues in a timely manner. For more information about alerting, see Introduction to alerting.

VM instance patch states

To create the alert for patch jobs, use the VM instance patch state as the filter condition for the alert. The following VM instance patch states are available:

  • ACKED: the OS Config agent received the patch notification, but has not started patching yet
  • APPLYING_PATCHES: the OS Config agent is applying patches to the VM
  • DOWNLOADING_PATCHES: the OS Config agent is downloading patches to the VM
  • FAILED: the patch job failed
  • INACTIVE: the VM is not running
  • NO_AGENT_DETECTED: the Patch service is unable to communicate with the OS Config agent on the VM. Ensure your VMs are properly connected. For more information, see Setting up VM Manager.
  • NOTIFIED: the OS Config agent on the VM is notified, but the patch job has not started
  • PATCH_STATE_UNSPECIFIED: the state of the patch job is unknown
  • PENDING: the VM hasn't received a patch task as yet. This happens because either the patch job has recently started or the VM is awaiting a gradual rollout.
  • REBOOTING: the VM is rebooting
  • RUNNING_PRE_PATCH_STEP: the OS Config agent is running pre patch steps
  • RUNNING_POST_PATCH_STEP: the OS Config agent is running post patch steps
  • STARTED: the patch job has started on the VM
  • SUCCEEDED: the patch job completed successfully
  • SUCCEEDED_REBOOT_REQUIRED: the patch job completed successfully but a VM reboot is required
  • TIMED_OUT: the patching process timed out

Example patch alert

The following example creates a patch alert that notifies you if there are more than five failed VMs in a recent patch job.

You can create alerts by using either the Google Cloud console or the Cloud Monitoring API. The following example uses the Google Cloud console. For detailed information about creating an alert using the Google Cloud console, see Creating an alerting policy.

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. In the Monitoring navigation pane, click Alerting.

  3. Click Create Policy.

  4. Click Add Condition.

    1. In the Resource type drop-down, select Patch Job.
    2. In the Metric drop-down, select VM instance patch state.
    3. In the Filter text box, specify state=FAILED.
    4. In the Configuration section set Any time series violate = 5.
    5. Click Add.

      Setup patch alert.

  5. Click Next to advance to the notifications section.

    1. Set up notification channels.

  6. Click Next to advance to the documentation section.

    1. Set up an alert policy name
    2. Provide fix instructions

  7. Click Save.

What's next?