Creating patch jobs

You can use OS patch management to apply operating system patches across a group of VM instances.

To apply patches to your VMs, complete the following steps:

  1. Set up your VM.
  2. Run a patch job.

Before you begin

Supported operating systems

  • Debian 9
  • Ubuntu 16.04 and 18.04
  • CentOS 6, 7, and 8
  • Red Hat Enterprise Linux (RHEL) 6, 7, and 8
  • Windows Server 2012R2, 2016, 2019, and semi-annual releases 1803 and 1809
  • SUSE Enterprise Linux Server (SLES) 12 and 15, openSUSE Leap 15

Setting up your VM

To use the OS patch management service, you need to set up the OS Config service API and install the OS Config agent. For detailed instructions, see Setting up OS Config.

Permissions

Owners of a project have full access to run and manage patch jobs. For all other users, you need to grant permissions. You can grant one of the following granular roles:

  • roles/osconfig.patchJobExecutor: Contains permissions to run, cancel, get, and list patch jobs. It also contains permissions to view instance details for a patch job.
  • roles/osconfig.patchJobViewer: Contains permissions for read-only access to get and list patch jobs. It also contains permissions to view instance details for a patch job.

For example, to grant a user access to run patch jobs, use the following command:

gcloud projects add-iam-policy-binding project-id \
    --member user:user-id@gmail.com \
    --role roles/osconfig.patchJobExecutor

Replace the following:

  • project-id: The project ID.
  • user-id: The user's G Suite username.

Running patch jobs

You can run a patch job by using either the Google Cloud Console, gcloud command-line tool, or the Cloud OS Config API.

When you run a patch job, the patching of the VMs starts simultaneously on all instances specified by the instance filter.

After you have started a patch job, you can monitor your patches using the OS patch management dashboard. It takes approximately 30 minutes after a patch job starts before the data is populated on the dashboard.

console

  1. In the Google Cloud Console, go to the OS patch management page.

    Go to the OS patch management page

  2. Click New patch deployment.
  3. In the Target VMs section, select the VM instances that you want to patch. You can either select all VM instances or use instance filters.

    For example, to run a patch job for instances that have the prefix test-, and that have the labels env=dev and app=web, your settings would resemble the following:

    OS patch management page showing instance filters.

  4. In the Patch config section, configure the patch.

    1. Specify a Name for your patch.
    2. Select the required updates for your operating system. For more information, see patch configuration.
  5. In the Scheduling section, select a schedule. To run the patch job immediately, select Start now.

  6. (Optional) In the Advanced options section, you can complete the following tasks:

  7. Click Deploy.

gcloud

Use the os-config patch-jobs execute command to run a patch job. Replace instance-filter with the instance filter that you want. For more information about instance filters, see instance filters.

gcloud compute os-config patch-jobs execute instance-filter

For more information about what updates get applied, see what is included in an OS patch job. To customize your updates, use the optional flags.

Examples

Example 1 To run a patch job with the following configurations:

  • Instance name: instance-1
  • Zone: us-east1-b
  • Description: patch for instance-1

    You would run the following command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-names="zones/us-east1-b/instances/instance-1" \
    --description "patch for instance-1"

Example 2 Suppose you want to run a patch job asynchronously with the following configurations:

  • Patch must be run on all instances in the project.
  • Patch job must timeout and stop after 1 hour and 30 minutes.
  • Machines must reboot based on the system settings after installing the updates.
  • On VMs running Apt, patching is done using apt dist-upgrade.
  • On VMs running Windows, only apply patches for the KB4339284 update.
  • On VMs running Yum, patching is done using the yum update-minimal --security utility.

You would run the following command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-all \
    --duration="1h30m" --reboot-config="DEFAULT" \
    --apt-dist --windows-exclusive-patches=KB4339284 \
    --yum-minimal --yum-security \
    --async

API

In the API, create a POST request to run a new patch job. You must explicitly define all of the required configuration fields as described in the patchJobs.execute API documentation.

For more information about what updates get applied, see what is included in an OS patch job. To customize your updates, use the PatchConfig parameters.

For example, a patch job with only the required fields looks like the following.

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter": instance-filter
}

Replace the following:

  • project-id: Your project ID.
  • instance-filter: The filter parameters that you want. For more information about instance filters, see instance filters.

Examples

Example 1 Suppose you want to run a patch job on an instance named instance1 located in us-east1-b. In this example, we add a description and specify that the job runs for 1 hour 30 minutes. Replace project-id with your project ID.

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "description":"patch instance1 in us-east1-b",
  "duration":"5400s",
  "instanceFilter":{
    "instances":[
      "zones/us-east1-b/instances/instance1"
    ]
  }
}

Example 2 The following patch job selects VMs that have the following configurations:

  • Have labels env=dev and app=web.
  • Are in either asia-east1-b or asia-east1-c.
  • Have the prefix test-.

In the following command, replace project-id with your project ID.

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter":{
    "groupLabels":[
      {
        "labels":{
          "env":"dev",
          "app":"web"
        }
      }
    ],
    "instanceNamePrefixes":[
      "test-"
    ],
    "zones":[
      "asia-east1-b",
      "asia-east1-c"
    ]
  }
}

Example 3

Suppose you want to run a patch job with the following configurations:

  • Patch must be run on all instances in the project.
  • Patch job must timeout and stop after 1 hour and 30 minutes. The API requires that time is expressed in seconds, so set this as 5400s.
  • Machines must reboot based on the system settings after installing the updates.
  • On VMs running Apt, patching is done using apt dist-upgrade.
  • On VMs running Windows, only apply patches for the KB4339284 update.
  • On VMs running Yum, patching is done using the yum update-minimal --security utility.

You would create the following request:

In the following command, replace project-id with your project ID.

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
 "duration":"5400s",
 "instanceFilter":{
   "all":true
 },
 "patchConfig":{
   "rebootConfig":"DEFAULT",
   "apt":{
     "type":"DIST"
   },
   "yum":{
     "security":true,
     "minimal":true
   },
   "windowsUpdate":{
     "exclusivePatches":"KB4339284"
   }
 }
}

Instance filters

You can specify the instances to be included in a patch job by using filters. The following filters are supported for patch jobs:

  • Filter by name: Limit patch job to instances with specific names. Instance names must be specified by using the full URI. Supported URI formats include the following:

    • zones/zone/instances/instance-name
    • projects/project-id/zones/zone/instances/instance-name
    • https://www.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/instance-name
  • Filter by name prefix: Limit patch job to instances with a specific prefix in their name.

  • Filter by zone: Limit patch job to instances in a specific zone.

  • Filter by label: Limit patch job to instances with specific labels.

You can also run patch jobs on all instances in a project by setting the all field in instanceFilter to true. For more information, see example instance filters.

Example instance filters

Scenario gcloud filter API filter
All instances in a project
--instance-filter-all
{
  "instanceFilter":{
    "all":"true"
  }
}
An instance with the name instance1 that is located in the zone us-east1-b.
--instance-filter-names="zones/us-east1-b/instances/instance1"
{
  "instanceFilter":{
    "instances":[
      "zones/us-east1-b/instances/instance1"
    ]
  }
}
Instances with the prefix app-
--instance-filter-name-prefixes="app-"
{
  "instanceFilter":{
    "instanceNamePrefixes":[
      "app-"
    ]
  }
}
Instances in zones us-east1-b or us-east1-c
--instance-filter-zones="us-east1-b","us-east1-c"
{
  "instanceFilter":{
    "zones":[
      "us-east1-b",
      "us-east1-c"
    ]
  }
}
Instances with the combination label of env=dev and app=web as well as instances with env=dev and app=worker.
--instance-filter-group-labels="env=dev,app=web"
--instance-filter-group-labels="env=dev,app=worker"
{
  "instanceFilter":{
    "groupLabels":[
      {
        "labels":{
          "env":"dev",
          "app":"web"
        }
      },
      {
        "labels":{
          "env":"dev",
          "app":"worker"
        }
      }
    ]
  }
}

Combining instance filters

Instance filters can also be combined. For example, to run a patch job for instances that have the prefix test-, that are located in the zone us-east1-c, and that have the labels env=dev and app=web, run the following command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-name-prefixes="test-" \
    --instance-filter-zones="us-east1-c" \
    --instance-filter-group-labels="env=prod,app=web"

Patch configuration

When running a patch job, you can specify parameters to control the patches that are applied on the VM. The patch configuration parameters are platform dependent and are often passed through to the underlying system update tools. The actual patches are sourced from the package repositories (Linux) or the Windows Update server (Windows) that is configured on the VM.

You can specify the following patch configurations for your VMs:

  • For Windows, you specify the classification of patches to apply (eg. Security and Critical) or target specific KBs to exclude. For more information about patch classification, see the Microsoft support documentation.
  • For RHEL and CentOS, the underlying system is yum. For patches that target these VMs, you can specify security and minimal packages. You can also exclude specific packages. For more information, see the yum man pages.
  • For Debian & Ubuntu, the underlying system is apt. For patches that target these VMs, you can specify dist-upgrade or a standard upgrade. You can also exclude specific packages. For more information, see either the Debian man pages or Ubuntu man pages.
  • For SuSE, the underlying system is zypper, specifically using zypper patches. For patches that target these VMs, you can specify options such as:

    • with update: update all packages not covered by patches
    • with optional: optional patches are treated as needed
    • The categories or severities of patches to apply

    You can also exclude specific patches.

Optionally, for all the supported operating systems, you can select to install approved patches only by specifying these updates. This allows you to enter a list of approved packages or patches. When you select these approved patches, only the approved packages or patches are installed. All other patch configuration parameters are skipped during the update.

Examples

console

  1. Follow the steps outlined in the console tab for creating either a patch job or a patch deployment.
  2. In the Patch config section, select the parameters for your patch job.

    OS patch configuration showing input fields.

  3. Make any additional configurations that are needed for your patch job or deployment.

  4. Click Deploy.

gcloud

For example, to run a patch job on all instances in the zone northamerica-northeast1-a with specific patch configurations for different operating systems, run the gcloud compute os-config patch-jobs execute command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-zones="northamerica-northeast1-a" \
    --apt-dist \
    --yum-security \
    --yum-minimal \
    --zypper-categories=security \
    --windows-classifications=critical,security \
    --reboot-config=default

To learn more about the supported options, run the following command:

gcloud compute os-config patch-jobs execute --help

API

For example, to run a patch job on all instances in the zone northamerica-northeast1-a with specific patch configurations for different operating systems, run the following command:

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute
{
    "instanceFilter":{
        "zones":[
            "northamerica-northeast1-a"
        ]
    },
    "patchConfig":{
        "apt": {
            "type": "dist-upgrade"
        },
        "yum": {
            "security": true,
            "minimal": true
        },
        "zypper": {
            "categories": ["security"]
        },
        "windowsUpdate": {
            "classifications": ["CRITICAL", "SECURITY"]
        },
        "rebootConfig": "DEFAULT"
    }
}

To learn more about the supported parameters, review the PatchConfig API documentation.

Reboot options

When running a patch job, you can specify the reboot options for the patch. The following options are available:

  • Default: The agent decides if a reboot is necessary by checking well known signals on each OS. Multiple reboots may occur during the patching and may occur before any patches are installed.
  • Always: The machine reboots after the update has completed.
  • Never: The machine does not reboot after the update has completed. In some cases, this might mean that not all patches are fully applied.

Pre-patch and post-patch scripts

When running a patch job, you can specify scripts to be run as part of the patching process. These scripts are useful for performing tasks such as shutting down an application and performing health checks.

  • Pre-patch scripts run before patching starts. If a system reboot is required before patching starts, the pre-patch script runs before the reboot.
  • Post-patch scripts run after patching completes. If a system reboot is required as part of the patching, the post-patch script runs after the reboot.

A patch job accepts one pre-patch and one post-patch script for Linux, and one pre-patch and one post-patch script for Windows. Linux and Windows scripts must be provided using the appropriate flags, parameters, or sections when specified from the gcloud command-line tool, Cloud OS Config API, or the Google Cloud Console respectively. Linux scripts run on Linux VMs only and Windows scripts run on Windows VMs only.

These script files can either be stored on the VM or in a versioned Cloud Storage bucket. If your Cloud Storage object is not publicly readable, ensure that the default Compute Engine service account for the project has the necessary IAM permissions to read Cloud Storage objects. To ensure that you have the correct permissions, check the permission settings on the Cloud Storage object.

If you want to use a Cloud Storage bucket to store your scripts, create a Cloud Storage bucket and upload your scripts to the bucket.

Examples

console

  1. Follow the steps outlined in the console tab for creating either a patch job or a patch deployment.
  2. In the Advanced options section, for both the pre-patch and post-patch sections, click Browse. A Cloud Storage object page displays.

    OS patch management page showing script upload sections.

  3. From the Cloud Storage object page, select the Cloud Storage bucket that contains the script, then select the Cloud Storage object or file.

  4. Make any additional configurations that are needed for your patch job or deployment.

  5. Click Deploy.

gcloud

For example, to run a patch job on all instances in the zone northamerica-northeast1-a with both pre and post patch script for Linux and Windows instances, run the following command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-zones="northamerica-northeast1-a" \
    --async \
    --pre-patch-linux-executable="/tmp/pre_patch_script.sh" \
    --post-patch-linux-executable="gs://my-patch-scripts/linux/post_patch_script#1523477886880" \
    --pre-patch-windows-executable="C:\\Users\\user\\pre-patch-script.cmd" \
    --post-patch-windows-executable="gs://my-patch-scripts/windows/post_patch_script.ps1#135920493447"

To learn more about acceptable file formats, run the following command:

gcloud compute os-config patch-jobs execute --help

API

For example, to run a patch job on all instances in the zone northamerica-northeast1-a with both pre and post patch script for Linux and Windows instances, run the following command:

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter":{
    "zones":[
      "northamerica-northeast1-a"
    ]
  },
  "patchConfig":{
    "preStep":{
      "linuxExecStepConfig":{
        "localPath":"/tmp/pre_patch_script.sh"
      },
      "windowsExecStepConfig":{
        "interpreter":"SHELL",
        "localPath":"C:\\Users\\user\\pre-patch-script.cmd"
      }
    },
    "postStep":{
      "linuxExecStepConfig":{
        "gcsObject":{
          "bucket":"my-patch-scripts",
          "generationNumber":"1523477886880",
          "object":"linux/post_patch_script"
        }
      },
      "windowsExecStepConfig":{
        "gcsObject":{
          "bucket":"my-patch-scripts",
          "generationNumber":"135920493447",
          "object":"windows/post_patch_script.ps1"
        },
        "interpreter":"POWERSHELL"
      }
    }
  }
}

To learn more about acceptable file formats, review the ExecStepConfig section of the PatchConfig API documentation.

Debugging a patch job

If your patch fails, you can use the following steps to help find and resolve the issues.

  1. Review the instance details for the affected patch job. This helps you to identify which instances failed or what state they are stuck in. The instance details list also contains a brief error message for each instance.

    If a patch fails with a status of NO_AGENT_DETECTED, this usually means that the service sent a request to the agent to begin patching but never heard back from the agent. Review the following possible causes and fixes:

    • The instance is not running. To fix this, start the VM instance.
    • The OS Config service is not enabled. To fix this, see Setting up your VM.
    • The OS Config agent is either not installed or not running on the instance. To fix this, see Setting up your VM.
    • The network settings on the instance didn't allow the agent to communicate with service. Check the network settings.
  2. If the instance details do not provide enough information, review the Cloud Logging logs or serial port console. The OS Config agent writes its log entries to both locations. In Cloud Logging, you can filter using the patch job ID to see all log entries related to that patch job. You can also enable debug logging by setting the osconfig-log-level=debug metadata value at the VM or project level.

What's next?