Cluster notifications

This page describes how Google Kubernetes Engine (GKE) publishes cluster notifications to Cloud Logging by default, and, optionally, to Pub/Sub. These notifications have information about events relevant to your cluster configuration, such as available or ongoing upgrades, security bulletins, and end of support dates.

Overview

When certain events occur that are relevant to your GKE clusters, such as important available upgrades or security bulletins, GKE sends these notifications to Cloud Logging. To find these logs in Cloud Logging, see Viewing cluster notifications in Cloud Logging.

GKE also publishes notifications about those events as messages to Pub/Sub topics that you configure. You can receive these notifications on a Pub/Sub subscription, integrate with third-party services, and filter for the notification types you want to receive. For more information about how to set up cluster notifications with Pub/Sub, see Receive cluster notifications through Pub/Sub.

Benefits

Cluster notifications provide the following benefits:

  • You are notified when security bulletins that are specific to your clusters are issued, which provides you with accurate risk and impact information.
  • You are notified when there is a new GKE version available for your cluster, allowing you to better plan for testing and qualifications, and to help ensure a smooth and predictable upgrade process. Previously, you had to check the GKE release notes or the GKE API to discover when a new GKE version was released.
  • You are notified when either GKE or a user initiates cluster upgrades, and when the upgrade operation finishes, providing you with more visibility into the background operations of your cluster.
  • You are notified when your cluster is running a GKE minor version that is at or near the end of support.

  • You can choose whether to use Cloud Logging or Pub/Sub:

    • Cluster notifications are sent to Cloud Logging by default. You can use all the capabilities of Cloud Logging, including querying and viewing logs, and configuring log-based alerting policies.
    • Pub/Sub is highly extensible, giving you flexibility in how you process incoming notifications. For example, you could integrate with Slack to forward notifications to a Slack channel, or initiate Cloud Run functions to run custom processes. When custom processes are required (for example, orchestrating a staging to production workflow to test and certify an upgrade), you can use the notification to auto-trigger these workflows.

Types of upgrade notifications

GKE sends the following cluster notification types:

If you view cluster notifications in Cloud Logging, you can use the capabilities of Cloud Logging to filter the logs. If you use Pub/Sub, you can filter the notifications you receive so that you're notified only for relevant events.

SecurityBulletinEvent

When GKE issues a security bulletin that directly correlates to your cluster configuration or version, GKE sends a SecurityBulletinEvent notification, providing you with information about the vulnerability, the impact, and, if applicable, actions you can take.

UpgradeAvailableEvent

When a new version becomes available on a release channel, GKE sends an UpgradeAvailableEvent notification to clusters on that release channel to inform the clusters that a new version is now available. This notification provides one week of advance notice for patch versions and at least 2-4 weeks for minor versions (depending on the channel). For more information, refer to What versions are available in a channel.

For clusters not on a release channel, GKE sends UpgradeAvailableEvent notifications for all new versions that the clusters can upgrade to, including patches on the current minor version and the next minor version.

If you use Windows Server node pools, Windows version information is sent as part of the UpgradeAvailableEvent notification.

UpgradeEvent

When you or GKE initiates an upgrade, GKE sends an UpgradeEvent notification, telling you that an upgrade has begun. Ideally, you should use the UpgradeAvailableEvent notification type to be aware of the upcoming upgrade so that you can either upgrade in advance or take necessary measures to prepare, such as setting up maintenance windows.

The UpgradeEvent notification is sent at the start of the upgrade operation. The operation ID is passed in the message.

UpgradeInfoEvent

GKE sends an UpgradeInfoEvent notification for different types of events, which are outlined in the next sections.

For more information about filtering for a specific type of UpgradeInfoEvent, see Filter UpgradeInfoEvent cluster notifications.

Upgrade operation is complete

When GKE finishes the operation to automatically or manually upgrade a cluster's control plane or nodes, GKE sends a notification to inform you that the operation is complete. The operation completes with one of the following states:

  • SUCCEEDED: GKE successfully upgraded the control plane or nodes.
  • FAILED: GKE failed to upgrade the control plane or nodes.
  • CANCELED: GKE canceled the upgrade operation for technical or business reasons, or you canceled the upgrade operation.

Use the notification to confirm the success of an upgrade operation.

Minor version at or near the end of support

When your cluster runs a GKE minor version that is near the end of standard support or end of extended support, or has reached either of those milestones, GKE sends notifications that you must upgrade your cluster control plane or nodes to the next supported minor version. Running a supported minor version ensures that you continue receiving security patches, bug fixes, and support. GKE sends one notification 30 days before the end of support, and one notification at the end of support, if your cluster still runs the minor version.

GKE sends cluster-level notifications, although multiple components of your cluster might be affected, and your cluster can run different minor versions at the same time. If the minor version is reaching the end of standard support and you need time to prepare for an upgrade to a supported version, you can switch to the Extended release channel to get long-term support. Otherwise, GKE schedules automatic upgrades at the end of support. These notifications help ensure that you're prepared for the enforcement of these end of support policies.

A notification includes the following details:

  • The cluster that is affected.
  • The current version that is at or near the end of support.
  • The end of support date.

For more details about the timeline of support for GKE minor versions, see the GKE minor version lifecycle.

New patch versions change to new Container-Optimized OS milestone during extended support

When your cluster is enrolled in the Extended channel during the extended support period, and the Container-Optimized OS milestone that's used by the GKE minor version reaches the end of support before the minor version, GKE sends out a cluster notification. GKE sends this notification when the first patch version to use the new milestone becomes available in the Extended channel.

This notification includes the following details:

  • The cluster that is affected.
  • The patch version which uses the new milestone.
  • The existing and new milestones.
  • How GKE pauses automatic patch upgrades for the nodes.

The patch version which uses the new milestone eventually becomes the patch auto-upgrade target for the cluster, and node auto-upgrades are paused. Cluster administrators must decide which of the following next steps to take:

  • Manually upgrade the cluster nodes to the next patch version which uses the next Container-Optimized OS milestone.
  • Manually upgrade the cluster to the next minor version.
  • Forgo patch upgrades until GKE upgrades the cluster to the next minor version towards the end of extended support.

To learn more, see The Container-Optimized OS milestone reaches the end of support before the minor version end of extended support.

Viewing cluster notifications in Cloud Logging

To view logs for GKE clusters, see Accessing your logs.

To opt out of storing these logs, you can configure an exclusion filter.

View logs in Cloud Logging with the following filter to see all types of cluster notifications:

logName=projects/PROJECT_ID/logs/container.googleapis.com%2Fnotifications

Replace PROJECT_ID with your Google Cloud project ID.

View logs with the following filter to see a specific cluster notification type, such as UpgradeEvent:

jsonPayload.@type=type.googleapis.com/google.container.v1beta1.NOTIFICATION_TYPE

Replace NOTIFICATION_TYPE with the cluster notification type for whichever logs you want to see.

Filter UpgradeInfoEvent cluster notifications

View logs with the following filter to see a specific UpgradeInfoEvent, such as the notification for where an upgrade operation is complete:

jsonPayload.@type=type.googleapis.com/google.container.v1beta1.UpgradeInfoEvent
jsonPayload.eventType=EVENT_TYPE

Replace EVENT_TYPE with one of the following:

Filtering notifications to Pub/Sub

You can filter cluster notifications to ensure that you receive only the notifications that you want in Pub/Sub. You can apply filtering for notifications to Pub/Sub in one of the following ways:

To view and filter notifications in Cloud Logging, see Viewing cluster notifications in Cloud Logging.

Filtering notifications to Pub/Sub in GKE

You can set up filtering to Pub/Sub for one or more available notification types when enabling cluster notifications by specifying values for filter in the --notification-config flag. filter takes a pipe ( | ) delimited list of the notification types you want to receive.

For example, specifying filter="UpgradeEvent|SecurityBulletinEvent" tells GKE to only send notifications for UpgradeEvent and SecurityBulletinEvent notification types.

Filtering notifications using filter has benefits such as the following:

  • Easier to use, because you filter on the notification type without using a specific syntax.
  • Notifications you filter out are never sent to Pub/Sub, so you aren't charged fees for undelivered messages.
  • You can edit the filter configuration at any time.

For instructions on filtering notifications in GKE, see Receive cluster notifications through Pub/Sub.

Filtering notifications in GKE doesn't affect which logs appear in Cloud Logging.

Filtering notifications in Pub/Sub

Pub/Sub supports filtering messages in your subscription using a filtering syntax. When you use this method, GKE delivers all notification types to your Pub/Sub topic. Pub/Sub filters messages based on your subscription configuration and delivers the messages you want to receive.

For example, you could filter for UpgradeEvent and UpgradeAvailableEvent notifications using the following syntax in your subscription:

attributes.type_url = "type.googleapis.com/google.container.v1beta1.UpgradeEvent" OR "type.googleapis.com/google.container.v1beta1.UpgradeAvailableEvent"

You are still charged for undelivered messages filtered by your subscription. You also cannot modify the filters after you've configured the subscription. However, the filtering syntax is more extensible than filtering in GKE.

To learn more about filtering your Pub/Sub subscription, see Filtering messages.

Consuming Pub/Sub messages

Pub/Sub messages contain two fields: data (string) and attributes (string-to-string map).

For GKE notifications, the data field contains human-readable information. The attributes field has generic notification information like the notification type, your project ID, cluster name, and cluster location. The attributes.payload field is a JSON-parsable string that contains specific notification information, such as the details of a security bulletin.

Notifications always contain the following attributes:

Attribute Description Example
project_id The project number that owns the cluster. 123456789
cluster_location The location of the cluster. us-central1-c
cluster_name The name of the cluster. example-cluster
type_url The type of notification. type.googleapis.com/google.container.v1beta1.UpgradeEvent
payload A JSON-parsable string carrying notification-specific information. 
{ "resourceType":"MASTER",
  "operation":"operation-1595889094437-87b7254a",
  "operationStartTime":"2020-07-27T22:31:34.437652293Z",
  "currentVersion":"1.15.12-gke.2",
  "targetVersion":"1.15.12-gke.9"}

GKE will always send beta notification types. However, you can parse the payload to display the corresponding GA notification type if it is available.

Sample cluster notification messages

In addition to the text in the data field, each message that GKE sends to Cloud Logging or Pub/Sub has specific values in the attributes.type_url and attributes.payload fields. The following tables show you examples of the information you might receive for each notification type:

SecurityBulletinEvent

The output is similar to the following for a SecurityBulletinEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.SecurityBulletinEvent
payload 
{    "resourceTypeAffected":"RESOURCE_TYPE_CONTROLPLANE",
         "bulletinId":"GCP-2021-001",
         "cveIds":[
            "CVE-2021-3156"
         ],
         "severity":"Medium",
         "briefDescription":"A vulnerability was recently discovered in the Linux utility sudo, described in CVE-2021-3156, that may allow an attacker with unprivileged local shell access on a system with sudo installed to escalate their privileges to root on the system.",
         "affectedSupportedMinors":["1.18", "1.19"],
         "patchedVersions":["1.18.9-gke.1900", "1.19.9-gke.1900"],
         "suggestedUpgradeTarget":"1.19.9-gke.1900",
         "bulletinUri":"https://cloud.google.com/anthos/clusters/docs/security-bulletins#gcp-2021-001"
      }

UpgradeAvailableEvent

The output is similar to the following for an UpgradeAvailableEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeAvailableEvent
payload 
{ "version":"1.17.15-gke.800",
  "resourceType":"MASTER",
  "releaseChannel":{"channel":"RAPID"},
  "windowsVersions": [
    {
      "imageType": "WINDOWS_SAC",
      "osVersion": "10.0.18363.1198",
      "supportEndDate": {
        "day": 10,
        "month": 5,
        "year": 2022
      }
    },
    {
      "imageType": "WINDOWS_LTSC",
      "osVersion": "10.0.17763.1577",
      "supportEndDate": {
        "day": 9,
        "month": 1,
        "year": 2024
      }
    }
  ]
}

UpgradeEvent

The output is similar to the following for an UpgradeEvent message:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeEvent
payload 
{ "resourceType":"MASTER",
  "operation":"operation-1595889094437-87b7254a",
  "operationStartTime":"2020-07-27T22:31:34.437652293Z",
  "currentVersion":"1.15.12-gke.2",
  "targetVersion":"1.15.12-gke.9"}

UpgradeInfoEvent

The output is similar to the following for an UpgradeInfoEvent message for when an upgrade operation completes, such as this example for a node pool upgrade:

Attributes
type_url type.googleapis.com/google.container.v1beta1.UpgradeInfoEvent
payload 
{ "currentVersion":"1.31.1-gke.1846000",
  "endTime":"2024-11-06T17:12:54.111640650Z",
  "operation":"operation-1730912205658-de2f88a8-6290-4718-b2c1-fb19611060b8",
  "resource":"projects/PROJECT_ID/locations/CLUSTER_LOCATION/clusters/CLUSTER_NAME/nodePools/NODE_POOL_NAME",
  "resourceType":"NODE_POOL"
  "startTime":"2024-11-06T16:56:45.658321844Z",
  "state":"SUCCEEDED",
  "targetVersion":"1.31.1-gke.2105000"}

This output is distinct from when the messages are for a minor version at or near the end of support, or when new patch versions change to new Container-Optimized OS milestone during extended support.

What's next