Beta Release Guide

This guide has details about the current and planned releases of Stackdriver Kubernetes Monitoring during the Beta period.

Release versions

Since the installation of Stackdriver Kubernetes Monitoring is now managed by GKE, manual installation is no longer documented.

The current releases and updates of Stackdriver Kubernetes Monitoring are listed below with the most recent release listed first:

  • Beta Update (Kubernetes 1.12.5): This update is strongly recommended for all users. To upgrade to this release, go to Installing Kubernetes Monitoring.

  • Other supported Beta versions (Kubernetes 1.10.12 / 1.11.6 / 1.11.7 / 1.11.8): To upgrade to any of these releases, go to Installing Kubernetes Monitoring.

  • Original Beta (Kubernetes 1.10.2): Due to a number of known issues, this release is not recommended for any purpose.

Features and known issues

This section lists the known issues in the current Beta release versions. The following issues remain as of April 3, 2019:

Beta update 1.12.6

  1. The metadata agent generates log spam

    The metadata agent has a known bug that generates log spam with a 400 Bad Request error. All metadata is still collected correctly, and features aren't impacted. However, this log spam consumes unnecessary log storage and may generate additional chargeable logs.

    Users should not upgrade to this release until this bug has been fixed.

    Schedule: to fix in next patch version of 1.12.6

Beta update 1.11.6

To upgrade to this release, go to Installing Kubernetes Monitoring:

  • Includes major improvements to Stackdriver Kubernetes Monitoring tabular UI loading and rendering latency.
  • Fixes most of the known issues with the 1.10.2 release.
  • Restores the ability to install the managed support for Stackdriver Kubernetes Monitoring.
  • Lets you upgrade existing clusters with Stackdriver Kubernetes Monitoring.
  • Removes the need for any manual installation on GKE.
  1. Stackdriver metadata agent restarts

    In the Beta release, the metadata agent can restart once per hour in normal operation, and might restart several times in succession.

    Fixed in 1.12.6-gke.7.

    Workaround: Do not worry about restart counts for the metadata agent.

Beta release 1.11.4

  1. Incorrect parsing of structured JSON logs as of Kubernetes 1.11.4

    Beginning with Kubernetes 1.11.4, structured JSON logs aren't parsed correctly. One effect is that alerts configured to trigger based on structured log content don't fire correctly. This can result in an alert failing to fire or firing when it shouldn't. Additionally, logs-based metrics that rely on structured log content are invalid.

    Workaround: Downgrade to GKE version 1.11.3:

    gcloud container clusters upgrade [CLUSTER_NAME] --zone [ZONE] --project [PROJECT] --master --cluster-version=1.11.3
    

    Fixed in 1.11.6-gke.2 and 1.12.4-gke.2.

Beta release 1.10.2

  1. Missing resource entries

    You might not see all your namespaces, workloads, pods, or other entities in Stackdriver Kubernetes Monitoring. This is due to a problem in gathering metadata. It can take up to 30 minutes for the missing entities to be found. They appear after refreshing the display.

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  2. Regional and multi-zonal clusters

    Entities belonging to regional and multi-zonal clusters are not displayed correctly. The Infrastructure, Workload, and Service tabs do not expand to the node/pod level for these clusters.

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  3. Severity missing in log entries

    Log entries ingested by the logging agent are missing the severity field, which causes all log entries to show up with default severity (a gray star), breaking Error Reporting integration.

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  4. Exception stack trace detection is broken

    Multi-line exception stack traces are split into multiple separate log entries, breaking Error Reporting integration.

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  5. Irrelevant error messages from agents

    The metadata agent can write irrelevant error log messages after a previous instance of the agent shuts down incorrectly. The messages are:

    terminate called after throwing an instance of 'boost::exception_ ...'
    what():  Error listening on provided port.
    terminate called recursively
    Aborted (core dumped)
    

    Also, the logging agent will log Error calling Metadata Agent after the metadata agent terminates.

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  6. Stackdriver agents don't work behind Compute Engine firewalls

    Stackdriver Kubernetes Monitoring does not work with clusters that are behind certain Compute Engine firewalls. In those situations, your metadata agent issues the following error message:

    Metadata request unsuccessful: Address family not supported by protocol.
    

    Fixed in Kubernetes versions 1.10.6 and 1.11.2. You can also use the Manual Installation steps to install newer versions of Heapster and the Stackdriver agents in a new cluster.

  7. Stackdriver metadata agent restarts

    In the Beta release, the metadata agent can restart once per hour in normal operation, and might restart several times in succession.

    Fixed in 1.12.6-gke.7.

    Workaround: Do not worry about restart counts for the metadata agent.

Was this page helpful? Let us know how we did:

Send feedback about...

Stackdriver Monitoring
Need help? Visit our support page.