Migrating to Stackdriver Kubernetes Engine Monitoring

There are two options for Stackdriver support in Google Kubernetes Engine (GKE). They are provided by all GKE versions available for new clusters and for updates to existing clusters:

This page explains the differences between these two options and what you must change to migrate from Legacy Stackdriver to Stackdriver Kubernetes Engine Monitoring.

When do I need to migrate?

You can migrate your existing Stackdriver configurations from Legacy Stackdriver to Stackdriver Kubernetes Engine Monitoring at any time. However, keep in mind that a future release of GKE might withdraw support for Legacy Stackdriver. If support for Legacy Stackdriver is removed, then you must migrate to Stackdriver Kubernetes Engine Monitoring before then if you want to continue to use Stackdriver support.

The following table summarizes the expected upcoming GKE release versions with their Stackdriver support options:

GKE version Legacy Stackdriver Stackdriver Kubernetes Engine Monitoring
1.10 – 1.12.5 Default Opt-in (Beta)
1.12.7 Default Optional
1.13 Default Optional
1.14 Optional Default
1.15 Not Available Default

What is changing?

Stackdriver Kubernetes Engine Monitoring uses a different data model to organize its metrics, logs, and metadata. Here are some specific changes for your clusters using Stackdriver Kubernetes Engine Monitoring:

  • Navigation change: Stackdriver Monitoring dashboards are in a new menu named Resources > Kubernetes Engine New. This dashboard doesn't appear if you don't have clusters using Stackdriver Kubernetes Engine Monitoring.

  • Monitored resource type names changes: For example, your Kubernetes nodes are listed under the monitored resource type k8s_node , which is a Kubernetes Node, rather than gce_instance (Compute Engine VM instance).

  • Kubernetes metric names changes: In Stackdriver Kubernetes Engine Monitoring, metric type names now start with the prefix kubernetes.io/ (Stackdriver Kubernetes metrics) rather than container.googleapis.com/.

The following table summarizes the preceding changes:

Change (Old) Legacy Stackdriver (New) Stackdriver Kubernetes Engine Monitoring
Dashboard menus Resources > Kubernetes Engine Resources > Kubernetes Engine New
Metric prefixes container.googleapis.com kubernetes.io
Resource types gke_container (Metrics)
container (Logs)
gce_instance
(none)
gke_cluster
k8s_container
k8s_container
k8s_node
k8s_pod
k8s_cluster

What do I need to do?

This section contains more specific information on the data model changes in Stackdriver Kubernetes Engine Monitoring and their impact on your existing Stackdriver configurations.

Using the migration status dashboard

To identify the Stackdriver configurations that you must update as part of the migration to Stackdriver Kubernetes Engine Monitoring, do the following:

  1. From the Cloud Console, go to Stackdriver > Monitoring:

    Go to Monitoring

  2. Select the Workspace containing the GKE clusters that you want to migrate to Stackdriver Kubernetes Engine Monitoring.

  3. Select Workspace Settings > Kubernetes Migration Status in the project selection dropdown menu.

The following sample dashboard shows nothing remaining to do:

Display of the Stackdriver migration dashboard.

Resource type changes

Stackdriver Kubernetes Engine Monitoring has new resource type names, new resource type display names, and new names for the labels that identify specific resources. These changes are listed in the following table.

Resource type changes
(Old) Legacy Stackdriver resource type (New) Stackdriver Kubernetes Engine Monitoring resource type
Table footnotes:
1 In the new resource type used for monitoring (only), instance_id becomes node_name in metadata.system_labels.
2 zone refers to the location of this container or instance. location refers to the location of the cluster master node.
3 metadata.system_labels.node_name is not available in k8s_container resource types used for logging. You cannot search by node name for logs.
4 The gce_instance resource type can represent Kubernetes nodes as well as non-Kubernetes VM instances. When upgrading to Stackdriver Kubernetes Engine Monitoring, node-related uses are changed to use the new resource type, k8s_node, including node-level logs with the following names: kubelet, docker, kube-proxy, startupscript, and node-problem-detector.
5 The k8s_pod and k8s_cluster nodes might include logs not present in the Legacy Stackdriver support.
Monitoring only:
gke_container (GKE Container)

Labels:
  cluster_name
  container_name
  instance_id1
  namespace_id
  pod_id
  project_id
  zone2

Logging only:
container (GKE Container)

Labels:
  cluster_name
  container_name
  instance_id1
  namespace_id
  pod_id
  project_id
  zone2

Monitoring and Logging:
k8s_container (Kubernetes Container)

Labels:
  cluster_name
  container_name
  metadata.system_labels.node_name3
  namespace_name
  pod_name
  project_id
  location2

Logging only::
gce_instance (Compute Engine VM Instance)4

Labels:
  cluster_name
  instance_id
  project_id
  zone2
Monitoring and Logging
k8s_node4 (Kubernetes Node)

Labels:
  cluster_name
  node_name
  project_id
  location2
 
(none)
Monitoring and Logging:
k8s_pod5 (Kubernetes Pod)

Labels:
  cluster_name
  namespace_name
  pod_name
  project_id
  location2

Logging only
gke_cluster (GKE_cluster)

Labels:
  cluster_name
  project_id
  location

Monitoring and Logging:
k8s_cluster5 (Kubernetes Cluster)

Labels:
  cluster_name
  project_id
  location

Metric name changes

The following table shows some samples of the different metric names. You must change every use of a metric whose name begins with container.googleapis.com/ to a new metric whose name begins with kubernetes.io/.

The new metric names might differ in other ways besides the new prefix. Look for the new metrics in kubernetes.io (Stackdriver Kubernetes metrics).

Metric name changes
(Old) Legacy Stackdriver metrics (New) Stackdriver Kubernetes Engine Monitoring metrics
Legacy GKE metrics
container.googleapis.com/

Examples:
  .../container/cpu/utilization
  .../container/uptime
  .../container/memory/bytes_total
Stackdriver Kubernetes metrics
kubernetes.io/

Examples:
  .../container/cpu/request_utilization
  .../container/uptime
  .../node/memory/total_bytes
  .../node/cpu/total_cores

Resource group changes

If you define your own resource groups and use any of the Legacy Stackdriver resource types shown in the preceding Resource type changes table or any Legacy Stackdriver metrics shown in the preceding Metric name changes table, then change those types and metrics to be the corresponding Stackdriver Kubernetes Engine Monitoring resource types and metrics. If your resource group includes custom charts, you might have to change them.

Custom chart and dashboard changes

If you define your own custom charts and dashboards, and use any of the Legacy Stackdriver resource types shown in the preceding Resource type changes table or any Legacy Stackdriver metrics shown in the preceding Metric name changes table, then change those types and metrics to the corresponding Stackdriver Kubernetes Engine Monitoring types and metrics.

For your custom charts and dashboards, you can get help by viewing the GKE migration status dashboard:

  1. Go to Monitoring and select a Workspace containing a GKE cluster to update to Stackdriver Kubernetes Engine Monitoring:

    Go to Monitoring

  2. To view the current migration status, go to Workspace Settings > Kubernetes Migration Status.

Alerting and uptime policy changes

If you define alerting policies or uptime checks, and use any of the Legacy Stackdriver resource types shown in the preceding Resource type changes table or any Legacy Stackdriver metrics shown in the preceding Metric name changes table, then change those types and metrics to the corresponding Stackdriver Kubernetes Engine Monitoring types and metrics.

Upgrading alerting policies and uptime checks can be the most difficult changes to perform and verify. One question to consider is when to make these changes. Do you change your policy configuration before upgrading your cluster or afterwards? Old policies fail after you update your cluster and new policies fail before you update your cluster.

Instead of changing policies in place, consider leaving the existing policies unchanged and creating new policies with the updated changes. This might make it easier to keep track of which policies you expect to fail and which you don't at different times during the update.

Here are some other tips:

  • Examine your policies and estimate how long your updated cluster will have to run before you've accumulated enough data for it to be operating in its steady state.

  • Have some idea of how your policies or individual metrics are performing before you update, so you can compare that behavior with the post-update behavior.

Logging queries

If you use queries to find and filter your logs in Stackdriver Logging, and you use any of the Legacy Stackdriver resource types shown in the preceding Resource type changes table, then change those types to the corresponding Stackdriver Kubernetes Engine Monitoring types.

Logs-based metrics

If you define your own logs-based metrics and use Legacy Stackdriver metrics or resource types shown in the previous Metric name changes or Resource type changes tables, then change those metrics and resource types to the corresponding Stackdriver Kubernetes Engine Monitoring ones.

Logs exports and exclusions

If you export or exclude any of your logs, and if your export or exclusion filters use Legacy Stackdriver resource types shown in the previous Resource type changes table, then change your export and exclusion filters to use the corresponding Stackdriver Kubernetes Engine Monitoring resource types.

Changes in log entry contents

When you update to Stackdriver Kubernetes Engine Monitoring, you might find that certain information in log entries has moved to differently-named fields. This information can appear in logs queries used in logs-based metrics, log sinks, and log exclusions.

The following table, Log entry changes, lists the new fields and labels. Here's a brief summary:

  • The logName field might change. Stackdriver Kubernetes Engine Monitoring log entries use stdout or stderr in their log names whereas Legacy Stackdriver used a wider variety of names, including the container name. The container name is still available as a resource label.
  • Check the labels field in the log entries. This field might contain information formerly in the metadata log entry fields.
  • Check the resource.labels field in the log entries. The new resource types have additional label values.
Log entry changes
(Old) Legacy Stackdriver log entries (New) Stackdriver Kubernetes Engine Monitoring log entries
Table footnotes:
1 Resource labels identify specific resources that yield metrics, such as specific clusters and nodes.
2 The labels field appears in new log entries that are part of Stackdriver Kubernetes Engine Monitoring and occasionally in some Legacy Stackdriver log entries. In Stackdriver Kubernetes Engine Monitoring, it is used to hold some information formerly in the metadata log entry fields.
Log entry resources
resource.labels (Resource labels1)
Log entry resources
resource.labels (Resource labels1)
Log entry metadata
labels (Log entry labels2)

labels (Examples)
  compute.googleapis.com/resource_name:
    "fluentd-gcp-v3.2.0-d4d9p"

  container.googleapis.com/namespace_name:
    "kube-system"

  container.googleapis.com/pod_name:
    "fluentd-gcp-scaler-8b674f786-d4pq2"

  container.googleapis.com/stream:
    "stdout"
Log entry metadata
labels

Changes in log locations

In Stackdriver Logging, your logs are stored with the resource type that generated them. Since these types have changed in Stackdriver Kubernetes Engine Monitoring, be sure to look for your logs in the new resource types like Kubernetes Container, not in the Legacy Stackdriver types such as GKE Container.

What's next

Hai trovato utile questa pagina? Facci sapere cosa ne pensi:

Invia feedback per...

Stackdriver Monitoring
Hai bisogno di assistenza? Visita la nostra pagina di assistenza.