Logging and monitoring

Google Distributed Cloud includes multiple options for cluster logging and monitoring, including cloud-based managed services, open source tools, and validated compatibility with third-party commercial solutions. This document explains these options and provides some basic guidance on selecting the proper solution for your environment.

Options for Google Distributed Cloud

You have several logging and monitoring options for your Google Distributed Cloud:

  • Cloud Logging and Cloud Monitoring, enabled by in-cluster agents deployed with Google Distributed Cloud.
  • Prometheus and Grafana, disabled by default.
  • Validated configurations with third-party solutions.

Cloud Logging and Cloud Monitoring

Google Cloud Observability (formerly Stackdriver) is the built-in observability solution for Google Cloud. It offers a fully managed logging solution, metrics collection, monitoring, dashboarding, and alerting. Cloud Monitoring monitors Google Distributed Cloud clusters in a similar way as cloud-based GKE clusters.

You can configure the in-cluster agents for the scope of monitoring and logging, as well as the level of metrics collected:

  • Scope of logging and monitoring can be set to system components only (the default) or for system components and applications
  • Level of metrics collected can be configured for an optimized set of metrics or for full metrics

See Configuring logging and monitoring agents for Anthos clusters on VMware on this document for more information.

Cloud Logging and Cloud Monitoring provide an ideal solution for customers wanting a single, easy-to-configure, powerful cloud-based observability solution. We highly recommend Logging and Monitoring when running workloads only on Google Distributed Cloud, or workloads on GKE and Google Distributed Cloud. For applications with components running on Google Distributed Cloud and traditional on-premises infrastructure, you might consider other solutions for an end-to-end view of those applications.

Prometheus and Grafana

Prometheus and Grafana are two popular open source monitoring products:

  • Prometheus collects application and system metrics.

  • Alertmanager handles sending alerts out with several different alerting mechanisms.

  • Grafana is a dashboarding tool.

Prometheus and Grafana can be enabled on each admin cluster and user cluster. Prometheus and Grafana is recommended for application teams with prior experience with those products, or for operational teams who prefer to retain application metrics within the cluster and for troubleshooting issues when network connectivity is lost.

Third-party solutions

Google has worked with several third-party logging and monitoring solution providers to help their products work well with Google Distributed Cloud. These include Datadog, Elastic, and Splunk. Additional validated third parties will be added in the future.

For more information about using third-party solutions with Google Distributed Cloud, see the following:

How logging and monitoring for Google Distributed Cloud works

Logging and monitoring agents are installed and activated in each cluster when you create a new admin or user cluster. The agents collect data about system components—the scope of which you can configure.

To view the collected data on the Google Cloud console, you must configure the Google Cloud project that stores the logs and metrics you want to view.

The logging and monitoring agents on each cluster include:

  • GKE metrics agent (gke-metrics-agent). A DaemonSet that sends metrics to the Cloud Monitoring API.

  • Log Forwarder. A Fluent Bit DaemonSet that forwards logs from each machine to Cloud Logging. The Log Forwarder buffers the log entries on the node locally and resends them for up to four hours. If the buffer gets full or if the Log Forwarder can't reach the Cloud Logging API for more than four hours, then logs are dropped.

  • Global GKE metrics agent (gke-metrics-agent-global). A Deployment that sends metrics to the Cloud Monitoring API.

  • Metadata agent (stackdriver-metadata-agent). A Deployment that sends metadata for Kubernetes resources such as pods, deployments, or nodes to the Stackdriver Resource Metadata API; this data is used to enrich metric queries by enabling you to query by deployment name, node name, or even Kubernetes service name.

You can see all of the Deployment agents by running the following command:

  kubectl --kubeconfig CLUSTER_KUBECONFIG get deployments -l "managed-by=stackdriver" --all-namespaces

where CLUSTER_KUBECONFIG is the path to your kubeconfig file for the cluster.

The output of this command is similar to the following:

gke-metrics-agent-global                      1/1     Running   0   4h31m
stackdriver-metadata-agent-cluster-level      1/1     Running   0   4h31m

You can see all the DaemonSet agents by running the following command:

  kubectl --kubeconfig CLUSTER_KUBECONFIG get daemonsets -l "managed-by=stackdriver" --all-namespaces

The output of this command is similar to the following:

gke-metrics-agent                             1/1     Running   0   4h31m
stackdriver-log-forwarder                     1/1     Running   0   4h31m

Configuring logging and monitoring agents for Google Distributed Cloud

The agents installed with Google Distributed Cloud collect data about system components, subject to your settings and configuration, for the purposes of maintaining and troubleshooting issues with your clusters.

System components only (default scope)

Upon installation, agents collect logs and metrics, including performance details (for example, CPU and memory utilization) and similar metadata, for Google-provided system components. These include all workloads in the admin cluster, and for user clusters, workloads in the kube-system, gke-system, gke-connect, istio-system, and config-management-system namespaces. You can configure or disable the agents as described in the following sections.

The scope of logs and metrics collected can be expanded to include applications, as well. For instructions to enable application logging and monitoring, see Enabling Logging and Monitoring for user applications.

Optimized metrics (default metrics)

By default, the metrics agents running in the cluster collect and report an optimized set of container and kubelet metrics to Google Cloud Observability (formerly Stackdriver). Fewer resources are needed to collect this optimized set of metrics, which improves overall performance and scalability. This is especially important for container-level metrics, due to the large quantity of objects to monitor.

To disable optimized metrics (not recommended), set the optimizedMetrics field to false in your Stackdriver custom resource. For more information on changing your Stackdriver custom resource, see Configuring Stackdriver component resources. All Google Distributed Cloud metrics, including those excluded by default, are described in GKE Enterprise metrics.

Enable and disable Stackdriver

You can enable or disable logging and monitoring agents by enabling or disabling the Stackdriver custom resource. Before you disable the logging and monitoring agents, see the support page for details about how this affects Google Cloud Support's SLAs.

To disable the Stackdriver custom resource for Google Distributed Cloud:

  1. Scale down the Stackdriver operator:

    kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-system scale deploy stackdriver-operator --replicas=0

  2. Confirm that the Stackdriver operator Pod is terminated:

    kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-system get pods -l k8s-app=stackdriver-operator

  3. Delete the logging and monitoring pipeline agents. For example, run the following command to delete the log forwarder :

    kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-system delete daemonset stackdriver-log-forwarder

  4. When you are ready to redeploy Stackdriver, scale the operator back up by running:

    kubectl --kubeconfig CLUSTER_KUBECONFIG -n kube-system scale deploy stackdriver-operator --replicas=1

Logging and monitoring agents capture data stored locally, subject to your storage and retention configuration. The data is replicated to the Google Cloud project specified at installation by using a service account that is authorized to write data to that project. You can disable these agents at any time, as described earlier.

You can also manage and delete data that the logging and monitoring agents have sent to Cloud Logging and Cloud Monitoring. For more information, see Cloud Monitoring documentation.

Configuration requirements for logging and monitoring

To view Cloud Logging and Cloud Monitoring data, you must configure the Google Cloud project that stores the logs and metrics you want to view. This Google Cloud project is called your logging-monitoring project.

  1. Enable the following APIs in your logging-monitoring project:

  2. Grant following IAM roles to your logging-monitoring service account on your logging-monitoring project.

    • logging.logWriter
    • monitoring.metricWriter
    • stackdriver.resourceMetadata.writer
    • monitoring.dashboardEditor

Pricing

There is no charge for GKE Enterprise system logs and metrics. Control plane logs, control plane metrics, and a curated subset of Kube state metrics are enabled by default for GKE clusters on Google Cloud that are registered at cluster creation time in a GKE Enterprise enabled project. Control plane logs incur Cloud Logging charges, while default-on metrics are included at no additional charge.

For the list of included GKE logs and metrics, see What logs are collected and Available metrics.

In a Google Distributed Cloud cluster, GKE Enterprise system logs and metrics include the following:

  • Logs and metrics from all components in an admin cluster
  • Logs and metrics from components in these namespaces in a user cluster: kube-system, gke-system, gke-connect, knative-serving, istio-system, monitoring-system, config-management-system, gatekeeper-system, cnrm-system

For more information, see Pricing for Google Cloud Observability.

To learn about credit for Cloud Logging metrics, contact sales for pricing.

How Prometheus and Grafana for Google Distributed Cloud work

Each Google Distributed Cloud cluster is created with Prometheus and Grafana disabled by default. You can follow the installation guide to enable them.

The Prometheus Server is set up in a highly-available configuration with two replicas running on two separate nodes. Resource requirements are adjusted to support clusters running up to five nodes, with each handling up to 30 Pods that serve custom metrics. Prometheus has a dedicated PersistentVolume with disk space preallocated to fit data for a retention period of four days plus an added safety buffer.

The admin control plane, as well as each user cluster, has a dedicated monitoring stack that you can configure independently. Each admin and user cluster includes a monitoring stack that delivers a full set of features: Prometheus Server for monitoring, Grafana for observability, and Prometheus Alertmanager for alerting.

All monitoring endpoints, transferred metric data, and monitoring APIs are secured with Istio components by using mTLS and RBAC rules. Access to monitoring data is restricted only to cluster administrators.

Metrics collected by Prometheus

Prometheus collects the following metrics and metadata from the admin control plane and user clusters:

  • Resource usage, such as CPU utilization on Pods and nodes.
  • Kubernetes control plane metrics.
  • Metrics from add-ons and Kubernetes system components running on nodes, such as kubelet.
  • Cluster state, such as health of Pods in a Deployment.
  • Application metrics.
  • Machine metrics, such as network, entropy, and inodes.

Multi-cluster monitoring

The Prometheus and Grafana instance installed on the admin cluster is specially configured to provide insight across the entire Google Distributed Cloud instance, including the admin cluster and each user cluster. This enables you to:

  • Use a Grafana dashboard to access metrics from all user clusters and admin clusters.
  • View metrics from individual user clusters on Grafana dashboards; the metrics are available for direct queries in full resolution.
  • Access user clusters' node-level and workload metrics for aggregated queries, dashboards and alerting (workload metrics are limited to workloads running in the kube-system namespace).
  • Configure alerts for specific clusters.

What's next