Kubelet/cAdvisor

The Kubelet exposes metrics about itself as well as cAdvisor metrics about containers running on its node. They can be ingested by updating the OperatorConfig.

You can configure managed collection to scrape Kubelet and cAdvisor metrics by editing the OperatorConfig resource.

You can install manifests by copying them to a local file and then running kubectl apply -f FILE_NAME. You can also install manifests by using Terraform.

  1. Open the OperatorConfig resource for editing:

    kubectl -n gmp-public edit operatorconfig config

  2. Add the following collection section, shown in bold type, to the resource:

    apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
  namespace: gmp-public
  name: config
collection:
  kubeletScraping:
    interval: 30s

  3. Save the file and close the editor.

After a short time, the Kubelet metric endpoints will be scraped and the metrics become available for querying in Managed Service for Prometheus.

Kubelet scraping comes with some default filters, and you can add additional filters to the collection section of the OperatorConfig resource as shown below. The filter.matchOneOf configuration section has the same semantics as the match[] parameters for Prometheus federation.

The following example filters out metrics that start with container_foo or container_bar:

collection:
  filter:
    matchOneOf:
    - '{__name__!~"container_foo.*|container_bar.*"}'

You can also use this OperatorConfig collection filter for debugging purposes. For example, you might want to temporarily disable collection of a particular metric in your cluster without modifying any PodMonitoring resource.