Version 1.7. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem). Refer to the release notes for more details. This is not the most recent version.

Application logging and monitoring

Stay organized with collections Save and categorize content based on your preferences.

This page shows how to configure a user cluster for Anthos clusters on VMware (GKE on-prem) so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.

Enabling Logging and Monitoring for user applications

The configuration for Logging and Monitoring is held in a Stackdriver object named stackdriver.

  1. Open the stackdriver object for editing:

    kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver

    Replace USER_CLUSTER_KUBECONFIG with the path of your user cluster kubeconfig file.

  2. Under spec, set enableStackdriverForApplications to true:

      kind: Stackdriver
        name: stackdriver
        namespace: kube-system
        projectID: ...
        clusterName: ...
        clusterLocation: ...
        proxyConfigSecretName: ...
        enableStackdriverForApplications: true
        enableVPC: ...
        optimizedMetrics: true
  3. Close the edited file.

Annotating workloads

To enable the collection of custom metrics from an application, add the "true" annotation to the application's Service or Deployment manifest:

Running an example application

In this section, you create an application that writes custom logs and exposes a custom metric.

  1. Save the following Service and Deployment manifests to a file named my-app.yaml. Notice that the Service has the annotation "true":

    kind: Service
    apiVersion: v1
      name: "monitoring-example"
      namespace: "default"
      annotations: "true"
        app: "monitoring-example"
        - name: http
          port: 9090
    apiVersion: apps/v1
    kind: Deployment
      name: "monitoring-example"
      namespace: "default"
        app: "monitoring-example"
      replicas: 1
          app: "monitoring-example"
            app: "monitoring-example"
          - image:
            name: prometheus-example-exporter
            imagePullPolicy: Always
            - /bin/sh
            - -c
            - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090
                cpu: 100m
  2. Create the Deployment and the Service:

    kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml

Viewing application logs


  1. Go to the Logs explorer in the Google Cloud console.

    Go to the Logs explorer

  2. Click Resource. Under ALL_RESOURCE_TYPES, select Kubernetes Container.

  3. Under CLUSTER_NAME, select the name of your user cluster.

  4. Under NAMESPACE_NAME, select default.

  5. Click Add and then click Run Query.

  6. Under Query results, you can see log entries from the monitoring-example Deployment. For example:

      "textPayload": "2020/11/14 01:24:24 Starting to listen on :9090\n",
      "insertId": "1oa4vhg3qfxidt",
      "resource": {
        "type": "k8s_container",
        "labels": {
          "pod_name": "monitoring-example-7685d96496-xqfsf",
          "cluster_name": ...,
          "namespace_name": "default",
          "project_id": ...,
          "location": "us-west1",
          "container_name": "prometheus-example-exporter"
      "timestamp": "2020-11-14T01:24:24.358600252Z",
      "labels": {
        "k8s-pod/pod-template-hash": "7685d96496",
        "k8s-pod/app": "monitoring-example"
      "logName": "projects/.../logs/stdout",
      "receiveTimestamp": "2020-11-14T01:24:39.562864735Z"


  1. Run this command:

    gcloud logging read 'resource.labels.project_id="PROJECT_ID" AND \
        resource.type="k8s_container" AND resource.labels.namespace_name="default"'

    Replace PROJECT_ID with the ID of your logging-monitoring project.

  2. In the output, you can see log entries from the monitoring-example Deployment. For example:

    insertId: 1oa4vhg3qfxidt
      k8s-pod/app: monitoring-example
      k8s- pod/pod-template-hash: 7685d96496
    logName: projects/.../logs/stdout
    receiveTimestamp: '2020-11-14T01:24:39.562864735Z'
        cluster_name: ...
        container_name: prometheus-example-exporter
        location: us-west1
        namespace_name: default
        pod_name: monitoring-example-7685d96496-xqfsf
        project_id: ...
      type: k8s_container
    textPayload: |
      2020/11/14 01:24:24 Starting to listen on :9090
    timestamp: '2020-11-14T01:24:24.358600252Z'

Viewing application metrics in the Google Cloud console

Your example application exposes a custom metric named example_monitoring_up. You can view the values of that metric in the Google Cloud console.

  1. Go to the Metrics explorer in the Google Cloud console.

    Go to the Metrics explorer

  2. For Resource type, select Kubernetes Pod.

  3. For metric, select external/prometheus/example_monitoring_up.

  4. In the chart, you can see that example_monitoring_up has a repeated value of 1.