This page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics
from user applications are sent to Cloud Logging and Cloud Monitoring.
Enabling Logging and Monitoring for user applications
The configuration for Logging and Monitoring is
held in a Stackdriver object named stackdriver.
To enable the collection of custom metrics from an application, add the
prometheus.io/scrape: "true" annotation to the application's Service or
Deployment manifest.
To avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.
Running an example application
In this section, you create an application that writes custom logs and exposes
a custom metric.
Save the following Service and Deployment manifests to a file named
my-app.yaml. Notice that the Service has the annotation
prometheus.io/scrape: "true":
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-03 UTC."],[[["\u003cp\u003eThis guide explains how to configure a user cluster in Google Distributed Cloud to send custom application logs and metrics to Cloud Logging and Cloud Monitoring.\u003c/p\u003e\n"],["\u003cp\u003eTo enable logging and monitoring for user applications, you must edit the \u003ccode\u003estackdriver\u003c/code\u003e object within the \u003ccode\u003ekube-system\u003c/code\u003e namespace and set \u003ccode\u003eenableStackdriverForApplications\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e in the specification.\u003c/p\u003e\n"],["\u003cp\u003eTo collect custom metrics from an application, you should add the \u003ccode\u003eprometheus.io/scrape: "true"\u003c/code\u003e annotation to the application's Service or Deployment manifest.\u003c/p\u003e\n"],["\u003cp\u003eYou can view custom application logs by using the Logs explorer in the Google Cloud console or by using the \u003ccode\u003egcloud logging read\u003c/code\u003e command, filtering by the appropriate resource labels.\u003c/p\u003e\n"],["\u003cp\u003eCustom metrics, such as \u003ccode\u003eexample_monitoring_up\u003c/code\u003e, can be viewed in the Metrics explorer within the Google Cloud console by selecting the Kubernetes Pod resource type and the relevant metric.\u003c/p\u003e\n"]]],[],null,["# Application logging and monitoring\n\n\u003cbr /\u003e\n\nThis page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics\nfrom user applications are sent to Cloud Logging and Cloud Monitoring.\n\nEnabling Logging and Monitoring for user applications\n-----------------------------------------------------\n\nThe configuration for Logging and Monitoring is\nheld in a Stackdriver object named `stackdriver`.\n\n1. Open the `stackdriver` object for editing:\n\n ```\n kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver\n ```\n\n Replace \u003cvar translate=\"no\"\u003eUSER_CLUSTER_KUBECONFIG\u003c/var\u003e with the path of your user\n cluster kubeconfig file.\n2. Under `spec`, set `enableStackdriverForApplications` to `true`:\n\n ```\n apiVersion: addons.sigs.k8s.io/v1alpha1\n kind: Stackdriver\n metadata:\n name: stackdriver\n namespace: kube-system\n spec:\n projectID: ...\n clusterName: ...\n clusterLocation: ...\n proxyConfigSecretName: ...\n enableStackdriverForApplications: true\n enableVPC: ...\n optimizedMetrics: true\n ```\n3. Close the edited file.\n\nAnnotating workloads\n--------------------\n\nTo enable the collection of custom metrics from an application, add the\n`prometheus.io/scrape: \"true\"` annotation to the application's Service or\nDeployment manifest.\n\nTo avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.\n\nRunning an example application\n------------------------------\n\nIn this section, you create an application that writes custom logs and exposes\na custom metric.\n\n1. Save the following Service and Deployment manifests to a file named\n `my-app.yaml`. Notice that the Service has the annotation\n `prometheus.io/scrape: \"true\"`:\n\n ```\n kind: Service\n apiVersion: v1\n metadata:\n name: \"monitoring-example\"\n namespace: \"default\"\n annotations:\n prometheus.io/scrape: \"true\"\n spec:\n selector:\n app: \"monitoring-example\"\n ports:\n - name: http\n port: 9090\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: \"monitoring-example\"\n namespace: \"default\"\n labels:\n app: \"monitoring-example\"\n spec:\n replicas: 1\n selector:\n matchLabels:\n app: \"monitoring-example\"\n template:\n metadata:\n labels:\n app: \"monitoring-example\"\n spec:\n containers:\n - image: gcr.io/google-samples/prometheus-dummy-exporter:latest\n name: prometheus-example-exporter\n imagePullPolicy: Always\n command:\n - /bin/sh\n - -c\n - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090\n resources:\n requests:\n cpu: 100m\n ```\n2. Create the Deployment and the Service:\n\n ```\n kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml\n ```\n\nViewing application logs\n------------------------\n\n### Console\n\n1. Go to the Logs explorer in the Google Cloud console.\n\n [Go to the Logs explorer](https://console.cloud.google.com/logs/query)\n2. Click **Resource** . Under **ALL_RESOURCE_TYPES** , select `Kubernetes\n Container`.\n\n3. Under **CLUSTER_NAME**, select the name of your user cluster.\n\n4. Under **NAMESPACE_NAME** , select `default`.\n\n5. Click **Add** and then click **Run Query**.\n\n6. Under **Query results** , you can see log entries from the\n `monitoring-example` Deployment. For example:\n\n ```\n {\n \"textPayload\": \"2020/11/14 01:24:24 Starting to listen on :9090\\n\",\n \"insertId\": \"1oa4vhg3qfxidt\",\n \"resource\": {\n \"type\": \"k8s_container\",\n \"labels\": {\n \"pod_name\": \"monitoring-example-7685d96496-xqfsf\",\n \"cluster_name\": ...,\n \"namespace_name\": \"default\",\n \"project_id\": ...,\n \"location\": \"us-west1\",\n \"container_name\": \"prometheus-example-exporter\"\n }\n },\n \"timestamp\": \"2020-11-14T01:24:24.358600252Z\",\n \"labels\": {\n \"k8s-pod/pod-template-hash\": \"7685d96496\",\n \"k8s-pod/app\": \"monitoring-example\"\n },\n \"logName\": \"projects/.../logs/stdout\",\n \"receiveTimestamp\": \"2020-11-14T01:24:39.562864735Z\"\n }\n ```\n\n### gcloud\n\n1. Run this command:\n\n ```\n gcloud logging read 'resource.labels.project_id=\"PROJECT_ID\" AND \\\n resource.type=\"k8s_container\" AND resource.labels.namespace_name=\"default\"'\n ```\n\n Replace \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e with the ID of your\n [logging-monitoring project](/anthos/clusters/docs/on-prem/1.10/how-to/multiple-gcp-projects#logging_monitoring_project).\n2. In the output, you can see log entries from the `monitoring-example`\n Deployment. For example:\n\n ```\n insertId: 1oa4vhg3qfxidt\n labels:\n k8s-pod/app: monitoring-example\n k8s- pod/pod-template-hash: 7685d96496\n logName: projects/.../logs/stdout\n receiveTimestamp: '2020-11-14T01:24:39.562864735Z'\n resource:\n labels:\n cluster_name: ...\n container_name: prometheus-example-exporter\n location: us-west1\n namespace_name: default\n pod_name: monitoring-example-7685d96496-xqfsf\n project_id: ...\n type: k8s_container\n textPayload: |\n 2020/11/14 01:24:24 Starting to listen on :9090\n timestamp: '2020-11-14T01:24:24.358600252Z'\n ```\n\nViewing application metrics in the Google Cloud console\n-------------------------------------------------------\n\nYour example application exposes a custom metric named `example_monitoring_up`.\nYou can view the values of that metric in the Google Cloud console.\n\n1. Go to the Metrics explorer in the Google Cloud console.\n\n [Go to the Metrics explorer](https://console.cloud.google.com/monitoring/metrics-explorer)\n2. For **Resource type** , select `Kubernetes Pod`.\n\n3. For **metric** , select `external/prometheus/example_monitoring_up`.\n\n4. In the chart, you can see that `example_monitoring_up` has a repeated value\n of 1."]]