Application logging and monitoring

This page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.

Enabling Logging and Monitoring for user applications

The configuration for Logging and Monitoring is held in a Stackdriver object named stackdriver.

1. Open the stackdriver object for editing:

   kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver
   

   Replace USER_CLUSTER_KUBECONFIG with the path of your user cluster kubeconfig file.

2. Under spec, set enableStackdriverForApplications to true:

     apiVersion: addons.sigs.k8s.io/v1alpha1
     kind: Stackdriver
     metadata:
       name: stackdriver
       namespace: kube-system
     spec:
       projectID: ...
       clusterName: ...
       clusterLocation: ...
       proxyConfigSecretName: ...
       enableStackdriverForApplications: true
       enableVPC: ...
       optimizedMetrics: true
   

3. Close the edited file.

Annotating workloads

To enable the collection of custom metrics from an application, add the prometheus.io/scrape: "true" annotation to the application's Service or Deployment manifest.

To avoid garbage collection of metrics, we recommend setting the metrics scrape interval to one minute.

Running an example application

In this section, you create an application that writes custom logs and exposes a custom metric.

1. Save the following Service and Deployment manifests to a file named my-app.yaml. Notice that the Service has the annotation prometheus.io/scrape: "true":

   kind: Service
   apiVersion: v1
   metadata:
     name: "monitoring-example"
     namespace: "default"
     annotations:
       prometheus.io/scrape: "true"
   spec:
     selector:
       app: "monitoring-example"
     ports:
       - name: http
         port: 9090
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: "monitoring-example"
     namespace: "default"
     labels:
       app: "monitoring-example"
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: "monitoring-example"
     template:
       metadata:
         labels:
           app: "monitoring-example"
       spec:
         containers:
         - image: gcr.io/google-samples/prometheus-dummy-exporter:latest
           name: prometheus-example-exporter
           imagePullPolicy: Always
           command:
           - /bin/sh
           - -c
           - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090
           resources:
             requests:
               cpu: 100m
   

2. Create the Deployment and the Service:

   kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml
   

Viewing application logs

Console

1. Go to the Logs explorer in the Google Cloud console.

   Go to the Logs explorer

2. Click Resource. Under ALL_RESOURCE_TYPES, select Kubernetes Container.

3. Under CLUSTER_NAME, select the name of your user cluster.

4. Under NAMESPACE_NAME, select default.

5. Click Add and then click Run Query.

6. Under Query results, you can see log entries from the monitoring-example Deployment. For example:

   {
     "textPayload": "2020/11/14 01:24:24 Starting to listen on :9090\n",
     "insertId": "1oa4vhg3qfxidt",
     "resource": {
       "type": "k8s_container",
       "labels": {
         "pod_name": "monitoring-example-7685d96496-xqfsf",
         "cluster_name": ...,
         "namespace_name": "default",
         "project_id": ...,
         "location": "us-west1",
         "container_name": "prometheus-example-exporter"
       }
     },
     "timestamp": "2020-11-14T01:24:24.358600252Z",
     "labels": {
       "k8s-pod/pod-template-hash": "7685d96496",
       "k8s-pod/app": "monitoring-example"
     },
     "logName": "projects/.../logs/stdout",
     "receiveTimestamp": "2020-11-14T01:24:39.562864735Z"
   }
   

gcloud

1. Run this command:

   gcloud logging read 'resource.labels.project_id="PROJECT_ID" AND \
       resource.type="k8s_container" AND resource.labels.namespace_name="default"'
   

   Replace PROJECT_ID with the ID of your logging-monitoring project.

2. In the output, you can see log entries from the monitoring-example Deployment. For example:

   insertId: 1oa4vhg3qfxidt
   labels:
     k8s-pod/app: monitoring-example
     k8s- pod/pod-template-hash: 7685d96496
   logName: projects/.../logs/stdout
   receiveTimestamp: '2020-11-14T01:24:39.562864735Z'
   resource:
     labels:
       cluster_name: ...
       container_name: prometheus-example-exporter
       location: us-west1
       namespace_name: default
       pod_name: monitoring-example-7685d96496-xqfsf
       project_id: ...
     type: k8s_container
   textPayload: |
     2020/11/14 01:24:24 Starting to listen on :9090
   timestamp: '2020-11-14T01:24:24.358600252Z'
   

Viewing application metrics in the Google Cloud console

Your example application exposes a custom metric named example_monitoring_up. You can view the values of that metric in the Google Cloud console.

1. Go to the Metrics explorer in the Google Cloud console.

   Go to the Metrics explorer

2. For Resource type, select Kubernetes Pod.

3. For metric, select external/prometheus/example_monitoring_up.

4. In the chart, you can see that example_monitoring_up has a repeated value of 1.