Enable application logging and monitoring

This page shows how to configure a cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.

Enabling Logging and Monitoring for user applications

To enable Logging and Monitoring for your applications, use the spec.clusterOperations.enableApplication field in the cluster configuration file.

  1. Update the cluster configuration file to set enableApplication to true:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: cluster-user-basic
    ---
    apiVersion: baremetal.cluster.gke.io/v1
    kind: Cluster
    metadata:
      name: user-basic
      namespace: cluster-user-basic
    spec:
      type: user
      ...
      clusterOperations:
        projectID: project-fleet
        location: us-central1
        enableApplication: true
        ...
    
  2. Use bmctl update to apply your changes:

    bmctl update cluster -c CLUSTER_NAME --admin-kubeconfig=ADMIN_KUBECONFIG
    

    Replace the following:

    • CLUSTER_NAME: the name of the cluster to upgrade.
    • ADMIN_KUBECONFIG: the path to the admin cluster kubeconfig file.

Annotate workloads

To enable the collection of custom metrics from an application, add the prometheus.io/scrape: "true" annotation to the application's Service or Pod manifest, or add the same annotation to the spec.template section in Deployment or DaemonSet manifest in order to be passed to their Pods.

Run an example application

In this section, you create an application that writes custom logs and exposes a custom metric.

  1. Save the following Service and Deployment manifests to a file named my-app.yaml. Notice that the Service has the annotation prometheus.io/scrape: "true":

    kind: Service
    apiVersion: v1
    metadata:
      name: "monitoring-example"
      namespace: "default"
      annotations:
        prometheus.io/scrape: "true"
    spec:
      selector:
        app: "monitoring-example"
      ports:
        - name: http
          port: 9090
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: "monitoring-example"
      namespace: "default"
      labels:
        app: "monitoring-example"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: "monitoring-example"
      template:
        metadata:
          labels:
            app: "monitoring-example"
        spec:
          containers:
          - image: gcr.io/google-samples/prometheus-dummy-exporter:latest
            name: prometheus-example-exporter
            imagePullPolicy: Always
            command:
            - /bin/sh
            - -c
            - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090
            resources:
              requests:
                cpu: 100m
    
  2. Create the Deployment and the Service:

    kubectl --kubeconfig CLUSTER_KUBECONFIG apply -f my-app.yaml

View application logs

Console

  1. Go to the Logs explorer in the Google Cloud console.

    Go to the Logs explorer

  2. Click Resource. Under ALL_RESOURCE_TYPES, select Kubernetes Container.

  3. Under CLUSTER_NAME, select the name of your user cluster.

  4. Under NAMESPACE_NAME, select default.

  5. Click Add and then click Run Query.

  6. Under Query results, you can see log entries from the monitoring-example Deployment. For example:

    {
      "textPayload": "2020/11/14 01:24:24 Starting to listen on :9090\n",
      "insertId": "1oa4vhg3qfxidt",
      "resource": {
        "type": "k8s_container",
        "labels": {
          "pod_name": "monitoring-example-7685d96496-xqfsf",
          "cluster_name": ...,
          "namespace_name": "default",
          "project_id": ...,
          "location": "us-west1",
          "container_name": "prometheus-example-exporter"
        }
      },
      "timestamp": "2020-11-14T01:24:24.358600252Z",
      "labels": {
        "k8s-pod/pod-template-hash": "7685d96496",
        "k8s-pod/app": "monitoring-example"
      },
      "logName": "projects/.../logs/stdout",
      "receiveTimestamp": "2020-11-14T01:24:39.562864735Z"
    }
    

gcloud

  1. Run this command:

    gcloud logging read 'resource.labels.project_id="PROJECT_ID" AND \
        resource.type="k8s_container" AND resource.labels.namespace_name="default"'

    Replace PROJECT_ID with the ID of your project.

  2. In the output, you can see log entries from the monitoring-example Deployment. For example:

    insertId: 1oa4vhg3qfxidt
    labels:
      k8s-pod/app: monitoring-example
      k8s- pod/pod-template-hash: 7685d96496
    logName: projects/.../logs/stdout
    receiveTimestamp: '2020-11-14T01:24:39.562864735Z'
    resource:
      labels:
        cluster_name: ...
        container_name: prometheus-example-exporter
        location: us-west1
        namespace_name: default
        pod_name: monitoring-example-7685d96496-xqfsf
        project_id: ...
      type: k8s_container
    textPayload: |
      2020/11/14 01:24:24 Starting to listen on :9090
    timestamp: '2020-11-14T01:24:24.358600252Z'
    

View application metrics in the Google Cloud console

Your example application exposes a custom metric named example_monitoring_up. You can view the values of that metric in the Google Cloud console.

  1. Go to the Metrics explorer in the Google Cloud console.

    Go to the Metrics explorer

  2. For Resource type, select Kubernetes Pod or Kubernetes Container.

  3. For metric, select external.googleapis.com/prometheus/example_monitoring_up.

  4. In the chart, you can see that example_monitoring_up has a repeated value of 1.