This page shows how to configure a user cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.
Enabling Logging and Monitoring for user applications
The configuration for Logging and Monitoring is
held in a Stackdriver object named stackdriver
.
Open the
stackdriver
object for editing:kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG --namespace kube-system edit stackdriver stackdriver
Replace USER_CLUSTER_KUBECONFIG with the path of your user cluster kubeconfig file.
Under
spec
, setenableStackdriverForApplications
totrue
:apiVersion: addons.sigs.k8s.io/v1alpha1 kind: Stackdriver metadata: name: stackdriver namespace: kube-system spec: projectID: ... clusterName: ... clusterLocation: ... proxyConfigSecretName: ... enableStackdriverForApplications: true enableVPC: ... optimizedMetrics: true
Close the edited file.
Annotating workloads
To enable the collection of custom metrics from an application, add the
prometheus.io/scrape: "true"
annotation to the application's Service or
Deployment manifest:
Running an example application
In this section, you create an application that writes custom logs and exposes a custom metric.
Save the following Service and Deployment manifests to a file named
my-app.yaml
. Notice that the Service has the annotationprometheus.io/scrape: "true"
:kind: Service apiVersion: v1 metadata: name: "monitoring-example" namespace: "default" annotations: prometheus.io/scrape: "true" spec: selector: app: "monitoring-example" ports: - name: http port: 9090 --- apiVersion: apps/v1 kind: Deployment metadata: name: "monitoring-example" namespace: "default" labels: app: "monitoring-example" spec: replicas: 1 selector: matchLabels: app: "monitoring-example" template: metadata: labels: app: "monitoring-example" spec: containers: - image: gcr.io/google-samples/prometheus-dummy-exporter:latest name: prometheus-example-exporter imagePullPolicy: Always command: - /bin/sh - -c - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090 resources: requests: cpu: 100m
Create the Deployment and the Service:
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml
Viewing application logs
Console
Go to the Logs explorer in the Google Cloud console.
Click Resource. Under ALL_RESOURCE_TYPES, select
Kubernetes Container
.Under CLUSTER_NAME, select the name of your user cluster.
Under NAMESPACE_NAME, select
default
.Click Add and then click Run Query.
Under Query results, you can see log entries from the
monitoring-example
Deployment. For example:{ "textPayload": "2020/11/14 01:24:24 Starting to listen on :9090\n", "insertId": "1oa4vhg3qfxidt", "resource": { "type": "k8s_container", "labels": { "pod_name": "monitoring-example-7685d96496-xqfsf", "cluster_name": ..., "namespace_name": "default", "project_id": ..., "location": "us-west1", "container_name": "prometheus-example-exporter" } }, "timestamp": "2020-11-14T01:24:24.358600252Z", "labels": { "k8s-pod/pod-template-hash": "7685d96496", "k8s-pod/app": "monitoring-example" }, "logName": "projects/.../logs/stdout", "receiveTimestamp": "2020-11-14T01:24:39.562864735Z" }
gcloud
Run this command:
gcloud logging read 'resource.labels.project_id="PROJECT_ID" AND \ resource.type="k8s_container" AND resource.labels.namespace_name="default"'
Replace PROJECT_ID with the ID of your logging-monitoring project.
In the output, you can see log entries from the
monitoring-example
Deployment. For example:insertId: 1oa4vhg3qfxidt labels: k8s-pod/app: monitoring-example k8s- pod/pod-template-hash: 7685d96496 logName: projects/.../logs/stdout receiveTimestamp: '2020-11-14T01:24:39.562864735Z' resource: labels: cluster_name: ... container_name: prometheus-example-exporter location: us-west1 namespace_name: default pod_name: monitoring-example-7685d96496-xqfsf project_id: ... type: k8s_container textPayload: | 2020/11/14 01:24:24 Starting to listen on :9090 timestamp: '2020-11-14T01:24:24.358600252Z'
Viewing application metrics in the Google Cloud console
Your example application exposes a custom metric named example_monitoring_up
.
You can view the values of that metric in the Google Cloud console.
Go to the Metrics explorer in the Google Cloud console.
For Resource type, select
Kubernetes Pod
.For metric, select
external/prometheus/example_monitoring_up
.In the chart, you can see that
example_monitoring_up
has a repeated value of 1.