This page shows how to configure a cluster for Google Distributed Cloud so that custom logs and metrics from user applications are sent to Cloud Logging and Cloud Monitoring.
Enabling Logging and Monitoring for user applications
To enable Logging and Monitoring for your
applications, use the
spec.clusterOperations.enableApplication
field in the cluster configuration file.
Update the cluster configuration file to set
enableApplication
totrue
:apiVersion: v1 kind: Namespace metadata: name: cluster-user-basic --- apiVersion: baremetal.cluster.gke.io/v1 kind: Cluster metadata: name: user-basic namespace: cluster-user-basic spec: type: user ... clusterOperations: projectID: project-fleet location: us-central1 enableApplication: true ...
Use
bmctl update
to apply your changes:bmctl update cluster -c CLUSTER_NAME --admin-kubeconfig=ADMIN_KUBECONFIG
Replace the following:
CLUSTER_NAME
: the name of the cluster to upgrade.ADMIN_KUBECONFIG
: the path to the admin cluster kubeconfig file.
Annotate workloads
To enable the collection of custom metrics from an application, add the
prometheus.io/scrape: "true"
annotation to the application's Service or Pod
manifest, or add the same annotation to the spec.template
section in
Deployment or DaemonSet manifest in order to be passed to their Pods.
Run an example application
In this section, you create an application that writes custom logs and exposes a custom metric.
Save the following Service and Deployment manifests to a file named
my-app.yaml
. Notice that the Service has the annotationprometheus.io/scrape: "true"
:kind: Service apiVersion: v1 metadata: name: "monitoring-example" namespace: "default" annotations: prometheus.io/scrape: "true" spec: selector: app: "monitoring-example" ports: - name: http port: 9090 --- apiVersion: apps/v1 kind: Deployment metadata: name: "monitoring-example" namespace: "default" labels: app: "monitoring-example" spec: replicas: 1 selector: matchLabels: app: "monitoring-example" template: metadata: labels: app: "monitoring-example" spec: containers: - image: gcr.io/google-samples/prometheus-dummy-exporter:latest name: prometheus-example-exporter imagePullPolicy: Always command: - /bin/sh - -c - ./prometheus-dummy-exporter --metric-name=example_monitoring_up --metric-value=1 --port=9090 resources: requests: cpu: 100m
Create the Deployment and the Service:
kubectl --kubeconfig CLUSTER_KUBECONFIG apply -f my-app.yaml
View application logs
Console
Go to the Logs explorer in the Google Cloud console.
Click Resource. Under ALL_RESOURCE_TYPES, select
Kubernetes Container
.Under CLUSTER_NAME, select the name of your user cluster.
Under NAMESPACE_NAME, select
default
.Click Add and then click Run Query.
Under Query results, you can see log entries from the
monitoring-example
Deployment. For example:{ "textPayload": "2020/11/14 01:24:24 Starting to listen on :9090\n", "insertId": "1oa4vhg3qfxidt", "resource": { "type": "k8s_container", "labels": { "pod_name": "monitoring-example-7685d96496-xqfsf", "cluster_name": ..., "namespace_name": "default", "project_id": ..., "location": "us-west1", "container_name": "prometheus-example-exporter" } }, "timestamp": "2020-11-14T01:24:24.358600252Z", "labels": { "k8s-pod/pod-template-hash": "7685d96496", "k8s-pod/app": "monitoring-example" }, "logName": "projects/.../logs/stdout", "receiveTimestamp": "2020-11-14T01:24:39.562864735Z" }
gcloud
Run this command:
gcloud logging read 'resource.labels.project_id="PROJECT_ID" AND \ resource.type="k8s_container" AND resource.labels.namespace_name="default"'
Replace
PROJECT_ID
with the ID of your project.In the output, you can see log entries from the
monitoring-example
Deployment. For example:insertId: 1oa4vhg3qfxidt labels: k8s-pod/app: monitoring-example k8s- pod/pod-template-hash: 7685d96496 logName: projects/.../logs/stdout receiveTimestamp: '2020-11-14T01:24:39.562864735Z' resource: labels: cluster_name: ... container_name: prometheus-example-exporter location: us-west1 namespace_name: default pod_name: monitoring-example-7685d96496-xqfsf project_id: ... type: k8s_container textPayload: | 2020/11/14 01:24:24 Starting to listen on :9090 timestamp: '2020-11-14T01:24:24.358600252Z'
View application metrics in the Google Cloud console
Your example application exposes a custom metric named example_monitoring_up
.
You can view the values of that metric in the Google Cloud console.
Go to the Metrics explorer in the Google Cloud console.
For Resource type, select
Kubernetes Pod
orKubernetes Container
.For metric, select
external.googleapis.com/prometheus/example_monitoring_up
.In the chart, you can see that
example_monitoring_up
has a repeated value of 1.