Monitoring apps running on multiple GKE clusters using Prometheus and Cloud Monitoring

This tutorial shows how to deploy Prometheus servers on multiple Google Kubernetes Engine (GKE) clusters that are running in a hybrid environment. The tutorial then shows you how to collect metrics from apps running in these clusters, and send them to Cloud Monitoring for centralized observability.

Note: Prometheus is a third-party tool and not an official Google product. Google doesn't support the Prometheus server.

Prometheus is an open source monitoring and alerting toolkit that was influenced by a Google internal monitoring system: Borgmon. Borg inspired the Kubernetes open source project, and Borgmon inspired Prometheus. The tools work well with each other.

With Prometheus, you can configure scrape targets that are queried (or scraped) at configurable intervals to discover and pull in metrics from fleets of machines. Scrape targets are generally HTTP endpoints exposed from an app, using a well-defined exposition format with one metric per line. By using HTTP as the default transmission mechanism for a scrape target, you can expose metrics from many different languages and endpoints. Metrics collected from a scrape target are stored in the Prometheus time-series database.

In production environments, you export the metrics from the local Prometheus database to a more robust monitoring solution like Cloud Monitoring. Cloud Monitoring is a Google-managed monitoring and management solution for services, containers, apps, and infrastructure. Cloud Monitoring provides a robust storage solution for metrics, logs, traces, and events. It also provides a suite of observability tooling that includes dashboards, reporting, alerting, and many other features.

In this tutorial, you use the Prometheus server installed on multiple Kubernetes clusters to scrape metrics from an app (PostgreSQL). You use a prometheus-stackdriver-sidecar container, running alongside a prometheus container, to authenticate and send metrics to Monitoring, where they are stored and used for analysis and alerting. The sidecar approach shown is considered a best practice because it isolates scraping for metrics from the workload to be monitored. This approach ensures that the scraping and monitoring processes don't interfere with each other and that resources can be allocated as needed for each.

In this solution, you build the following architecture in a Google Cloud project.

Objectives

• Create two GKE clusters, with one cluster simulating a non-Google Cloud Kubernetes cluster, for example, a Kubernetes cluster running on-premises. In this tutorial, both clusters are GKE clusters.
• Install Prometheus servers in both clusters.
• Configure the prometheus-stackdriver exporter sidecar in both clusters. The sidecar is responsible for authenticating and sending metrics to Monitoring.
• Install an app in both clusters. This tutorial uses PostgreSQL as an example app; however, this approach works with any app configured to export Prometheus metrics.
• Monitor exported metrics from both clusters in Monitoring.

Costs

This tutorial uses the following billable components of Google Cloud:

• Google Kubernetes Engine
• Monitoring
• Compute Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

4. Enable the GKE and Monitoring APIs.

   Enable the APIs

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Setting up your environment

You run all the terminal commands in this tutorial from Cloud Shell.

Clone the repository

1. Open Cloud Shell:

   Open Cloud Shell

2. Download the required files for this tutorial by cloning the following repository. Make the repository folder your working folder ($WORKDIR), where you do all the tasks related to this tutorial.

   cd $HOME
   git clone https://github.com/GoogleCloudPlatform/prometheus-stackdriver-gke
   cd $HOME/prometheus-stackdriver-gke
   WORKDIR=$(pwd)
   

Install Helm, kubectx, and kubens

1. In Cloud Shell, install Helm in the $WORKDIR folder:

   HELM_VERSION=v2.13.0
   HELM_PATH="$WORKDIR"/helm-"$HELM_VERSION"
   wget https://storage.googleapis.com/kubernetes-helm/helm-"$HELM_VERSION"-linux-amd64.tar.gz
   tar -xvzf helm-"$HELM_VERSION"-linux-amd64.tar.gz
   mv linux-amd64 "$HELM_PATH"
   rm $WORKDIR/helm-"$HELM_VERSION"-linux-amd64.tar.gz
   

   Helm is a package manager you can use to configure and deploy Kubernetes apps. This installation doesn't interfere with other Helm installations that you've installed.

2. Install kubectx and kubens:

   git clone https://github.com/ahmetb/kubectx $WORKDIR/kubectx
   export PATH=$PATH:$WORKDIR/kubectx
   

   These tools facilitate working with multiple GKE clusters by making it easier to switch contexts and namespaces.

Install pgbench

In this tutorial, you use PostgreSQL as a sample app for Prometheus monitoring. In order to generate traffic to PostgreSQL so that the Prometheus server can collect metrics, you use a tool called pgbench, a simple program for running benchmark tests on PostgreSQL.

• In Cloud Shell, install pgbench:

  sudo apt-get install postgresql-contrib
  

Configure IAM

You create an Identity and Access Management (IAM) service account to delegate permissions to the Prometheus sidecar container, allowing it to write metrics data to Monitoring.

1. In Cloud Shell, create a Google Cloud service account to be used by the Prometheus Monitoring sidecar:

   gcloud iam service-accounts create prometheus --display-name prometheus-service-account
   

2. Store the service account email address and your current project ID in environment variables for use in later commands:

   export PROJECT_ID=$(gcloud info --format='value(config.project)')
   PROMETHEUS_SA_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:prometheus-service-account" \
       --format='value(email)')
   

3. Bind the monitoring.metricWriter role to the Prometheus service account:

   gcloud projects add-iam-policy-binding ${PROJECT_ID} --role roles/monitoring.metricWriter --member serviceAccount:${PROMETHEUS_SA_EMAIL}
   

   The monitoring.metricWriter role lets the Prometheus sidecar container write metrics data to Monitoring. Monitoring stores and uses these metrics for dashboards, alerts, and more.

4. Download the service account key to your working directory.

   gcloud iam service-accounts keys create $WORKDIR/prometheus-service-account.json --iam-account ${PROMETHEUS_SA_EMAIL}
   

The Prometheus sidecar container uses the service account key as a secret to authenticate to the Cloud Monitoring API.

GKE clusters don't require you to manually configure these keys. Pods (and containers) running in GKE can access the Compute Engine instance metadata and retrieve the service account information attached to the GKE nodes (or Compute Engine instances) that the Prometheus server is running on. For GKE clusters, the Prometheus sidecar container can authenticate to the Cloud Monitoring API by using the service account information from the instance metadata service.

Non-GKE clusters, such as clusters running in an on-premises data center, don't have access to Google Cloud service accounts. These clusters use a manually configured service account key to authenticate themselves to the Cloud Monitoring API.

In this tutorial, the GKE cluster called gke uses the Compute Engine instance metadata to authenticate to the Cloud Monitoring API to write metrics. The second GKE cluster, called onprem, simulates a non-GKE cluster that doesn't have access to the Monitoring API to write metrics, thus requiring you to use the service account key.

Creating and configuring GKE clusters

GKE nodes are Compute Engine managed instance groups. Every such instance has an identity. The identity of a Compute Engine instance is a Google Cloud service account that's associated with it upon its creation.

The IAM roles and permissions associated with that service account also determine the permissions associated with that instance. If you don't explicitly define a service account when you create a GKE cluster, the cluster is created using the project default service account. Pods that run in the cluster can access the node's (Compute Engine instance) metadata service and use the associated service account for access to various Google Cloud services, for instance, the Cloud Monitoring API. The default service account and the default GKE scopes have permissions to write metrics to the Cloud Monitoring API.

In this tutorial, when you create the two clusters, gke and onprem, you use the default service account. Only the gke cluster uses the Compute Engine instance metadata service and uses the default service account to write metrics to Monitoring. The onprem cluster doesn't use the default service account. Instead, it uses the Prometheus Google Cloud service account key for access to Monitoring.

Create GKE clusters

1. Create the first cluster, gke, in the us-west2-a zone, and enable Cloud Monitoring for GKE on the cluster:

   gcloud beta container clusters create gke \
       --zone us-west2-a \
       --num-nodes 3 \
       --machine-type n1-standard-2 \
       --enable-stackdriver-kubernetes \
       --verbosity=none --async
   

2. Create the second cluster, onprem, in the us-east4-a zone:

   gcloud container clusters create onprem \
       --zone us-east4-a \
       --num-nodes 3 \
       --machine-type n1-standard-2 \
       --verbosity=none
   

3. Wait a few minutes until both clusters are successfully created. Ensure that the clusters are running:

   gcloud container clusters list
   

   The output looks similar to this:

   NAME    LOCATION    MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
   onprem  us-east4-a  1.11.8-gke.6    <external IP>   n1-standard-2  1.11.8-gke.6  3          RUNNING
   gke     us-west2-a  1.11.8-gke.6    <external IP>   n1-standard-2  1.11.8-gke.6  3          RUNNING
   

Configure cluster connections and access

1. Connect to both clusters to generate entries in the kubeconfig file:

   gcloud container clusters get-credentials gke --zone us-west2-a --project ${PROJECT_ID}
   gcloud container clusters get-credentials onprem --zone us-east4-a --project ${PROJECT_ID}
   

   The kubeconfig file is used to create authentication to clusters by creating users and contexts for each cluster. Once you've created the kubeconfig file, you can quickly switch context between clusters.

2. Use kubectx to rename the context names for convenience:

   kubectx gke=gke_${PROJECT_ID}_us-west2-a_gke
   kubectx onprem=gke_${PROJECT_ID}_us-east4-a_onprem
   

3. Give your Google identity cluster-admin permission for both clusters so that you can perform administrator-level tasks on the clusters:

   kubectl create clusterrolebinding user-admin-binding \
       --clusterrole=cluster-admin \
       --user=$(gcloud config get-value account) \
             --context gke
   kubectl create clusterrolebinding user-admin-binding \
       --clusterrole=cluster-admin \
       --user=$(gcloud config get-value account) \
       --context onprem
   

Installing and configuring the Prometheus server

The workflows differ for configuring the Prometheus server on GKE and non-GKE clusters. To avoid confusion, this tutorial walks through both setups.

Install Prometheus on the gke cluster (GKE workflow)

In this section, you configure the gke cluster with Prometheus. If you are running GKE clusters, follow this workflow for every cluster.

1. In Cloud Shell, create a dedicated namespace in the gke cluster for Prometheus:

   kubectl create namespace prometheus --context gke
   

2. Create a Prometheus Kubernetes service account, a ClusterRole role, and a cluster role binding:

   cd $WORKDIR
   kubectl apply -f prometheus-service-account.yaml --context gke
   

   The cluster role gives permissions to retrieve metrics from deployments running in all namespaces. The cluster role binding assigns this role to the Prometheus service account.

3. Create the Prometheus configmap to scrape metrics from apps running in the gke cluster:

   kubectl apply -f prometheus-configmap.yaml --context gke
   

   For more details about the Prometheus configuration file, see the Prometheus documentation.

   Your next task is to create the Prometheus server deployment. The manifest in the next step creates the Prometheus deployment with a single pod. The pod is composed of two containers: the Prometheus server container and the Monitoring sidecar. The Prometheus server container collects metrics from pods in the GKE cluster that are exporting Prometheus metrics. The server uses the Monitoring sidecar container to push metrics to Monitoring.

4. Define environment variables used in the Prometheus deployment manifest:

   export KUBE_NAMESPACE=prometheus
   export KUBE_CLUSTER=gke
   export GCP_REGION=us-west2-a
   export GCP_PROJECT=$(gcloud info --format='value(config.project)')
   export DATA_DIR=/prometheus
   export DATA_VOLUME=prometheus-storage-volume
   export SIDECAR_IMAGE_TAG=release-0.3.2
   

5. Apply the Prometheus deployment manifest by using the environment variables you just defined:

   envsubst < gke-prometheus-deployment.yaml | kubectl --context gke apply -f -
   

6. Wait a few moments and confirm that the Prometheus pod is running:

   kubectl get pods -n prometheus --context gke
   

   The output looks similar to this:

   NAME                                     READY     STATUS    RESTARTS   AGE
   prometheus-deployment-6b5df7b5ff-ghqb2   2/2       Running   0          20s
   

   Note that two containers are running in the Prometheus pod.

7. Verify the container images:

   kubectl --context gke get pods -n prometheus -o json | jq '.items[].spec.containers[].image'
   

   The output looks similar to this:

   "prom/prometheus:v2.6.1"
   "gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:release-0.3.2"
   

   You see the Prometheus server and the stackdriver-prometheus-sidecar container running in the pod.

8. Set up port forwarding to the Prometheus server UI that's running in the gke cluster:

   export PROMETHEUS_POD_GKE=$(kubectl get pods --namespace prometheus -l "app=prometheus-server" \
       -o jsonpath="{.items[0].metadata.name}" \
       --context gke)
   kubectl --context gke port-forward --namespace prometheus $PROMETHEUS_POD_GKE 9090:9090 >> /dev/null &
   

   The next step is to open the Prometheus UI.

9. In Cloud Shell, click Web preview, and then click Change port.

   1. For Port Number, enter 9090.
   2. Click Change and Preview. The Prometheus server UI is displayed.

   

Inspect Prometheus on the gke cluster

Using the Prometheus Kubernetes service account, Prometheus discovers resources that are running inside a GKE cluster. Some of these resources are already configured to export Prometheus metrics.

1. In the Prometheus UI, click Status > Service Discovery. The list that's displayed shows the Kubernetes resources that Prometheus discovered.

   

   Service Discovery
   kubernetes-apiservers (1/16 active targets)
   kubernetes-cadvisor (3/3 active targets)
   kubernetes-nodes (3/3 active targets)
   kubernetes-pods (0/42 active targets)
   kubernetes-service-endpoints (0/16 active targets)
   

2. Click Status > Targets.

   Targets are the HTTP(S) endpoints defined in resources that are exporting Prometheus metrics at regular intervals. You see various Kubernetes resources that are exporting metrics, for example, the Kubernetes API server exporting metrics from the /metrics HTTPS endpoint.

Install PostgreSQL on the gke cluster

In this section, you install an app on the gke cluster. The app is instrumented to export Prometheus metrics. For this tutorial, you install PostgreSQL by using the stable Helm chart. However, any app that exports Prometheus metrics follows the same workflow.

1. Switch to the gke cluster context:

   kubectx gke
   

2. Grant Tiller, the server side of Helm, the cluster-admin role in the gke cluster:

   kubectl create serviceaccount tiller --namespace kube-system
   kubectl create clusterrolebinding tiller-admin-binding \
       --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
   

3. Initialize Helm and install Tiller in the gke cluster:

   ${HELM_PATH}/helm init --service-account=tiller
   ${HELM_PATH}/helm repo update
   

4. Ensure that Helm is properly installed by running the following command:

   ${HELM_PATH}/helm version
   

   The output looks similar to this:

   Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
   Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
   

   If Helm is correctly installed, v2.13.0 appears for both client and server. You might need to run the version command multiple times until you see both the client and server installed.

5. Install PostgreSQL using the stable Helm release:

   ${HELM_PATH}/helm install --name gke --set postgresUser=user,postgresPassword=password,postgresDatabase=postgres \
       stable/postgresql --set metrics.enabled=true \
       --set postgresqlDatabase=prometheusdb
   

   PostgreSQL takes a couple of minutes to initialize.

6. Ensure that PostgreSQL is running before you proceed:

   kubectl get pods
   

   The output looks similar to this:

   NAME               READY     STATUS    RESTARTS   AGE
   gke-postgresql-0   2/2       Running   0          1m
   

   Inspect the two containers running inside the PostgreSQL pod:

   kubectl --context gke get pods gke-postgresql-0 -ojson | jq '.spec.containers[].image'
   

   The output looks similar to this:

   "docker.io/bitnami/postgresql:10.7.0"
   "docker.io/wrouesnel/postgres_exporter:v0.4.7"
   

   The first container is the PostgreSQL container. The second container is the Prometheus metrics exporter for Postgres.

7. Inspect the services running in gke:

   kubectl get services
   

   The output looks similar to this:

   NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP
   gke-postgresql-metrics    ClusterIP   10.23.255.97    <none>        9187/TCP   1m
   

   PostgreSQL exposes Prometheus metrics using the gke-postgresql-metrics service.

8. Inspect the annotations on the gke-postgresql-metrics service:

   kubectl --context gke get service gke-postgresql-metrics -ojson | jq '.metadata.annotations'
   

   The output looks similar to this:

   {
     "prometheus.io/port": "9187",
     "prometheus.io/scrape": "true"
   }
   

   Prometheus discovers Kubernetes services by using the Kubernetes service registry. It uses Kubernetes annotations on the services to determine the Prometheus target configuration. These annotations describe Prometheus configurations such as port, target endpoints, which services to scrape, and other settings.

   In the output, you see two annotations describing the port 9187 for Prometheus metrics and scrape setting set to true. The default target endpoint is /metrics unless otherwise defined as an annotation. For a detailed list of Prometheus Kubernetes annotations and configuration, see the Prometheus documentation.

9. In the Prometheus UI window, click or refresh the Status > Targets page, scroll to the kubernetes-service-endpoints section, and click show more next to the kubernetes-service-endpoints link:

   

   Note how Prometheus is scraping metrics from the PostgreSQL pod on port 9187 and scrape target /metrics. The IP address is that of the PostgreSQL pod.

You have successfully installed a Prometheus server (with the Monitoring sidecar) on a GKE cluster. You also installed an app (PostgreSQL) that is properly configured to export Prometheus metrics. Using Kubernetes service annotations, Prometheus is now scraping metrics from this app. In the next section, you install Prometheus and PostgreSQL on the onprem cluster.

Install Prometheus on the onprem cluster (non-GKE workflow)

In this section, you configure the onprem cluster with Prometheus. Follow this workflow for non-GKE clusters.

1. Create a dedicated namespace in the onprem cluster for Prometheus:

   kubectl create namespace prometheus --context onprem
   

2. Create a Prometheus Kubernetes service account and the ClusterRole role:

   kubectl apply -f prometheus-service-account.yaml --context onprem
   

   The ClusterRole role gives the Prometheus server permissions to retrieve metrics from deployments running in all namespaces.

3. Create the Prometheus configmap to scrape metrics from apps running in the onprem cluster:

   kubectl apply -f prometheus-configmap.yaml --context onprem
   

Next, you create the Prometheus deployment. The Prometheus server uses the Kubernetes service account called prometheus (created in step 2) to scrape metrics from apps running in the Kubernetes cluster. The Prometheus server uses the Monitoring sidecar container to send these metrics to Monitoring. The sidecar requires a Google Cloud service account to authenticate to the Cloud Monitoring API and to push metrics. The onprem cluster is simulating a non-GKE cluster; therefore, it can't access Google Cloud service accounts using the Compute Engine instance metadata service. To configure the sidecar, you use the Google Cloud service account's JSON key that you downloaded earlier.

The following steps vary from the GKE workflow that you configured on the gke cluster.

1. Create a secret using the Google Cloud service account's JSON key:

   kubectl create secret -n prometheus generic prometheus-key --from-file=$WORKDIR/prometheus-service-account.json --context onprem
   

2. Define the environment variables needed to create the Prometheus deployment:

   export KUBE_NAMESPACE=prometheus
   export KUBE_CLUSTER=onprem
   export GCP_REGION=us-east4-a
   export GCP_PROJECT=$(gcloud info --format='value(config.project)')
   export DATA_DIR=/prometheus
   export DATA_VOLUME=prometheus-storage-volume
   export SIDECAR_IMAGE_TAG=release-0.3.2
   

3. Create the Prometheus deployment:

   envsubst < onprem-prometheus-deployment.yaml | kubectl --context onprem apply -f -
   

4. Wait a few moments and confirm that the Prometheus pod is running:

   kubectl get pods -n prometheus --context onprem
   

   The output looks similar to this:

   NAME                                     READY     STATUS    RESTARTS   AGE
   prometheus-deployment-75857dc9fc-vp5cr   2/2       Running   0          55s
   

   Note the two containers running in the Prometheus pod.

5. Verify the container images by running the following command.

   kubectl --context onprem get pods -n prometheus -ojson | jq '.items[].spec.containers[].image'
   

   The output looks similar to this:

   "prom/prometheus:v2.6.1"
   "gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:release-0.3.2"
   

   You see the Prometheus server and the stackdriver-prometheus-sidecar container running in the pod. The sidecar pod uses the secret prometheus-key for authentication to the Cloud Monitoring API. Inspect the volumes, volumeMounts, and env for the Prometheus deployment.

6. Inspect the volumes in Prometheus deployment:

   kubectl --context onprem -n prometheus get deploy prometheus-deployment -ojson | jq '.spec.template.spec.volumes'
   

   This command uses the secretName prometheus-key that you created earlier to create a volume called prometheus-key. The output looks similar to this excerpt:

     {
       "name": "prometheus-key",
       "secret": {
         "defaultMode": 420,
         "secretName": "prometheus-key"
       }
   
   

7. Inspect volumeMounts on the sidecar container:

   kubectl --context onprem -n prometheus get deploy prometheus-deployment -ojson | jq '.spec.template.spec.containers[1].volumeMounts'
   

   The output looks similar to this excerpt:

   {
     "mountPath": "/var/secrets/google",
     "name": "prometheus-key"
   }
   

   The volume prometheus-key is mounted in the mount path /var/secrets/google in the stackdriver-sidecar container. This path is where the service account key resides.

8. Inspect the environment variables defined for the Monitoring sidecar container. The environment variable [GOOGLE_APPLICATION_CREDENTIALS](/docs/authentication/getting-started#setting_the_environment_variable) is a special environment variable used by Google client libraries for authentication. This variable is assigned the value that points to the path of the prometheus-service-account.json key. The Monitoring sidecar uses Google client libraries and therefore uses this variable to authenticate to the Cloud Monitoring API.

   kubectl --context onprem -n prometheus get deploy prometheus-deployment -ojson | jq '.spec.template.spec.containers[1].env'
   

   The output looks similar to this:

   [
     {
       "name": "GOOGLE_APPLICATION_CREDENTIALS",
       "value": "/var/secrets/google/prometheus-service-account.json"
     }
   ]
   

9. Set up port forwarding to the Prometheus server UI running in the onprem cluster:

   export PROMETHEUS_POD_ONPREM=$(kubectl get pods --namespace prometheus -l "app=prometheus-server" \
       -o jsonpath="{.items[0].metadata.name}" --context onprem)
   kubectl --context onprem port-forward \
       --namespace prometheus $PROMETHEUS_POD_ONPREM 9091:9090 >> /dev/null &
   

   The next step is to open the Prometheus UI

10. In Cloud Shell, click Web Preview, and then click Change port.

    1. For Port Number, enter 9091.
    2. Click Change and Preview. The Prometheus server UI is displayed. This confirms that the Prometheus server is running.

    

Install PostgreSQL on the onprem cluster

In this section, you use Helm to install PostgreSQL on the onprem cluster. These steps are identical to the PostgreSQL install procedure for the gke cluster.

1. Switch to the onprem context:

   kubectx onprem
   

2. Grant Tiller, the server side of Helm, the cluster-admin role in the onprem cluster:

   kubectl create serviceaccount tiller --namespace kube-system
   kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
   

3. Initialize Helm and install Tiller in the onprem cluster:

   ${HELM_PATH}/helm init --service-account=tiller
   ${HELM_PATH}/helm update
   

   Ensure that Helm is properly installed by running the following command:

   ${HELM_PATH}/helm version
   

   If Helm is correctly installed, v2.13.0 appears for both client and server. You might need to run the command multiple times until you see both the client and server installed.

   Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
   Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
   

4. Install PostgreSQL using the stable Helm release:

   ${HELM_PATH}/helm install --name onprem \
   --set postgresUser=user,postgresPassword=password,postgresDatabase=postgres \
   stable/postgresql --set metrics.enabled=true --set postgresqlDatabase=prometheusdb
   

5. Ensure that PostgreSQL is running:

   kubectl get pods
   

   The output looks like this:

   onprem-postgresql-0   2/2       Running   0          39s
   

   Inspect the two containers running in the PostgreSQL pod:

   kubectl --context onprem get pods onprem-postgresql-0 -ojson | jq '.spec.containers[].image'
   

   The first container is the PostgreSQL container. The second container is the Prometheus metrics exporter for Postgres. The output looks like this:

   "docker.io/bitnami/postgresql:10.7.0"
   "docker.io/wrouesnel/postgres_exporter:v0.4.7"
   

6. Inspect the services running in the onprem cluster. PostgreSQL exposes Prometheus metrics using the onprem-postgresql-metrics service.

   kubectl get services --context onprem
   

   The output looks like this:

   onprem-postgresql-metrics    ClusterIP   10.47.240.77    <none>        9187/TCP   1m
   

7. Inspect the annotations on the onprem-postgresql-metrics service.

   kubectl --context onprem get service onprem-postgresql-metrics -ojson | jq '.metadata.annotations'
   

   The output looks like this:

   {
     "prometheus.io/port": "9187",
     "prometheus.io/scrape": "true"
   }
   

8. In the Prometheus UI window, click the Status > Targets page, scroll to the kubernetes-service-endpoints section, and click the show more button next to the kubernetes-service-endpoints link:

   

   Prometheus is scraping metrics from the PostgreSQL pod on port 9187 and scrape target /metrics.

You have successfully installed a Prometheus server (with the sidecar) on a non-GKE cluster, for example, a Kubernetes cluster running in an on-premises data center. You also installed PostgreSQL in the cluster, which is properly configured to export Prometheus metrics. Using Kubernetes service annotations, Prometheus is now scraping metrics from the PostgreSQL database.

In the next section, you set up Monitoring for monitoring and generate traffic to PostgreSQL databases running in both the gke and onprem clusters and monitor metrics in Monitoring.

Set up Monitoring

In this section, you configure Monitoring and create a dashboard. To complete these steps, you must have one of the following IAM roles on that project:

• Project Owner
• Monitoring Editor
• Monitoring Admin
• Monitoring Accounts Editor

To configure a Cloud Monitoring Workspace for your project, do the following:

1. In the Cloud Console, select your Google Cloud project.
   Go to Cloud Console
2. In the navigation pane, select Monitoring.

   If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

To create a dashboard, do the following:

1. In the Cloud Monitoring navigation pane, click Dashboards.
2. Click Create dashboard.
3. In the Dashboard name field, enter PostgreSQL.
4. Click Add Chart.
5. Ensure Metric is the selected tab.
6. Click in the box labeled Find resource type and metric, and enter pg_stat_database_blks_read.
7. In the Group By drop-down list, select cluster_name.

8. For Aggregator, enter sum.

   

   Note the metric name has a prefix of external/prometheus/. This indicates that the metrics are coming from the Prometheus server. In the chart, you see metrics coming in from both clusters, which means metrics are being exported by both the gke and onprem clusters.

9. Click Save Chart. Name the chart.
10. Click Add Chart.
11. For this chart, enter the metric pg_stat_database_blks_hit.
12. In the Group By drop-down list, select cluster_name.
13. For Aggregator, enter sum.
14. Click Save Chart. Name the chart.
15. Click Dashboards and select PostgreSQL. There are two charts in this dashboard.

16. Click list to see the legend and confirm that you are receiving metrics from both clusters.

    

    You can also add metrics from the dashboard by clicking Add Chart in the dashboard.

Generate traffic to PostgreSQL

In this section, you generate traffic to PostgreSQL databases running in both clusters to monitor metrics in the Monitoring dashboard. You use pgbench, a PostgreSQL benchmarking tool, to generate traffic. You use two Cloud Shell windows to generate traffic to both PostgreSQL databases at the same time.

Generate traffic to gke PostgreSQL instance

1. Next to the Cloud Shell tab in the top bar, open a second Cloud Shell tab:

   

   From the first Cloud Shell window, run the following commands to generate traffic to the PostgreSQL database running in the gke cluster.

2. Get the PostgreSQL database password in the gke cluster:

   export POSTGRES_PASSWORD_GKE=$(kubectl --context gke get secret --namespace default gke-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
   

3. Use port-forwarding to expose the PostgreSQL database on port 5432 in the gke cluster:

   kubectl --context gke port-forward --namespace default svc/gke-postgresql 5432:5432 >> /dev/null &
   

4. Log in to the PostgreSQL database:

   PGPASSWORD="$POSTGRES_PASSWORD_GKE" psql --host 127.0.0.1 -p 5432 -U postgres -d prometheusdb
   

5. Create a database named gketest for benchmarking:

   CREATE DATABASE gketest;
   

6. Exit the PostgreSQL database:

   \q
   

7. Initialize the gketest database for benchmarking:

   PGPASSWORD="$POSTGRES_PASSWORD_GKE" pgbench -i -h localhost -p 5432 -U postgres -d gketest
   

8. Start a benchmark on gketest database in the gke cluster. The following test runs for 10 mins as configured by the -T option (in seconds).

   PGPASSWORD="$POSTGRES_PASSWORD_GKE" pgbench -c 10 -T 600 -h localhost -p 5432 -U postgres -d gketest
   

Generate traffic to the onprem PostgreSQL instance

From the second Cloud Shell window, run the following commands to generate traffic to the PostgreSQL database running in the onprem cluster.

1. Get the PostgreSQL database password in the onprem cluster:

   export POSTGRES_PASSWORD_ONPREM=$(kubectl --context onprem get secret --namespace default onprem-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
   

2. Use port-forwarding to expose the PostgreSQL database on port 5431 in the onprem cluster:

   kubectl --context onprem port-forward --namespace default svc/onprem-postgresql 5431:5432 >> /dev/null &
   

3. Log in to the PostgreSQL database:

   PGPASSWORD="$POSTGRES_PASSWORD_ONPREM" psql --host 127.0.0.1 -p 5431 -U postgres -d prometheusdb
   

4. Create a database named onpremtest for benchmarking:

   CREATE DATABASE onpremtest;
   

5. Exit the PostgreSQL database.

   \q
   

6. Initialize the onpremtest database for benchmarking:

   PGPASSWORD="$POSTGRES_PASSWORD_ONPREM" pgbench -i -h localhost -p 5431 -U postgres -d onpremtest
   

7. Start a benchmark on onpremtest database in the onprem cluster. The following test runs for 10 mins.

   PGPASSWORD="$POSTGRES_PASSWORD_ONPREM" pgbench -c 10 -T 600 -h localhost -p 5431 -U postgres -d onpremtest
   

Review monitoring data

While the test is running, return to the Monitoring page and inspect the charts.

1. In the Cloud Console, go to Monitoring.

   Go to Monitoring

   

2. Go to Dashboard > PostgreSQL.

3. To enable auto refresh, click Auto Refresh . After a few minutes, the charts populate with database blocks read and hit data.

You are now receiving Prometheus metrics from apps running on multiple Kubernetes clusters into Monitoring.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Learn about white-box app monitoring for GKE with Prometheus.
• Read about the Cloud Monitoring integration with Prometheus.
• Learn about Cloud Monitoring on GKE.
• Try out other Google Cloud features for yourself. Have a look at our tutorials.