Use Policy Controller metrics

This page explains how to use metrics to monitor Policy Controller.

Policy Controller includes multiple metrics related to policy usage. For example, there are metrics recording the number of constraints and constraint templates, and the number of audit violations detected. To create and record these metrics, Policy Controller uses OpenTelemetry. You can configure Policy Controller to export these metrics to Prometheus or Cloud Monitoring. The default setting for exporting metrics exports the metrics to both Prometheus and Cloud Monitoring.

Configure metrics export

In Anthos Config Management versions 1.12.0 and later, you can configure how Policy Controller exports its metrics. You can choose Prometheus, Cloud Monitoring, both, or neither when installing Policy Controller. By default, Policy Controller attempts to export metrics to both Prometheus and Cloud Monitoring.

In Anthos Config Management versions earlier than 1.12.0, Policy Controller exports metrics to Prometheus only.

Export metrics to Cloud Monitoring

If Policy Controller is running inside a Google Cloud environment that has a default service account, Policy Controller automatically exports metrics to Cloud Monitoring in Anthos Config Management versions 1.12.0 and later.

If GKE Workload Identity or fleet Workload Identity are enabled, complete the following steps:

Create, or ask your administrator to create, a service account with the Monitoring Metric Writer (roles/monitoring.metricWriter) IAM role.
Bind the Kubernetes service account gatekeeper-admin in the namespace gatekeeper-system to a Google service account that you created in the previous step:
```
gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[gatekeeper-system/gatekeeper-admin]" \
    GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
```
Replace the following:
- PROJECT_ID: your project ID.
- GSA_NAME: the name of the Google service account that you created in the previous step.
This action requires iam.serviceAccounts.setIamPolicy permission on the project.

Annotate the Kubernetes service account using the email address of the Google service account:

kubectl annotate serviceaccount \
    --namespace gatekeeper-system \
    gatekeeper-admin \
    iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Restart the gatekeeper-controller-manager Pod:

kubectl rollout restart deployment gatekeeper-controller-manager -n gatekeeper-system

For examples on how to view these metrics, see Read OpenCensus metrics in Cloud Monitoring.

Export metrics to Prometheus

Policy Controller exports metrics for Prometheus on port 8888 of the gatekeeper-controller-manager-* Pod under the gatekeeper-system namespace. You can also configure Cloud Monitoring to pull custom metrics from Prometheus. For more information, see Managed Prometheus.

View metrics

Policy Controller metrics are exported to your Cloud Monitoring project in Prometheus format. As a result, you can query metrics by using the Cloud Monitoring API and a dashboard in the Google Cloud console. You can edit this dashboard to meet your business and operational needs.

To query the Cloud Monitoring API, use either Prometheus Query Language (PromQL) (the de-facto query language for Kubernetes metrics) or Monitoring Query Language (MQL) (Google's proprietary metrics query language).

To create the Policy Controller dashboard:

In the Google Cloud console, go to the Dashboards page.

Go to Dashboards
On the Dashboards overview page, click the Sample library tab. This tab shows you all of the dashboards that you can import.
In the Categories column, select Anthos Config Management.
In the Anthos Config Management samples table, select the Policy Controller checkbox.
Click Import.
To create the dashboard, click Confirm in the confirmation window.

To view and edit the Policy Controller dashboard:

On the Dashboards overview page, select the Dashboard list tab.
Select the Policy Controller dashboard.
To customize the dashboard, click Edit dashboard.
Make the required changes and click Save. To learn more about customizing dashboards, see Filter a dashboard in the Cloud Monitoring documentation.

Create alerts

To receive notifications when your metrics meet certain thresholds, create alerting policies in Cloud Monitoring.

Third party integration

By using the Cloud Monitoring API, any third party observability tool can ingest Policy Controller metrics.

For example, if you're using Grafana dashboards, add the Cloud Monitoring API as the data source in Grafana. To learn more, see Google Cloud Monitoring in the Grafana documentation.

Available metrics

If Policy Controller is enabled on your cluster, you can query the following metrics (all prefixed with OpenCensus/):

Name	Type	Labels	Description
`OpenCensus/audit_duration_seconds`	Cumulative		Audit cycle duration distribution
`OpenCensus/audit_last_run_time`	Gauge		The epoch timestamp since the last audit runtime, given as seconds in floating-point
`OpenCensus/constraint_template_ingestion_count`	Cumulative	status	Total number of constraint template ingestion actions
`OpenCensus/constraint_template_ingestion_duration_seconds`	Cumulative	status	Constraint Template ingestion duration distribution
`OpenCensus/constraint_templates`	Gauge	status	Current number of constraint templates
`OpenCensus/validation_request_count`	Counter	admission_status	Count of admission requests from the API server
`OpenCensus/validation_request_duration_seconds`	Cumulative	admission_status	Admission request duration distribution
`OpenCensus/violations`	Gauge	enforcement_action	Number of audit violations detected in the last audit cycle
`OpenCensus/watch_manager_intended_watch_gvk`	Gauge		How many unique GroupVersionKinds Policy Controller is meant to be watching. This metric is a combination of synced resources and constraints. Not currently implemented
`OpenCensus/watch_manager_watched_gvk`	Gauge		How many unique GroupVersionKinds Policy Controller is actually watching. This metric is meant to converge on being equal to OpenCensus/watch_manager_intended_watch_gvk. Not currently implemented

What's next

View Policy Controller status in the Google Cloud console