Automatically audit workloads for configuration issues

This page shows you how to automatically audit your workload configurations for security concerns and get actionable recommendations to improve the security posture of your Google Kubernetes Engine (GKE) Autopilot and Standard clusters. Workload configuration auditing is a feature of the security posture dashboard. For more information, see About workload configuration auditing.

Pricing

The security posture dashboard is offered at no extra charge in GKE through the Container Security API.

Entries added to Cloud Logging use Cloud Logging pricing.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Requirements

  • To get the permissions that you need to use workload configuration auditing, ask your administrator to grant you the Security Posture Viewer (roles/containersecurity.viewer) IAM role on your Google Cloud project. For more information about granting roles, see Manage access to projects, folders, and organizations.

    This predefined role contains the permissions required to use workload configuration auditing. To see the exact permissions that are required, expand the Required permissions section:

    Required permissions

    The following permissions are required to use workload configuration auditing:

    • resourcemanager.projects.get
    • resourcemanager.projects.list
    • containersecurity.locations.list
    • containersecurity.locations.get
    • containersecurity.clusterSummaries.list
    • containersecurity.findings.list

    You might also be able to get these permissions with custom roles or other predefined roles.

  • Workload configuration auditing requires GKE version 1.21 and later.

Enable workload configuration auditing

Workload configuration auditing is enabled by default in new Autopilot and Standard clusters running version 1.27 and later. You can also manually enable this feature using the gcloud CLI or the Google Cloud console.

Enable configuration auditing on a new cluster

gcloud

Create a new GKE cluster using the gcloud CLI: 

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --security-posture=standard

Replace the following:

Console

  1. Go to the Google Kubernetes Engine page in the Google Cloud console.

    Go to Google Kubernetes Engine
  2. Click Create.
  3. In the GKE Autopilot section, click Configure.
  4. In the navigation pane, click Advanced settings. If you're creating a Standard cluster, click Security instead.
  5. In the Security section, select the Configuration audit checkbox.
  6. Configure other options for your cluster and click Create when you're ready.

Enable configuration auditing on an existing cluster

gcloud

Update the cluster: 

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --security-posture=standard

Replace the following:

Console

  1. Go to the Security Posture page in the Google Cloud console.

    Go to Security Posture
  2. Click the Settings tab.
  3. In the Configuration audit enabled clusters section, click Select clusters.
  4. Select the checkboxes for the clusters that you want to add.
  5. In the Select action drop-down menu, select Set to Basic.
  6. Click Apply.

If you use Google Kubernetes Engine (GKE) Enterprise edition to manage fleets of clusters, you can also configure fleet-level configuration auditing settings that apply to all member clusters. For instructions, see Configure GKE security posture dashboard features at fleet-level.

Deploy a test workload

Deploy a sample application that intentionally violates the Pod Security Standards.

  1. Save the following manifest as misconfig-sample.yaml:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloweb
  labels:
    app: hello
spec:
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
      - name: hello-app
        image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
        ports:
        - containerPort: 8080
        securityContext:
          runAsNonRoot: false
        resources:
          requests:
            cpu: 200m

  2. Deploy the application to your cluster:

    kubectl apply -f misconfig-sample.yaml

If you want to try other violations, modify misconfig-sample.yaml with the corresponding "bad" configuration.

View and action configuration audit results

The initial audit takes up to 15 minutes to return results. GKE displays the results on the security posture dashboard and automatically adds entries to the cluster logs.

View results

To see an overview of discovered concerns across your project's clusters and workloads, do the following:

  1. Go to the Security Posture page in the Google Cloud console.

    Go to Security Posture

  2. Click the Concerns tab.

  3. In the Filter concerns pane, in the Concern type section, select the Configuration checkbox.

View concern details and recommendations

To view detailed information about a specific configuration concern, click the row containing that concern.

The Configuration Concern pane shows the following information:

  • Description: a description of the concern.
  • Recommended action: an overview of actions that you can take to fix the configuration issue. This section includes the following details:
    • Which resources need the fix
    • Sample commands that you can run to apply the fix to affected resources
    • The Google Cloud console instructions, if applicable, to fix the issue

View logs for discovered concerns

GKE adds entries to the _Default log bucket in Logging for each discovered concern. These logs are only retained for a specific period. For details, see Logs retention periods.

  1. In the Google Cloud console, go to the Logs Explorer:

    Go to Logs Explorer

  2. In the Query field, specify the following query:

    resource.type="k8s_cluster"
jsonPayload.@type="type.googleapis.com/cloud.kubernetes.security.containersecurity_logging.Finding"
jsonPayload.type="FINDING_TYPE_MISCONFIG"

  3. Click Run query.

To receive notifications when GKE adds new findings to Logging, set up log-based alerts for this query. For more information, see Configure log-based alerts.

Clean up

  1. Delete the sample workload that you deployed.

    kubectl delete deployment helloweb

  2. Optionally, delete the cluster that you used.

    gcloud container clusters delete CLUSTER_NAME \
    --region=COMPUTE_REGION

Disable workload configuration auditing

You can disable workload configuration auditing using either the gcloud CLI or the Google Cloud console.

gcloud

Run the following command:

gcloud container clusters update CLUSTER_NAME \
    --region=LOCATION \
    --security-posture=disabled

Replace the following:

Console

  1. Go to the Security Posture page in the Google Cloud console.

    Go to Security Posture
  2. Click the Settings tab.
  3. In the Configuration audit enabled clusters section, click Select clusters.
  4. In the Audit enabled tab, select the checkboxes for the clusters that you want to remove.
  5. Click Disable audit, then click Confirm to disable auditing on those clusters.

Limitations of workload configuration auditing

  • Windows Server node pools aren't supported.
  • Workload configuration auditing doesn't scan GKE-managed workloads, such as workloads in the kube-system namespace.
  • Workload configuration auditing is only available for clusters with less than 1,000 nodes.

  • The security posture dashboard supports up to 150,000 active workload configuration auditing findings for each cluster. When the number of findings for a cluster exceeds this maximum, the security posture dashboard stops showing configuration findings for that cluster.

    To resolve this issue, use the logs in Logging to identify configuration issues and deploy updated manifests. When the number of configuration findings is less than 150,000, the security posture dashboard starts displaying findings for the cluster.

What's next