Run privileged workloads from GKE Autopilot partners

Autopilot

This page shows you how to run privileged workloads from Google Kubernetes Engine (GKE) Autopilot partners. You learn how to set up a synchronization workload that installs an allowlist in your cluster and keeps the allowlist up to date.

This page is for the following types of roles:

Security engineers who want to ensure that third-party workloads need an allowlist to run on your clusters and come from GKE-approved sources.
Platform engineers who want to enable third-party workloads on clusters to unblock application teams.

To learn more about the common roles and example tasks that we reference in our documentation, see Common GKE Enterprise user roles and tasks.

Before reading this page, ensure that you're familiar with the following:

About privileged partner workloads in Autopilot

GKE allows a subset of approved partners to run privileged workloads in Autopilot clusters. These privileged workloads can bypass some of the security constraints that Autopilot enforces. For example, a partner might need to run a workload that uses specific Linux capabilities or requires a privileged container.

Partners create and maintain allowlists for their privileged workloads. Each allowlist is a file that matches a specific privileged partner workload. Partners submit these allowlist files to GKE for approval. After approval, GKE hosts the allowlist file in a Google-managed repository.

To run a partner workload, you install the corresponding allowlist file in your cluster. GKE provides a Kubernetes custom resource named the AllowlistSynchronizer that installs allowlists and keeps them up to date. After an allowlist installs successfully, you can deploy the corresponding privileged partner workload.

Bugs and feature requests for privileged workloads and allowlists

Partners are responsible for creating, developing, and maintaining their privileged workloads and allowlists. If you encounter a bug or have a feature request for a privileged workload or allowlist, contact the corresponding partner.

About the `AllowlistSynchronizer` controller

The AllowlistSynchronizer is a controller that runs in your GKE control plane. You deploy a new AllowlistSynchronizer as a YAML manifest, similar to how you deploy any other Kubernetes workload. In the manifest, you specify the path to the allowlist file that you want to install, which you get from the third-party partner. The synchronizer finds the partner's allowlist file in a Google-managed repository and installs the allowlist in your cluster.

Every 10 minutes, the synchronizer checks for updates to the allowlist file. If an update exists, the synchronizer installs the updated allowlist in your cluster.

To stop allowing specific workloads, update existing AllowlistSynchronizers to remove the corresponding allowlist file paths, and then delete the WorkloadAllowlist object from your cluster. If you delete an installed WorkloadAllowlist object without removing the path from your allowlist synchronizer, the synchronizer re-installs the allowlist. Partners can't delete allowlist files from the Google-managed repository.

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Requirements

The AllowlistSynchronizer custom resource requires GKE version 1.32.1-gke.1729000 or later.
You must know which partner workload you want to run in your cluster. Check your partner's documentation for instructions to install the privileged workload.

Create a new `AllowlistSynchronizer`

To run a privileged workload from a partner, you add the path to the corresponding allowlist file to an AllowlistSynchronizer custom resource. You then deploy the AllowlistSynchronizer to your cluster.

In a text editor, create a new YAML file.
Add the following content to the YAML file:
```
apiVersion: auto.gke.io/v1
kind: AllowlistSynchronizer
metadata:
  name: ALLOWLIST_SYNCHRONIZER_NAME
spec:
  allowlistPaths:
  - ALLOWLIST1_PATH
  - ALLOWLIST2_PATH
```
Replace the following:
- ALLOWLIST_SYNCHRONIZER_NAME: the name of the new synchronizer. Choose a descriptive name that identifies the workload or team that the allowlist supports.
- ALLOWLIST1_PATH, ALLOWLIST2_PATH, ...: one or more paths to partner allowlist files to install. Check the documentation for your chosen partner's workload for this path. You can specify entire directories or individual files.
Deploy the YAML file to your cluster:
```
kubectl apply -f PATH_TO_YAML_FILE
```
Replace PATH_TO_YAML_FILE with the path to the YAML file that you created in the previous step.

The AllowlistSynchronizer controller installs allowlist files from the specified paths in your cluster.

Wait until the synchronizer reports a Ready status:

kubectl wait --for=condition=Ready allowlistsynchronizer/ALLOWLIST_SYNCHRONIZER_NAME \
  --timeout=60s

You can also integrate partner workload deployment into your continuous integration and continuous deployment (CI/CD) pipeline. Configure your workflow to wait until the allowlist successfully installs before deploying the corresponding workload.

Update an existing AllowlistSynchronizer

You can update an existing AllowlistSynchronizer to add or remove allowlist files. You might update existing synchronizers in situations like the following:

The partner adds a new allowlist file that has a different name.
You want to add a new workload allowlist to an existing synchronizer that groups related allowlists.
You want to remove an allowlist from a synchronizer because you no longer want to use the corresponding workload.

To update an existing AllowlistSynchronizer object, do the following:

List the existing synchronizers in your cluster:
```
kubectl get allowlistsynchronizer
```
Open the specification of the synchronizer that you want to update in a text editor.

Best Practice: To ensure that you track changes to synchronizers, update the YAML file that you used to create the synchronizer. Store these files in a version control system.
Update the spec.allowlistPaths field to add, modify, or remove allowlist file paths.
Save and close the text editor.
Apply the updated configuration to the cluster:
```
kubectl apply -f PATH_TO_YAML_FILE
```
Replace PATH_TO_YAML_FILE with the path to the YAML file that you updated in the previous step.

When you deploy an updated synchronizer configuration, the managedAllowlistStatus.generation field in the status of the AllowlistSynchronizer object increments by one. The AllowlistSynchronizer controller then applies your changes.

Monitor the status of allowlist synchronization

After you install an AllowlistSynchronizer or update an existing synchronizer, you can monitor the synchronization status. The status helps you to track the installation, removal, or modifications of allowlist files as well as any errors that might occur.

To monitor the general status of the synchronization, run the following command:

kubectl get allowlistsynchronizer ALLOWLIST_SYNCHRONIZER_NAME -o yaml

The output is similar to the following:

...
status:
  conditions:
  - type: Ready
    status: "False"
    reason: "SyncError"
    message: "some allowlists failed to sync: example-allowlist-1.yaml"
    lastTransitionTime: "2024-10-12T10:00:00Z"
    observedGeneration: 2
  managedAllowlistStatus:
    - filePath: "gs://path/to/allowlist1.yaml"
      generation: 1
      phase: Installed
      lastSuccessfulSync: "2024-10-10T10:00:00Z"
    - filePath: "gs://path/to/allowlist2.yaml"
      phase: Failed
      lastError: "Initial install failed: invalid contents"
      lastSuccessfulSync: "2024-10-08T10:00:00Z"

For a description of these fields, see AllowlistSynchronizer status.

Verify that an allowlist exists in your cluster

To verify that an allowlist exists in your cluster, run the following command:

kubectl get workloadallowlist

The output is a list of the installed allowlists in the cluster. Check that the output includes the allowlist that you want to use.

Deploy the privileged workload

After an allowlist installs successfully, you can deploy the corresponding workload in your cluster. The partner that provides the workload should also provide you with installation instructions for the workload. For a list of Autopilot partners and links to their documentation, see Autopilot partners.

Delete a privileged workload

To stop allowing a privileged workload to run in your clusters, remove the path to the corresponding allowlist from your AllowlistSynchronizer. The synchronizer uninstalls the allowlist.

If you delete a WorkloadAllowlist object from your cluster instead of updating the synchronizer, the synchronizer re-installs the allowlist. Ensure that you remove the path from the AllowlistSynchronizer.

To uninstall an allowlist, do the following:

In the YAML manifest for the AllowlistSynchronizer that manages the allowlist, remove the path to the allowlist that you want to uninstall. For instructions, see the Update an existing AllowlistSynchronizer section.
To verify that the allowlist was uninstalled, get a list of WorkloadAllowlist objects in your cluster:
```
kubectl get workloadallowlist
```
In the output, ensure that the allowlist that you wanted to remove doesn't appear.
Delete the workload from your cluster. For instructions, see the workload provider's documentation.

Prevent allowlist installation in your clusters

To prevent the installation of privileged workload allowlists in specific clusters, use a ValidatingAdmissionPolicy. Validating admission policies ensure that Kubernetes resources meet specific criteria before being allowed to run in the cluster. For example, you can validate that a label has a specific value.

To prevent installing allowlists in a cluster, do the following:

Save the following ValidatingAdmissionPolicy manifest as disallow-allowlists.yaml:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "disallow-allowlists"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["auto.gke.io"]
      apiVersions: ["*"]
      operations:  ["*"]
      resources:   ["allowlistsynchronizers"]
  validations:
  - expression: "false"
    message: 'AllowlistSynchronizer creation is not allowed'

Save the following ValidatingAdmissionPolicyBinding manifest as disallow-allowlists-binding.yaml:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "disallow-allowlists-binding"
spec:
  policyName: "disallow-allowlists"
  validationActions: [Deny]

Deploy the ValidatingAdmissionPolicy in your cluster:

kubectl apply -f disallow-allowlists.yaml
kubectl apply -f disallow-allowlists-binding.yaml

This policy prevents the creation of new AllowlistSynchronizers in the cluster.

Troubleshoot

If the synchronization or the workload deployment fail, see Troubleshoot deploying privileged Autopilot workloads.