This document shows you how to set up Binary Authorization for Anthos clusters. It then shows you how to configure a sample Binary Authorization policy.
Before you begin
You must have Anthos clusters registered with Connect. Binary Authorization supports the following environments.
Anthos on bare metal
Anthos clusters on bare metal 1.14 or later.
Anthos clusters on VMware
Anthos clusters on VMware 1.4 or later.
The Binary Authorization service uses a public IP address, accessible through a regular internet connection. Configure your firewall rules for HTTPS to let the user cluster access the endpoint
binaryauthorization.googleapis.com
.Anthos on bare metal
Configure Anthos clusters on bare metal firewall rules.
Anthos clusters on VMware
Configure Anthos clusters on VMware firewall rules.
If you want to use centralized Cloud Audit Logs to view audit log entries, including those from Binary Authorization for Anthos clusters, you must configure Cloud Audit Logs in your cluster configuration.
Anthos on bare metal
Configure Cloud Audit Logs in Anthos clusters on bare metal.
Anthos clusters on VMware
Configure Cloud Audit Logs in Anthos clusters on VMware.
You must enable the Binary Authorization API as follows:
Go to Google Cloud console.
In the project drop-down list, select your Connect project. You can find this Google Cloud project in the
gkeConnect
section of your user cluster configuration file. This is the Google Cloud project that connects your user cluster to Google Cloud.
Set up Binary Authorization
In this section, you set up Binary Authorization for Anthos clusters in your cluster.
Specify installation environment variables
To specify the environment variables, do the following:
Using Workload Identity
Specify your Connect project:
export PROJECT_ID=PROJECT_ID
Specify the Fleet membership ID of your cluster:
export MEMBERSHIP_ID=MEMBERSHIP_ID
Using service account key
Specify your Connect project:
export PROJECT_ID=PROJECT_ID
Replace PROJECT_ID with the Google Cloud project in the
gkeConnect
section of your user cluster configuration file.Specify the path of the user cluster's kubeconfig file:
export KUBECONFIG=PATH
Replace PATH with the path of your user cluster kubeconfig file.
Choose a name for the Binary Authorization API access service account:
export SA_NAME=SERVICE_ACCOUNT_NAME
Replace SERVICE_ACCOUNT_NAME with the service account name of your choice. The Binary Authorization Module uses this service account to access the Binary Authorization API.
Specify the path to the service account key file you download later in this guide:
export SA_JSON_PATH=SA_KEY_FILE_PATH
Replace SA_KEY_FILE_PATH with the path of the JSON key file for the service account.
Install the Binary Authorization module in your user cluster
To install the Binary Authorization module, do the following:
Using Workload Identity
Fleet Workload Identity lets the workloads in your cluster authenticate to Google without requiring you to download, manually rotate, and generally manage Google Cloud service account keys. You can find out more about how fleet Workload Identity works and the advantages of using it in Use fleet Workload Identity.
Grant the
binaryauthorization.policyEvaluator
role to the Kubernetes service account on your Connect project:gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${PROJECT_ID}.svc.id.goog[binauthz-system/binauthz-admin]" \ --role="roles/binaryauthorization.policyEvaluator"
Create a working directory:
Make a directory called
binauthz
.Change to the directory.
Download the
manifest-wi-0.2.4.yaml.tmpl
file, which you use to install the Binary Authorization Module in your Anthos clusters user cluster:Anthos on bare metal
gsutil cp gs://anthos-baremetal-release/binauthz/manifest-wi-0.2.4.yaml.tmpl .
Anthos clusters on VMware
gsutil cp gs://gke-on-prem-release/binauthz/manifest-wi-0.2.4.yaml.tmpl .
Substitute the environment variables in the template:
envsubst < manifest-wi-0.2.4.yaml.tmpl > manifest-0.2.4.yaml
Install the Binary Authorization Module in your user cluster:
kubectl apply -f manifest-0.2.4.yaml
Verify the Deployment was created:
kubectl get pod --namespace binauthz-system
You see the Pod
binauthz-module-deployment-*
listed withStatus
ofRunning
and 1/1 Pods ready, similar to this output:NAME READY STATUS RESTARTS AGE binauthz-module-deployment-5fddf9594f-qjprz 1/1 Running 0 11s
Using service account key
Set the default project for the Google Cloud CLI:
gcloud config set project ${PROJECT_ID}
Create a Binary Authorization API access service account:
gcloud iam service-accounts create ${SA_NAME}
Grant the
binaryauthorization.policyEvaluator
role to the Binary Authorization API access service account on your Connect project:gcloud projects add-iam-policy-binding ${PROJECT_ID}\ --member="serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/binaryauthorization.policyEvaluator"
Create a working directory:
Make a directory called
binauthz
.Change to the directory.
Download the
manifest-0.2.4.yaml
file, which you use to install the Binary Authorization Module in your Anthos clusters user cluster:anthos_on_bare_metal_name
gsutil cp gs://anthos-baremetal-release/binauthz/manifest-0.2.4.yaml .
anthos_clusters_on_vmware
gsutil cp gs://gke-on-prem-release/binauthz/manifest-0.2.4.yaml .
Create a YAML file for the
binauthz-system
namespace.Copy the following to a file called
namespace.yaml
:apiVersion: v1 kind: Namespace metadata: labels: control-plane: binauthz-controller name: binauthz-system
Create the namespace in your user cluster:
kubectl apply -f namespace.yaml
You see output similar to the following:
namespace/binauthz-system created
Download a JSON key file for your service account:
gcloud iam service-accounts keys create ${SA_JSON_PATH} --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
Save the service account key as a Kubernetes Secret in your user cluster:
kubectl --namespace binauthz-system create secret generic binauthz-sa --from-file=key.json=${SA_JSON_PATH}
Install the Binary Authorization Module in your user cluster:
kubectl apply -f manifest-0.2.4.yaml
Verify the Deployment was created:
kubectl get pod --namespace binauthz-system
You see the Pod
binauthz-module-deployment-*
listed withStatus
ofRunning
and 1/1 Pods ready, similar to this output:NAME READY STATUS RESTARTS AGE binauthz-module-deployment-5fddf9594f-qjprz 1/1 Running 0 11s
Set up and use Binary Authorization policies
This section shows you how to set up and use Binary Authorization policies for Anthos clusters.
In each example, you configure the policy and then test it by attempting to deploy a container image in your Anthos cluster.
Allow all
This section demonstrates a success case. You configure the Binary Authorization policy so that a container image satisfies the policy and is deployed.
In Google Cloud, do the following:
Console
In the Google Cloud console, go to the Binary Authorization page.
Make sure to select your Connect project ID.
Click Edit Policy.
Under Project default rule, select Allow all images.
Click Save Policy.
gcloud
Set the
PROJECT_ID
for your Connect project. You can find this project ID in thegkeConnect
field in your user cluster configuration file.export PROJECT_ID=PROJECT_ID
Set the default Google Cloud project.
gcloud config set project ${PROJECT_ID}
Export the policy YAML file to your local system:
gcloud container binauthz policy export > policy.yaml
Your YAML file looks like the following:
admissionWhitelistPatterns: - namePattern: gcr.io/google_containers/* - namePattern: gcr.io/google-containers/* - namePattern: k8s.gcr.io/** - namePattern: gke.gcr.io/** - namePattern: gcr.io/stackdriver-agents/* globalPolicyEvaluationMode: ENABLE defaultAdmissionRule: evaluationMode: ALWAYS_ALLOW enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG name: projects/<var>PROJECT_ID</var>/policy
Edit
policy.yaml
.Set
evaluationMode
toALWAYS_ALLOW
.If you have a
requireAttestationsBy
block in the file, delete this block.Save the file.
Import
policy.yaml
as follows:gcloud container binauthz policy import policy.yaml
On your Anthos clusters admin workstation, do the following:
Create a manifest file for a Pod.
Save the following to a file called
pod.yaml
:apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: gcr.io/google-samples/hello-app@sha256:c62ead5b8c15c231f9e786250b07909daf6c266d0fcddd93fea882eb722c3be4
Create the Pod:
kubectl apply -f pod.yaml
You see that the Pod was successfully deployed.
Delete the Pod:
kubectl delete -f pod.yaml
Disallow all
This section demonstrates a failure case. In this section, you configure the default policy to disallow your container image from being deployed.
In Google Cloud do the following:
Console
In the Google Cloud console, go to the Binary Authorization page.
Make sure your Connect project is selected.
Click Edit Policy.
Under Project default rule, select Disallow all images.
Click Save policy.
gcloud
Set the
PROJECT_ID
to your Connect project ID.export PROJECT_ID=PROJECT_ID
Set the default Google Cloud project.
gcloud config set project ${PROJECT_ID}
Export the policy YAML file:
gcloud container binauthz policy export > policy.yaml
Edit
policy.yaml
.Set
evaluationMode
toALWAYS_DENY
.If you have a
requireAttestationsBy
block in the file, delete this block.Save the file.
Import
policy.yaml
as follows:gcloud container binauthz policy import policy.yaml
On your Anthos clusters admin workstation, do the following:
Create a manifest file for a Pod.
Save the following to a file called
pod.yaml
:apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: gcr.io/google-samples/hello-app@sha256:c62ead5b8c15c231f9e786250b07909daf6c266d0fcddd93fea882eb722c3be4
Create the Pod:
kubectl apply -f pod.yaml
You see that the Pod was was blocked from being deployed. The output looks like the following:
Error from server (VIOLATES_POLICY): error when creating "pod.yaml": admission webhook "binaryauthorization.googleapis.com" denied the request: Denied by default admission rule. Overridden by evaluation mode
Get the user cluster resource ID
This section shows you how to compose the cluster resource ID for your user cluster. In your Binary Authorization policy, you can create cluster-specific rules. You associate these rules with a cluster-specific resource ID, which is based on your cluster ID.
You get the resource ID as follows:
Console
In the Google Cloud console, go to the Anthos Clusters page.
Select the Connect project ID for Anthos clusters. You can find this project ID in the
gkeConnect
section of your user cluster configuration file.Under Anthos Managed Clusters, find your cluster ID under the Name column.
To create the resource ID, add the prefix
global.
to the cluster ID so that the resource ID has the following format:global.CLUSTER_ID
.
gcloud
Use SSH to connect to your Anthos clusters admin workstation.
On the admin workstation, run the following command:
kubectl get membership -o yaml
Get the cluster ID from
spec.owner.id
field of the output. Example output follows:apiVersion: v1 items: - apiVersion: hub.gke.io/v1 kind: Membership ... spec: owner: id: //gkehub.googleapis.com/projects/PROJECT_NUMBER/locations/global/memberships/my-cluster-id
In the example output, the cluster ID is
my-cluster-id
.To create the resource ID, add the prefix
global.
to the cluster ID. In the example, the resource ID isglobal.my-cluster-id
.
You use this resource ID when you define cluster-specific rules. Learn how to set cluster-specific rules by using the Google Cloud console or the gcloud CLI.
Update the failure policy
The Binary Authorization Module webhook can be configured to fail open or fail close.
Fail close
To update the failure policy to fail close, do the following:
Edit the manifest-0.2.4.yaml file and set failurePolicy to
Fail
Re-enable the webhook:
kubectl apply -f manifest-0.2.4.yaml
You see output similar to the following:
serviceaccount/binauthz-admin unchanged role.rbac.authorization.k8s.io/binauthz-role configured clusterrole.rbac.authorization.k8s.io/binauthz-role configured rolebinding.rbac.authorization.k8s.io/binauthz-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/binauthz-rolebinding unchanged secret/binauthz-tls unchanged service/binauthz unchanged deployment.apps/binauthz-module-deployment unchanged validatingwebhookconfiguration.admissionregistration.k8s.io/binauthz-validating-webhook-configuration configured
Fail open
To update the failure policy to fail open, do the following:
Edit the manifest-0.2.4.yaml file and set failurePolicy to
Ignore
Re-enable the webhook:
kubectl apply -f manifest-0.2.4.yaml
You see output similar to the following:
serviceaccount/binauthz-admin unchanged role.rbac.authorization.k8s.io/binauthz-role configured clusterrole.rbac.authorization.k8s.io/binauthz-role configured rolebinding.rbac.authorization.k8s.io/binauthz-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/binauthz-rolebinding unchanged secret/binauthz-tls unchanged service/binauthz unchanged deployment.apps/binauthz-module-deployment unchanged validatingwebhookconfiguration.admissionregistration.k8s.io/binauthz-validating-webhook-configuration configured
To learn more, see webhook failure policy.
Clean up
The following code sample shows how to disable the webhook:
kubectl delete ValidatingWebhookConfiguration/binauthz-validating-webhook-configuration
The following code sample shows how to re-enable the webhook:
kubectl apply -f manifest-0.2.4.yaml
The following code sample shows how to delete all of the resources related to Binary Authorization:
kubectl delete -f manifest-0.2.4.yaml kubectl delete namespace binauthz-system
What's next
- To check policy compliance at Pod creation time without actually blocking the Pod from being created, see Enable dry run.
- To force a Pod to be created that the Binary Authorization enforcer would otherwise block, see Use breakglass.
- Use the
built-by-cloud-build
attestor to deploy only images built by Cloud Build (Preview). - To implement an attestor-based Binary Authorization policy, see the following:
- Configure a policy using the Google Cloud console or the command-line tool. Set the default or cluster-specific rule to require attestations.
- Create an attestor using the Google Cloud console or the command-line tool.
- Create an attestation.
- To see log events related to Binary Authorization for Anthos clusters, see View Cloud Audit Logs events.
- Monitor metrics.