Setting up Binary Authorization for GKE on-prem

This document shows you how to set up Binary Authorization for GKE on-prem. It then shows you how to configure a Binary Authorization policy that can enforce GKE on-prem container image deployment.

Before you begin

  1. You must have GKE on-prem 1.4 or later installed and configured, with an admin and a user cluster. Your user cluster must be registered with Connect.

  2. The Binary Authorization service uses a public IP address, accessible through a regular internet connection. Configure your GKE on-prem firewall rules for HTTPS to let the user cluster access the endpoint binaryauthorization.googleapis.com.

  3. If you want to use centralized Cloud Audit Logs to view audit log entries, including those from Binary Authorization for GKE on-prem, you must configure Cloud Audit Logs in your user cluster configuration before you create the user cluster.

  4. You must enable the Binary Authorization API. To do this:

    1. Go to Google Cloud Console.

      Enable APIs

    2. In the project drop-down list, select your Connect project. You can find this Google Cloud project in the gkeConnect section of your user cluster configuration file. This is the Cloud project that connects your user cluster to Google Cloud.

Set up Binary Authorization for GKE on-prem

In this section, you set up Binary Authorization for GKE on-prem in your user cluster.

Specify installation environment variables

To specify the environment variables, do the following:

  1. Use SSH to connect to your GKE on-prem admin workstation.

  2. Specify your Connect project:

    export PROJECT_ID=PROJECT_ID
    

    Replace PROJECT_ID with the Cloud project in the gkeConnect section of your user cluster configuration file.

  3. Specify the path of the user cluster's kubeconfig file:

    export KUBECONFIG=PATH
    

    Replace PATH with the path of your user cluster kubeconfig file.

  4. Choose a name for the Binary Authorization API access service account:

    export SA_NAME=SERVICE_ACCOUNT_NAME
    

    Replace SERVICE_ACCOUNT_NAME with the service account name of your choice. The Binary Authorization Module uses this service account to access the Binary Authorization API.

  5. Specify the path to the service account key file you download later in this guide:

    export SA_JSON_PATH=SA_KEY_FILE_PATH
    

    Replace SA_KEY_FILE_PATH with the path of the JSON key file for the service account.

Install the Binary Authorization Module in your user cluster

To install the Binary Authorization Module, do the following:

  1. Set the default project for the gcloud command-line tool:

    gcloud config set project ${PROJECT_ID}
    
  2. Create a Binary Authorization API access service account:

    gcloud iam service-accounts create ${SA_NAME}
    
  3. Grant the binaryauthorization.policyEvaluator role to the Binary Authorization API access service account on your Connect project:

    gcloud projects add-iam-policy-binding ${PROJECT_ID}\
      --member serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\
      --role roles/binaryauthorization.policyEvaluator
    
  4. Create a working directory:

    1. Make a directory called binauthz.

    2. Change to the directory.

  5. Download the manifest.yaml file, which you use to install the Binary Authorization Module in your GKE on-prem user cluster:

    gsutil cp gs://gke-on-prem-release/binauthz/manifest-0.2.1.yaml .
    
  6. Create a YAML file for thebinauthz-system namespace.

    Copy the following to a file called namespace.yaml:

    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        control-plane: binauthz-controller
      name: binauthz-system
    
  7. Create the namespace in your user cluster:

    kubectl apply -f namespace.yaml
    

    You see output similar to the following:

    namespace/binauthz-system created
    
  8. Download a JSON key file for your service account:

    gcloud iam service-accounts keys create ${SA_JSON_PATH} --iam-account ${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
    
  9. Save the service account key as a Kubernetes Secret in your user cluster:

    kubectl --namespace binauthz-system create secret generic binauthz-sa --from-file=key.json=${SA_JSON_PATH}
    
  10. Install the Binary Authorization Module in your user cluster:

    kubectl apply -f manifest-0.2.1.yaml
    
  11. Verify the Deployment was created:

    kubectl get pod --namespace binauthz-system
    

    You see the Pod binauthz-module-deployment-* listed with Status of Running and 1/1 Pods ready, similar to this output:

NAME                                          READY   STATUS    RESTARTS   AGE
binauthz-module-deployment-5fddf9594f-qjprz   1/1     Running   0          11s

Use Binary Authorization for GKE on-prem

This section shows you how to set up and use Binary Authorization policies.

In each example, you configure the policy and then test it by attempting to deploy a container image in your GKE on-prem user cluster.

Allow all

This section demonstrates a success case. You configure the Binary Authorization policy so that a container image satisfies the policy and is deployed.

In Google Cloud, do the following:

Google Cloud Console

  1. In the Google Cloud Console, go to the Binary Authorization page.

Go to the Binary Authorization page

  1. Make sure to select your Connect project ID.

  2. Click Edit Policy.

  3. Under Project default rule, select Allow all images.

  4. Click Save policy.

command-line

  1. Set the PROJECT_ID for your Connect project. You can find this project ID in the gkeConnect field in your user cluster configuration file.

    export PROJECT_ID=PROJECT_ID
    

    Set the default Google Cloud project.

    gcloud config set project ${PROJECT_ID}
    
  2. Export the policy YAML file to your local system:

    gcloud container binauthz policy export  > policy.yaml
    

    Your YAML file looks like the following:

    admissionWhitelistPatterns:
    - namePattern: gcr.io/google_containers/*
    - namePattern: gcr.io/google-containers/*
    - namePattern: k8s.gcr.io/*
    - namePattern: gke.gcr.io/*
    - namePattern: gcr.io/stackdriver-agents/*
    globalPolicyEvaluationMode: ENABLE
    defaultAdmissionRule:
      evaluationMode: ALWAYS_ALLOW
      enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
    name: projects/<var>PROJECT_ID</var>/policy
    
  3. Edit policy.yaml.

  4. Set evaluationMode to ALWAYS_ALLOW.

  5. If you have a requireAttestationsBy block in the file, delete this block.

  6. Save the file.

  7. Import policy.yaml as follows:

    gcloud container binauthz policy import policy.yaml
    

On your GKE on-prem admin workstation, do the following:

  1. Create a manifest file for a Pod.

    Save the following to a file called pod.yaml:

    apiVersion: v1
    kind: Pod
    metadata:
      name: test-pod
    spec:
      containers:
      - name: test-container
        image: gcr.io/google-samples/hello-app@sha256:c62ead5b8c15c231f9e786250b07909daf6c266d0fcddd93fea882eb722c3be4
    
  2. Create the Pod:

    kubectl apply -f pod.yaml
    

    You see that the Pod was successfully deployed.

  3. Delete the Pod:

    kubectl delete -f pod.yaml
    

Deny all

This section demonstrates a failure case. In this section, you configure the default policy to disallow your container image from being deployed.

In Google Cloud do the following:

Google Cloud Console

  1. In the Google Cloud Console, go to the Binary Authorization page.

Go to the Binary Authorization page

  1. Make sure your Connect project is selected.

  2. Click Edit Policy.

  3. Under Project default rule, select Disallow all images.

  4. Click Save policy.

command-line

  1. Set the PROJECT_ID to your Connect project ID.

    export PROJECT_ID=PROJECT_ID
    
  2. Set the default Cloud project.

    gcloud config set project ${PROJECT_ID}
    
  3. Export the policy YAML file:

    gcloud container binauthz policy export  > policy.yaml
    
  4. Edit policy.yaml.

  5. Set evaluationMode to ALWAYS_DENY.

  6. If you have a requireAttestationsBy block in the file, delete this block.

  7. Save the file.

  8. Import policy.yaml as follows:

    gcloud container binauthz policy import policy.yaml
    

On your GKE on-prem admin workstation, do the following:

  1. Create a manifest file for a Pod.

    Save the following to a file called pod.yaml:

    apiVersion: v1
    kind: Pod
    metadata:
      name: test-pod
    spec:
      containers:
      - name: test-container
        image: gcr.io/google-samples/hello-app@sha256:c62ead5b8c15c231f9e786250b07909daf6c266d0fcddd93fea882eb722c3be4
    
  2. Create the Pod:

    kubectl apply -f pod.yaml
    

You see that the Pod was was blocked from being deployed. The output looks like the following:

Error from server (VIOLATES_POLICY): error when creating "pod.yaml": admission webhook "binaryauthorization.googleapis.com" denied the request: Denied by default admission rule. Overridden by evaluation mode

Getting the user cluster resource ID for cluster-level policies

This section shows you how to find the cluster resource ID for your user cluster. Binary Authorization policy rules can be defined for specific clusters. A cluster-level policy rule associates a policy rule with a resource ID, which is based on your cluster ID.

You can get cluster-id as follows:

  1. Use SSH to connect to your GKE on-prem admin workstation.

  2. Get the user cluster ID:

    1. Run the following command:

      kubectl get membership -o yaml
      
    2. Get the cluster ID from spec.owner.id field of the output:

         apiVersion: v1
         items:
         - apiVersion: hub.gke.io/v1
           kind: Membership
           ...
           spec:
             owner:
               id: //gkehub.googleapis.com/projects/PROJECT_NUMBER/locations/global/memberships/my-cluster-id
      

      The cluster ID in the example output is my-cluster-id.

      The resource ID is global.my-cluster-id.

You use this resource ID when you define cluster-specific rules. For more information, see "Set cluster specific rules" using the Google Cloud Console or using a command-line tool.

Clean up

  1. The following code sample shows how to disable the webhook:

    kubectl delete ValidatingWebhookConfiguration/binauthz-validating-webhook-configuration
    
  2. The following code sample shows how to re-enable the webhook:

    kubectl apply -f manifest-0.2.1.yaml
    
  3. The following code sample shows how to delete all of the resources related to Binary Authorization:

    kubectl delete -f manifest-0.2.1.yaml
    kubectl delete namespace binauthz-system
    

What's next