Using container image digests in Kubernetes manifests

This tutorial shows developers and operators who deploy containers to Kubernetes how to use container image digests to identify container images. A container image digest uniquely and immutably identifies a container image.

Deploying containers images by using the image digest provides several benefits compared to using image tags. For more information on image digests, see the accompanying document on using container image digests before you continue this tutorial.

The image argument for containers in a Kubernetes Pod specification accepts images with digests. This argument applies everywhere you use a Pod specification, such as in the template section of Deployment, StatefulSet, DaemonSet, ReplicaSet, and Job resources.

To deploy an image by using the digest, you use the image name, followed by @sha256: and the digest value. The following is an example of a Deployment resource that uses an image with a digest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-deployment
spec:
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
        ports:
        - containerPort: 8080

One downside of using image digests is that you don't know the digest value until after you have published your image to a registry. As you build new images, the digest value changes, and you need a way to update your Kubernetes manifests each time you deploy.

This tutorial shows how you can use tools such as Skaffold, kpt, digester, kustomize, gke-deploy, ko, and Bazel to use image digests in your manifests.

Recommendations

This document presents several ways to use image digests in Kubernetes deployments. The tools described in this document are complementary. For example, you can use the output of a kpt function with kustomize to create variants for different environments. Skaffold can build images using Bazel and deploy the images to your Kubernetes clusters using kpt or kustomize.

The reason the tools are complementary is that they perform structured edits based on the Kubernetes resource model (KRM). This model makes the tools pluggable, and you can evolve your use of the tools to create processes and pipelines that help you deploy your apps and services.

To get started, we recommend the approach that works best with your existing tools and processes:

  • By using the digester tool as a mutating admission webhook in your Kubernetes clusters, you can add digests to all your deployments with minimal impact on your current processes for building and deployment container images. The digester webhook also simplifies adoption of Binary Authorization, because it only requires a label to be added to a namespace.

  • kpt is a great option if you need a flexible tool to manipulate Kubernetes manifests. The digester tool can be used as a client-side KRM function in a kpt pipeline.

  • Skaffold can be used to add digests to image references. You enable this function with a small configuration change. Adopting Skaffold provides additional benefits, such as abstracting away how different tools build and deploy container images.

  • If you already use kustomize to manage Kubernetes manifests across environments, we recommend that you take advantage of its image transformers to deploy images by digest.

  • ko is a great way to build and publish images for Go apps, and it is used by open source projects such as Knative, Tekton, and sigstore.

  • If you already use Bazel to build your apps, we recommend that you add the rules_docker ruleset to build images, and the rules_k8s ruleset to populate image digests in your Kubernetes manifests.

If you don't use any of the tools described in this document, we recommend that you start with Skaffold and the digester webhook. Skaffold is a common tool used by both developers and release teams, and it integrates with the other tools described in this tutorial. You can take advantage of these integration options as your requirements evolve. The digester Kubernetes webhook complements Skaffold by enabling digest-based deployments for an entire organization.

Objectives

  • Use Skaffold to build and push an image, and to insert the image name and digest in a Kubernetes manifest.
  • Use kpt setters to replace an image tag in a Kubernetes manifest with an image digest.
  • Use the digester client-side function and mutating admission webhook to add digests to images in Kubernetes Pods and Pod templates.
  • Use kustomize to generate a Kubernetes manifest with an image digest.
  • Use gke-deploy to resolve an image tag to a digest in a Kubernetes manifest.
  • Use ko to build and push an image, and to insert the image name and digest in a Kubernetes manifest.
  • Use Bazel to build and push an image, and to insert the image name and digest in a Kubernetes manifest.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  3. Enable the Container Registry API.

    Enable the API

  4. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

Using Skaffold

Skaffold is a command-line tool for continuous development and deployment of applications to Kubernetes clusters.

Use Skaffold to build an image, push the image to Container Registry, and replace the image placeholder value in a Kubernetes manifest template with the name, tag, and digest of the pushed image:

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/skaffold
    cd ~/container-image-digests-tutorial/skaffold
    
  2. Clone the Skaffold Git repository:

    git clone https://github.com/GoogleContainerTools/skaffold.git
    
  3. Go to the directory of the getting-started example:

    cd skaffold/examples/getting-started
    
  4. Checkout the Git tag that matches your version of Skaffold:

    git checkout $(skaffold version)
    
  5. View the skaffold.yaml configuration file:

    cat skaffold.yaml
    

    The file resembles the following:

    apiVersion: skaffold/v2beta26
    kind: Config
    build:
      artifacts:
      - image: skaffold-example
    deploy:
      kubectl:
        manifests:
          - k8s-*
    

    The build.artifacts section contains the image context—in this case, a placeholder name. Skaffold looks for this placeholder in the input manifest files. In this section, you also define the context directories that Skaffold uses to build images. By default, Skaffold uses the current directory as the build context.

    The deploy section tells Skaffold to read input manifests in the current directory with names matching the k8s-* pattern, and to deploy rendered manifests using kubectl apply.

    For an overview of all available options, see the skaffold.yaml reference documentation.

  6. View the Kubernetes manifest template:

    cat k8s-pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: getting-started
    spec:
      containers:
      - name: getting-started
        image: skaffold-example
    

    The skaffold-example placeholder value in the image field matches the value of the image field in the skaffold.yaml file. Skaffold replaces this placeholder value with the full image name and digest in the rendered output.

  7. Build and push the image to Container Registry:

    skaffold build \
        --default-repo=gcr.io/$(gcloud config get-value core/project) \
        --file-output=artifacts.json \
        --interactive=false \
        --push=true \
        --update-check=false
    

    This command uses the following flags:

    • The --file-output flag specifies the file where Skaffold saves information about the built image, including the digest value.
    • The --push flag instructs Skaffold to push the built image to the container image registry specified by the --default-repo flag.
    • The --interactive and --update-check flags are both set to false. Set these flags to false in non-interactive environments, such as build pipelines, but leave them as their default values (true for both flags) for local development.

    If you use Google Cloud Deploy to deploy to GKE, use the file from the --file-output flag as the value of the --build-artifacts flag when you create a release.

  8. Render the expanded Kubernetes manifest with the name, tag, and digest of the container image from the previous step:

    skaffold render \
        --build-artifacts=artifacts.json \
        --digest-source=none \
        --interactive=false \
        --offline=true \
        --output=rendered.yaml \
        --update-check=false
    

    This command uses the following flags:

    • The --build-artifacts flag references the output file from the skaffold build command in the previous step.
    • The --digest-source=none flag means that Skaffold uses the digest value from the file provided in the --build-artifacts flag, instead of resolving the digest from the container registry.
    • The --offline=true flag means that you can run the command without requiring access to a Kubernetes cluster.
    • The --output flag specifies the output file for the rendered manifest.
  9. View the rendered manifest:

    cat rendered.yaml
    

    The output resembles the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: getting-started
    spec:
      containers:
      - image: gcr.io/YOUR_PROJECT_ID/skaffold-example:TAG@sha256:DIGEST
        name: getting-started
    

    In this output, you see the following values:

    • YOUR_PROJECT_ID: your Cloud project ID
    • TAG: the tag that Skaffold assigned to the image.
    • DIGEST: the image digest value

Using kpt setters

kpt is a command-line tool to manage, manipulate, customize, and apply Kubernetes resource manifests.

You can use the create-setters and apply-setters KRM functions from the kpt Functions Catalog to update image digests in your Kubernetes manifests when you build new images.

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/kpt
    cd ~/container-image-digests-tutorial/kpt
    
  2. Download kpt:

    mkdir -p ${HOME}/bin
    curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v1.0.0-beta.1/kpt_linux_amd64 --output ${HOME}/bin/kpt
    chmod +x ${HOME}/bin/kpt
    export PATH=${HOME}/bin:${PATH}
    
  3. Create a kpt package in the current directory:

    kpt pkg init --description "Container image digest tutorial"
    
  4. Create a Kubernetes Pod manifest that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10
        ports:
        - containerPort: 8080
    EOF
    
  5. Use kpt to create a setter called echoimage for the manifest field, where the existing value is gcr.io/google-containers/echoserver:1.10:

    kpt fn eval . \
        --image gcr.io/kpt-fn/create-setters:v0.1@sha256:0220cc87f29ff9abfa3a3b5643aa50f18d355d5e9dc9e1f518119633ddc4895c \
        -- "echoimage=gcr.io/google-containers/echoserver:1.10"
    
  6. View the manifest:

    cat pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10 # kpt-set: ${echoimage}
        ports:
        - containerPort: 8080
    
  7. Get the digest value of the container image:

    DIGEST=$(gcloud container images describe \
        gcr.io/google-containers/echoserver:1.10 \
        --format='value(image_summary.digest)')
    
  8. Set the new field value:

    kpt fn eval . \
        --image gcr.io/kpt-fn/apply-setters:v0.2@sha256:4d4295727183396f0c3c6a75d2560254c2f9041a39e95dc1e5beffeb49cc1a12 \
        -- "echoimage=gcr.io/google-containers/echoserver:1.10@$DIGEST"
    

    When you run this command, kpt performs an in-place replacement of the image field value in the manifest.

  9. View the updated manifest:

    cat pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 # kpt-set: ${echoimage}
        ports:
        - containerPort: 8080
    

Using digester

Digester adds digests to container and init container images in Kubernetes Pod and Pod template specifications. Digester replaces container image references that use tags:

spec:
  containers:
  - image: gcr.io/google-containers/echoserver:1.10

With references that use the image digest:

spec:
  containers:
  - image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229

Digester can run either as a mutating admission webhook in a Kubernetes cluster, or as a client-side KRM function with the kpt or kustomize command-line tools.

Using the digester KRM function

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/digester-fn
    cd ~/container-image-digests-tutorial/digester-fn
    
  2. Download the digester binary:

    DIGESTER_VERSION=v0.1.7
    mkdir -p ${HOME}/bin
    curl -L "https://github.com/google/k8s-digester/releases/download/${DIGESTER_VERSION}/digester_$(uname -s)_$(uname -m)" --output ${HOME}/bin/digester
    chmod +x ${HOME}/bin/digester
    export PATH=${HOME}/bin:${PATH}
    
  3. Create a Kubernetes Pod manifest that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10
        ports:
        - containerPort: 8080
    EOF
    
  4. Run the digester KRM function using kpt with the manifests in the current directory (.):

    kpt fn eval . --exec digester
    

    When you run this command, kpt performs an in-place update of the manifests in the current directory. If you want kpt to show the updated manifest on the console and leave the manifest file unchanged, add the --output unwrap flag.

  5. View the updated manifest:

    cat pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
        - name: echoserver
          image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
          ports:
            - containerPort: 8080
    

Using the digester admission webhook

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/digester-webhook
    cd ~/container-image-digests-tutorial/digester-webhook
    
  2. Download kind:

    KIND_VERSION=v0.11.1
    mkdir -p ${HOME}/bin
    curl -L "https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64" --output ${HOME}/bin/kind
    chmod +x ${HOME}/bin/kind
    export PATH=${HOME}/bin:${PATH}
    

    Kind is a command-line tool to run local Kubernetes clusters using Docker.

  3. Create a local Kubernetes cluster:

    kind create cluster
    
  4. Download the digester webhook kpt package and store it in a directory called manifests:

    DIGESTER_VERSION=v0.1.7
    kpt pkg get https://github.com/google/k8s-digester.git/manifests@${DIGESTER_VERSION} manifests
    
  5. Deploy the digester webhook:

    kpt live init manifests
    kpt live apply manifests --reconcile-timeout=3m --output=table
    
  6. Create a Kubernetes namespace called digester-demo in the kind cluster:

    kubectl create namespace digester-demo
    
  7. Add the digest-resolution: enabled label to the digester-demo namespace:

    kubectl label namespace digester-demo digest-resolution=enabled
    

    The digester webhook adds digests to Pods in namespaces with this label.

  8. Create a Kubernetes Deployment manifest that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echo-deployment
    spec:
      selector:
        matchLabels:
          app: echo
      template:
        metadata:
          labels:
            app: echo
        spec:
          containers:
          - name: echoserver
            image: gcr.io/google-containers/echoserver:1.10
            ports:
            - containerPort: 8080
    EOF
    
  9. Apply the manifest in the digester-demo namespace:

    kubectl apply --filename deployment.yaml --namespace digester-demo \
        --output jsonpath='{.spec.template.spec.containers[].image}{"\n"}'
    

    The --output flag instructs kubectl to output the image name to the console, followed by a newline character. The output is the following:

    gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
    

    This output shows that the digester webhook added the image digest to the Pod template specification in the Deployment resource.

  10. Delete the kind cluster to free up resources in your Cloud Shell session:

    kind delete cluster
    

Using kustomize image transformers

kustomize is a command-line tool that lets you customize Kubernetes manifests by using overlays, patches, and transformers.

You can use the kustomize image transformer to update the image name, tag, and digest in your existing manifest.

The following kustomization.yaml snippet shows you how to configure the image transformer to use the transformer digest value for images where the Pod specification image value matches the transformer name value:

images:
- name: gcr.io/google-containers/echoserver
  digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229

To use a kustomize image transformer with an image digest, do the following:

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/kustomize
    cd ~/container-image-digests-tutorial/kustomize
    
  2. Create a kustomization.yaml file:

    kustomize init
    
  3. Create a Kubernetes manifest with a Pod specification that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10
        ports:
        - containerPort: 8080
    EOF
    
  4. Add the manifest as a resource in the kustomization.yaml file:

    kustomize edit add resource pod.yaml
    
  5. Use an image transformer to update the digest of the image:

    kustomize edit set image \
        gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
    
  6. View the image transformer in the kustomization.yaml file:

    cat kustomization.yaml
    

    The file is the following:

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    - pod.yaml
    images:
    - digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
      name: gcr.io/google-containers/echoserver
    
  7. View the resulting manifest:

    kustomize build .
    

    The output is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
        name: echoserver
        ports:
        - containerPort: 8080
    
  8. To apply the resulting manifest to a Kubernetes cluster, you can use kubectl apply with the --kustomize flag:

    kubectl apply --kustomize .
    

    If you want to apply the output later, you can redirect the output of the kustomize build command to a file. Kustomize has other features that are outside the scope of this document. For more information, see the kustomize documentation. Kustomize is also available as a Cloud Build community builder image.

Using gke-deploy

gke-deploy is a command-line tool that you use with Google Kubernetes Engine (GKE). gke-deploy wraps the kubectl command-line tool and can modify the resources that you create following Google's recommended practices.

If you use the gke-deploy sub-commands prepare or run, gke-deploy resolves your image tags to digests and saves the expanded manifests with the image digests in the file output/expanded/aggregated-resources.yaml by default.

You can use gke-deploy run to both substitute the image tag for a digest and apply the expanded manifest to your GKE cluster. Although this command is convenient, there is a downside: the image tag is substituted at deployment time. The image associated with the tag might have changed in the time between when you decided to deploy, and when you deployed, resulting in deploying an unexpected image. For production deployments, we recommend separate steps for generating and applying manifests.

To replace an image tag in a Kubernetes deployment manifest with the image digest, do the following:

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/gke-deploy
    cd ~/container-image-digests-tutorial/gke-deploy
    
  2. Install gke-deploy:

    go install github.com/GoogleCloudPlatform/cloud-builders/gke-deploy@latest
    
  3. Create a Kubernetes Deployment manifest that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echo-deployment
    spec:
      selector:
        matchLabels:
          app: echo
      template:
        metadata:
          labels:
            app: echo
        spec:
          containers:
          - name: echoserver
            image: gcr.io/google-containers/echoserver:1.10
            ports:
            - containerPort: 8080
    EOF
    
  4. Generate an expanded manifest based on the deployment.yaml manifest:

    gke-deploy prepare \
        --filename deployment.yaml \
        --image gcr.io/google-containers/echoserver:1.10 \
        --version 1.10
    
  5. View the expanded manifest:

    cat output/expanded/aggregated-resources.yaml
    

    The output is the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/managed-by: gcp-cloud-build-deploy
        app.kubernetes.io/version: "1.10"
      name: echo-deployment
      namespace: default
    spec:
      selector:
        matchLabels:
          app: echo
      template:
        metadata:
          labels:
            app: echo
            app.kubernetes.io/managed-by: gcp-cloud-build-deploy
            app.kubernetes.io/version: "1.10"
        spec:
          containers:
          - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
            name: echoserver
            ports:
            - containerPort: 8080
    

    In the expanded manifest, the image tag is replaced by the digest.

    The --version argument you used with the gke-deploy command sets the value of the recommended app.kubernetes.io/version label in the deployment and the Pod template metadata of the expanded manifest.

    The gke-deploy tool is also available as a pre-built image for Cloud Build. To learn how to use gke-deploy with Cloud Build, see the Cloud Build documentation for gke-deploy.

Using ko

ko is a command-line tool for building Go container images and deploying them to Kubernetes clusters. ko builds images without using the Docker daemon, so you can use it in environments where you can't install Docker.

The ko sub-command publish builds images and publishes them to a container registry or loads them into your local Docker daemon.

The ko sub-command resolve does the following:

  • Identifies the images to build by finding placeholders in the image fields of the Kubernetes manifests that you provide by using the --filename argument.
  • Builds and publishes your images.
  • Replaces the image value placeholders with the names and digests of the images it built.
  • Prints the expanded manifests.

The ko sub-commands apply, create, and run perform the same steps as resolve, and then execute kubectl apply, create, or run with the expanded manifests.

To build an image from Go source code, and add the digest of the image to a Kubernetes deployment manifest, do the following

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/ko
    cd ~/container-image-digests-tutorial/ko
    
  2. Download ko and add it to your PATH:

    mkdir -p ${HOME}/bin
    curl -L https://github.com/google/ko/releases/download/v0.9.3/ko_0.9.3_$(uname -s)_$(uname -m).tar.gz | tar -zxC ${HOME}/bin ko
    export PATH=${HOME}/bin:${PATH}
    
  3. Create a Go app with the module name example.com/hello-world in a new directory called app:

    mkdir -p app/cmd/ko-example
    
    cd app
    
    go mod init example.com/hello-world
    
    cat << EOF > cmd/ko-example/main.go
    package main
    
    import "fmt"
    
    func main() {
        fmt.Println("hello world")
    }
    EOF
    
  4. Define the image repository that ko uses to publish images:

    export KO_DOCKER_REPO=gcr.io/$(gcloud config get-value core/project)
    

    This example uses Container Registry, but you can use a different registry.

  5. To build and publish an image for your app, do one of the following steps:

    • Build and publish an image for your app by providing the path to your Go main package:

      ko publish --base-import-paths ./cmd/ko-example
      

      The optional argument --base-import-paths means that ko uses the short name of the main package directory as the image name.

      ko prints the image name and digest to stdout in the following format:

      gcr.io/YOUR_PROJECT_ID/ko-example@sha256:DIGEST_VALUE
      

      In this output:

      • YOUR_PROJECT_ID: your Cloud project ID
      • DIGEST_VALUE: image digest value
    • Use ko to replace a manifest placeholder with the name and digest of the image it builds and publishes:

      1. Create a Kubernetes Pod manifest. The manifest uses the placeholder ko://IMPORT_PATH_OF_YOUR_MAIN_PACKAGE as the value of the image field:

        cat << EOF > ko-pod.yaml
        apiVersion: v1
        kind: Pod
        metadata:
          name: ko-example
        spec:
          containers:
          - name: hello-world
            image: ko://example.com/hello-world/cmd/ko-example
        EOF
        
      2. Use ko resolve to build and publish an image for your app, and replace the manifest placeholder with the image name and digest:

        ko resolve --base-import-paths --filename ko-pod.yaml
        

        ko prints the manifest with the image name and digest to stdout:

        apiVersion: v1
        kind: Pod
        metadata:
          name: ko-example
        spec:
          containers:
          - name: hello-world
            image: gcr.io/YOUR_PROJECT_ID/ko-example@sha256:DIGEST
        

        In this output:

        • YOUR_PROJECT_ID: your Cloud project ID
        • DIGEST: the image digest value

Using Bazel

Bazel is an open source multi-language build tool, based on Google's internal Blaze build system.

rules_docker is a Bazel ruleset for building and pushing container images. The rules build reproducible images without using the Docker daemon, and Bazel's caching can speed up both building and pushing images to registries.

rules_k8s is a Bazel ruleset for working with Kubernetes manifests and clusters.

To build and push an image to populate a Kubernetes manifest with the image digest, do the following:

  1. In Cloud Shell, create and go to a directory to store the files that you create in this section:

    mkdir -p ~/container-image-digests-tutorial/bazel
    cd ~/container-image-digests-tutorial/bazel
    
  2. Clone the rules_k8s Git repository:

    git clone https://github.com/bazelbuild/rules_k8s.git
    
  3. Go to the Git repository directory:

    cd rules_k8s
    
  4. Check out a Git tag:

    git checkout v0.6
    
  5. In the WORKSPACE file, replace the base image repository name that Bazel uses to push images with a repository where you have permission to push images:

    sed -i~ "s/image_chroot.*/image_chroot = \"gcr.io\/$(gcloud config get-value core\/project)\",/" WORKSPACE
    

    This example uses Container Registry, but you can use a different registry.

  6. Install the Bazel version to use with the build:

    sudo apt-get install -y bazel-$(cat .bazelversion)
    
  7. Fix the apiVersion field in the example Kubernetes deployment manifest template:

    sed -i~ 's/v1beta1/v1/' examples/hellohttp/deployment.yaml
    
  8. View the Kubernetes deployment manifest template:

    cat examples/hellohttp/deployment.yaml
    

    The file is the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello-http-staging
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: hello-http-staging
        spec:
          containers:
          - name: hello-http
            image: hello-http-image:latest
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    
  9. Build the Java image for the hellohttp example, push it to Container Registry, and replace the image value in the Kubernetes deployment manifest template with the pushed image name and digest. Redirect the stdout output to capture the expanded manifest in a new file called deployment.yaml.out:

    bazel --output_user_root=/tmp/bazel \
        run //examples/hellohttp/java:staging > deployment.yaml.out
    

    This command takes several minutes the first time that you run it because it downloads the rulesets, compilers, and dependencies.

    The --output_user_root argument tells Bazel to use a temporary directory outside your home directory for its disk-based cache. This temporary directory is necessary in Cloud Shell to avoid filling up your home directory volume.

  10. View the expanded manifest:

    cat deployment.yaml.out
    

    The file resembles the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello-http-staging
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: hello-http-staging
        spec:
          containers:
          - image: gcr.io/YOUR_PROJECT_ID/hello-http-image@sha256:b9d0a4643e9f11efe7cd300dd219ad691077ddfaccc118144cd83b7c472a8e86
            imagePullPolicy: Always
            name: hello-http
            ports:
            - containerPort: 8080
    
  11. Examine the image that Bazel pushed to Container Registry:

    gcloud container images describe \
        gcr.io/$(gcloud config get-value core/project)/hello-http-image
    

Clean up

The easiest way to eliminate billing is to delete the Cloud project that you created for the tutorial. Alternatively, you can delete the individual resources.

Delete the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Cloud project that you used in this tutorial, delete the individual resources:

  1. Delete the images in Container Registry:

    for IMG in hello-http-image ko-example skaffold-example ; do \
        gcloud container images list-tags \
            gcr.io/$(gcloud config get-value core/project)/$IMG \
            --format 'value(digest)' | xargs -I {} gcloud container images \
            delete --force-delete-tags --quiet \
            gcr.io/$(gcloud config get-value core/project)/$IMG@sha256:{}
    done
    
  2. Delete files you created in this tutorial:

    cd
    rm -rf ~/container-image-digests-tutorial
    

What's next