Using container image digests in Kubernetes manifests

This tutorial shows developers and operators who deploy containers to Kubernetes how to use container image digests to identify container images. A container image digest uniquely and immutably identifies a container image.

Deploying containers images by using the image digest provides several benefits compared to using image tags. If you are not familiar with image digests, read the accompanying document on using container image digests before you continue this tutorial.

The image argument for containers in a Kubernetes Pod specification accepts images with digests. This argument applies everywhere you use a Pod specification, such as in the template section of Deployment, StatefulSet, DaemonSet, ReplicaSet, and Job resources.

To deploy an image by using the digest, you use the image name, followed by @sha256: and the digest value. The following is an example of a Deployment resource that uses an image with a digest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
        ports:
        - containerPort: 8080

One downside of using image digests is that you don't know the digest value until after you have published your image to a registry. As you build new images, the digest value changes, and you need a way to update your Kubernetes manifests each time you deploy.

This tutorial shows how you can use tools such as kustomize, kpt, Skaffold, gke-deploy, ko, and Bazel to use image digests in your manifests.

Recommendations

This document presents several ways to use image digests in Kubernetes deployments. The tools described in this document are complementary. For instance, you can use the output of a kpt function with kustomize to create variants for different environments, and Skaffold can build images using Bazel and deploy to your Kubernetes clusters using kpt or kustomize.

The reason the tools are complementary is that they perform structured edits based on the Kubernetes resource model (KRM). This model makes the tools pluggable, and you can evolve your use of the tools to create processes and pipelines that help you deploy your apps and services.

To get started, we recommend the approach that works best with your existing tools and processes:

  • If you already use kustomize to manage Kubernetes manifests across environments, we recommend that you take advantage of its image transformers to deploy images by digest.

  • kpt is a great option if you need a flexible tool to manipulate Kubernetes resources and manifests, where your customization needs go beyond what kustomize can provide with overlays, patches, and transformers.

  • ko is a great way to build and publish images for Go apps, and it is used by open source projects such as Knative and Tekton.

  • If you already use Bazel to build your apps, we recommend that you add the rules_docker ruleset to build images, and the rules_k8s ruleset to populate image digests in your Kubernetes manifests.

If you don't use any of the tools described in this document, we recommend that you start with Skaffold. Skaffold is a common tool used by both developers and release teams, and it integrates with the other tools described in this tutorial. You can take advantage of these integration options as your requirements evolve.

Objectives

  • Use kustomize to generate a Kubernetes manifest with an image digest.
  • Use kpt to replace an image tag in a Kubernetes manifest with an image digest.
  • Use Skaffold to build and push an image, and to insert the image name and digest in a Kubernetes manifest.
  • Use gke-deploy to resolve an image tag to a digest in a Kubernetes manifest.
  • Use ko to build and push an image, and to insert the image name and digest in a Kubernetes manifest.
  • Use Bazel to build and push an image, and to insert the image name and digest in a Kubernetes manifest.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to the project selector page

  2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  3. Enable the Container Registry API.

    Enable the API

  4. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

  5. In Cloud Shell, create and go to a directory to store the files that you create in this tutorial:

    mkdir -p ~/container-image-digests-tutorial
    
    cd ~/container-image-digests-tutorial
    

Using kustomize

The kustomize command-line tool lets you customize Kubernetes manifests by using overlays, patches, and transformers.

You can use the kustomize image transformer to update the image name, tag, and digest in your existing manifest.

The following kustomization.yaml snippet shows how to configure the image transformer to use the transformer digest value for images where the Pod specification image value matches the transformer name value:

images:
- name: gcr.io/google-containers/echoserver
  digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229

To use a kustomize image transformer with an image digest, do the following:

  1. In Cloud Shell, install kustomize to your current directory:

    curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh \
        | bash
    
  2. Create a kustomization.yaml file:

    ./kustomize init
    
  3. Create a Kubernetes manifest with a Pod specification that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10
        ports:
        - containerPort: 8080
    EOF
    
  4. Add the manifest as a resource in the kustomization.yaml file:

    ./kustomize edit add resource pod.yaml
    
  5. Use an image transformer to update the digest of the image:

    ./kustomize edit set image \
        gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
    
  6. View the image transformer in the kustomization.yaml file:

    cat kustomization.yaml
    

    The file is the following:

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
    - pod.yaml
    images:
    - digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
      name: gcr.io/google-containers/echoserver
    
  7. View the resulting manifest:

    ./kustomize build .
    

    The output is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
        name: echoserver
        ports:
        - containerPort: 8080
    
  8. To apply the output to a Kubernetes cluster immediately, you can pipe the output to kubectl:

    ./kustomize build . | kubectl apply -f -
    

    If you want to apply the output later, you can redirect the output of the kustomize build command to a file. kustomize has other features that are outside the scope of this document. For more information, see the kustomize documentation. kustomize is also available as a Cloud Build community builder image.

Using kpt

The kpt command-line tool lets you manage, manipulate, customize, and apply Kubernetes resources. You can use the built-in commands, and you can extend kpt with your own functions.

You can use kpt setters to update image digests in your Kubernetes manifests when you build new images.

To use kpt to update the image digest in the deploy.yaml manifest that you previously created, do the following.

  1. In Cloud Shell, install kpt:

    sudo apt-get install -y google-cloud-sdk-kpt
    
  2. Create a kpt package called image-digests-tutorial in your current directory:

    kpt pkg init . --name image-digests-tutorial \
        --description "Container image digest tutorial"
    
  3. Define a kpt setter called echoimage for a field with the key image where the existing value is gcr.io/google-containers/echoserver:1.10:

    kpt cfg create-setter . echoimage \
        --field image gcr.io/google-containers/echoserver:1.10
    
  4. View the manifest file:

    cat pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10 # {"$kpt-set":"echoimage"}
        ports:
        - containerPort: 8080
    
  5. Get the digest value that you want to substitute:

    DIGEST=$(gcloud container images describe \
        gcr.io/google-containers/echoserver:1.10 \
        --format='value(image_summary.digest)')
    
  6. Set the new field value:

    kpt cfg set . echoimage gcr.io/google-containers/echoserver:1.10@$DIGEST
    

    When you run this command, kpt performs an in-place replacement of the image field value in the manifest.

  7. View the updated manifest file:

    cat pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: echo
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 # {"$kpt-set":"echoimage"}
        ports:
        - containerPort: 8080
    
  8. View the setter definition in the Kptfile file:

    cat Kptfile
    

    The output is the following:

    apiVersion: kpt.dev/v1alpha1
    kind: Kptfile
    metadata:
      name: image-digests-tutorial
    packageMetadata:
      shortDescription: Container image digest tutorial
    openAPI:
      definitions:
        io.k8s.cli.setters.echoimage:
          x-k8s-cli:
            setter:
              name: echoimage
              value: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
    

Using Skaffold

Skaffold is a command-line tool for continuous development and deployment of applications to Kubernetes clusters.

By default, Skaffold uses Git tags or abbreviated Git commit SHAs to tag images. You can instead use image digests by configuring the tagPolicy value in the skaffold.yaml configuration file:

build:
  tagPolicy:
    sha256: {}

Use Skaffold to build an image, push the image to Container Registry, and replace the image placeholder value in a Kubernetes manifest template with the name and digest of the pushed image:

  1. In Cloud Shell, install Skaffold:

    sudo apt-get install -y google-cloud-sdk-skaffold
    
  2. Clone the Skaffold Git repository:

    git clone https://github.com/GoogleContainerTools/skaffold.git
    
  3. Go to the directory of the getting-started example:

    cd skaffold/examples/getting-started
    
  4. Check out the Git tag that matches your version of Skaffold:

    git checkout $(skaffold version)
    
  5. Modify the Skaffold configuration file to use image digests:

    sed -i.new $'s/deploy:/  tagPolicy:\\\n    sha256: {}\\\ndeploy:/' skaffold.yaml
    
  6. View the skaffold.yaml configuration file:

    cat skaffold.yaml
    

    The file is similar to the following:

    apiVersion: skaffold/v2beta9
    kind: Config
    build:
      artifacts:
      - image: skaffold-example
      tagPolicy:
        sha256: {}
    deploy:
      kubectl:
        manifests:
          - k8s-*
    

    The build.artifacts section contains the image context—in this case, a placeholder name. Skaffold looks for this placeholder in the input manifest files. In this section, you also define the context directories that Skaffold uses to build images. By default, Skaffold uses the current directory as the build context.

    The build.tagPolicy section instructs Skaffold to use sha256 image digests instead of tags to reference images.

    The deploy section tells Skaffold to read input manifest files in the current directory with names matching the k8s-* pattern, and to deploy rendered manifests using kubectl apply.

    For an overview of all available options, see the skaffold.yaml reference documentation.

  7. View the Kubernetes manifest template:

    cat k8s-pod.yaml
    

    The file is the following:

    apiVersion: v1
    kind: Pod
    metadata:
      name: getting-started
    spec:
      containers:
      - name: getting-started
        image: skaffold-example
    

    The skaffold-example placeholder value in the image field matches the value of the image field in the skaffold.yaml file. Skaffold replaces this placeholder value with the full image name and digest in the rendered output.

  8. Build and push the image to Container Registry, and render the expanded Kubernetes manifest with the name and digest of the pushed image:

    skaffold render \
        --default-repo=gcr.io/$(gcloud config get-value core/project) \
        --interactive=false \
        --offline=true \
        --update-check=false
    

    In this command, the following options are used:

    • The --interactive=false and --update-check=false command arguments ensure that the output only includes the populated manifest, without any user prompts.
    • The --offline=true argument means that you can run the command without requiring access to a Kubernetes cluster.

    The output is similar to the following:

    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        app.kubernetes.io/managed-by: skaffold
        skaffold.dev/run-id: UUID
      name: getting-started
    spec:
      containers:
      - image: gcr.io/YOUR_PROJECT_ID/skaffold-example:latest@sha256:DIGEST
        name: getting-started
    

    In this output, you see the following values:

    • UUID: a unique identifier used by Skaffold to track resources
    • YOUR_PROJECT_ID: your Cloud project ID
    • DIGEST: the image digest value

Using gke-deploy

gke-deploy is a command-line tool that you use with Google Kubernetes Engine (GKE) that wraps the kubectl command-line tool. gke-deploy can modify the resources that you create following Google's recommended practices.

If you use the gke-deploy sub-commands prepare or run, gke-deploy resolves your image tags to digests and saves the expanded manifests with the image digests in the file output/expanded/aggregated-resources.yaml by default.

You can use gke-deploy run to both substitute the image tag for a digest and apply the expanded manifest to your GKE cluster. Although this command is convenient, there is a downside: the image tag is substituted at deployment time. The image associated with the tag might have changed in the time between when you decided to deploy, and when you deployed, resulting in deploying an unexpected image. For production deployments, we recommend separate steps for generating and applying manifests.

To replace an image tag in a Kubernetes deployment manifest with the image digest, do the following:

  1. In Cloud Shell, install gke-deploy:

    (cd $(mktemp -d) ; GO111MODULE=on go get github.com/GoogleCloudPlatform/cloud-builders/gke-deploy)
    
  2. Create a Kubernetes deployment manifest with a Pod specification that references the image gcr.io/google-containers/echoserver using the tag 1.10:

    cat << EOF > deploy.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echo-deployment
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: echo
      template:
        metadata:
          labels:
            app: echo
        spec:
          containers:
          - name: echoserver
            image: gcr.io/google-containers/echoserver:1.10
            ports:
            - containerPort: 8080
    EOF
    
  3. Generate an expanded manifest where the image tag is replaced by the digest:

    gke-deploy prepare \
        --filename deploy.yaml \
        --image gcr.io/google-containers/echoserver:1.10 \
        --version 1.10
    
  4. View the expanded manifest:

    cat output/expanded/aggregated-resources.yaml
    

    The output is the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app.kubernetes.io/managed-by: gcp-cloud-build-deploy
        app.kubernetes.io/version: "1.10"
      name: echo-deployment
      namespace: default
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: echo
      template:
        metadata:
          labels:
            app: echo
            app.kubernetes.io/managed-by: gcp-cloud-build-deploy
            app.kubernetes.io/version: "1.10"
        spec:
          containers:
          - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
            name: echoserver
            ports:
            - containerPort: 8080
    

    The --version argument you used with the gke-deploy command sets the value of the recommended app.kubernetes.io/version label in the deployment and the Pod template metadata of the expanded manifest.

    The gke-deploy tool is also available as a pre-built image for Cloud Build. To learn how to use gke-deploy with Cloud Build, see the Cloud Build documentation for gke-deploy.

Using ko

ko is a command-line tool for building Go container images and deploying them to Kubernetes clusters. ko builds images without using the Docker daemon, so you can use it in environments where you can't install Docker.

The ko sub-command publish builds images and publishes them to a container registry or loads them into your local Docker daemon.

The ko sub-command resolve does the following:

  • Identifies the images to build by finding placeholders in the image fields of the Kubernetes manifests that you provide by using the --filename argument.
  • Builds and publishes your images.
  • Replaces the image value placeholders with the names and digests of the images it built.
  • Prints the expanded manifests.

The ko sub-commands apply, create, and run perform the same steps as resolve, and then execute kubectl apply, create, or run with the expanded manifests.

To build an image from Go source code, and add the digest of the image to a Kubernetes deployment manifest, do the following

  1. In Cloud Shell, install ko to your current directory:

    curl -L https://github.com/google/ko/releases/download/v0.7.0/ko_0.7.0_Linux_x86_64.tar.gz | tar -zxf - ko
    
  2. Create a Go app with the module name example.com/hello-world in a new directory:

    mkdir -p app/cmd/ko-example ; cd app
    
    go mod init example.com/hello-world
    
    cat << EOF > cmd/ko-example/main.go
    package main
    
    import "fmt"
    
    func main() {
        fmt.Println("hello world")
    }
    EOF
    
  3. Define the image repository that ko uses to publish images:

    export KO_DOCKER_REPO=gcr.io/$(gcloud config get-value core/project)
    

    This example uses Container Registry, but you can use a different registry.

  4. To build and publish an image for your app, do one of the following steps:

    • Build and publish an image for your app by providing the path to your Go main package:

      ./ko publish --base-import-paths ./cmd/ko-example
      

      The optional argument --base-import-paths means that ko uses the short name of the main package directory as the image name.

      ko prints the image name and digest to stdout in the following format:

      gcr.io/YOUR_PROJECT_ID/ko-example@sha256:DIGEST_VALUE
      

      In this output:

      • YOUR_PROJECT_ID: your Cloud project ID
      • DIGEST_VALUE: image digest value
    • Use ko to replace a manifest placeholder with the name and digest of the image it builds and publishes:

      1. Create a Kubernetes Pod manifest. The manifest uses the placeholder ko://IMPORT_PATH_OF_YOUR_MAIN_PACKAGE as the value of the image field:

        cat << EOF > ko-pod.yaml
        apiVersion: v1
        kind: Pod
        metadata:
          name: ko-example
        spec:
          containers:
          - name: hello-world
            image: ko://example.com/hello-world/cmd/ko-example
        EOF
        
      2. Use ko resolve to build and publish an image for your app, and replace the manifest placeholder with the image name and digest:

        ./ko resolve --base-import-paths --filename ko-pod.yaml
        

        ko prints the manifest with the image name and digest to stdout:

        apiVersion: v1
        kind: Pod
        metadata:
          name: ko-example
        spec:
          containers:
          - name: hello-world
            image: gcr.io/YOUR_PROJECT_ID/ko-example@sha256:DIGEST
        

        In this output:

        • YOUR_PROJECT_ID: your Cloud project ID
        • DIGEST: the image digest value

Using Bazel

Bazel is an open source multi-language build tool, based on Google's internal Blaze build system.

rules_docker is a Bazel ruleset for building and pushing container images. The rules build reproducible images without using the Docker daemon, and Bazel's caching can speed up both building and pushing images to registries.

rules_k8s is a Bazel ruleset for working with Kubernetes manifests and clusters.

To build and push an image to populate a Kubernetes manifest with the image digest, do the following:

  1. In Cloud Shell, clone the rules_k8s Git repository:

    git clone https://github.com/bazelbuild/rules_k8s.git
    
  2. Go to the Git repository directory:

    cd rules_k8s
    
  3. Check out a recent Git tag:

    git checkout v0.4
    
  4. In the WORKSPACE file, replace the base image repository name that Bazel uses to push images with a repository where you have permission to push images:

    sed -i~ "s/image_chroot.*/image_chroot = \"gcr.io\/$(gcloud config get-value core\/project)\",/" WORKSPACE
    

    This example uses Container Registry, but you can use a different registry.

  5. Define the Bazel version to use with the build:

    echo 2.2.0 > .bazelversion
    
  6. Install the Bazel version to use with the build:

    sudo apt-get install -y bazel-$(cat .bazelversion)
    
  7. Fix the apiVersion field in the example Kubernetes deployment manifest template:

    sed -i~ 's/v1beta1/v1/' examples/hellohttp/deployment.yaml
    
  8. View the Kubernetes deployment manifest template:

    cat examples/hellohttp/deployment.yaml
    

    The file is the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello-http-staging
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: hello-http-staging
        spec:
          containers:
          - name: hello-http
            image: hello-http-image:latest
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    
  9. Build the Java image for the hellohttp example, push it to Container Registry, and replace the image value in the Kubernetes deployment manifest template with the pushed image name and digest. Redirect the stdout output to capture the expanded manifest in a new file called deployment.yaml.out:

    bazel --output_user_root=/tmp/bazel \
        run //examples/hellohttp/java:staging > deployment.yaml.out
    

    This command takes several minutes because it downloads the rulesets, compilers, and dependencies the first time that you run it.

    The --output_user_root argument tells Bazel to use a temporary directory outside your home directory for its disk-based cache. This temporary directory is necessary in Cloud Shell to avoid filling up your home directory volume.

  10. View the expanded manifest:

    cat deployment.yaml.out
    

    The file is similar to the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello-http-staging
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: hello-http-staging
        spec:
          containers:
          - image: gcr.io/YOUR_PROJECT_ID/hello-http-image@sha256:02701eb2beda4237849d92c993d1e59d036dd0f3a1afa40f5e490fea7b23f4c6
            imagePullPolicy: Always
            name: hello-http
            ports:
            - containerPort: 8080
    
  11. Examine the image that Bazel pushed to Container Registry:

    gcloud container images describe \
        gcr.io/$(gcloud config get-value core/project)/hello-http-image
    

Cleaning up

The easiest way to eliminate billing is to delete the Cloud project that you created for the tutorial. Alternatively, you can delete the individual resources.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Cloud project that you used in this tutorial, delete the individual resources:

  1. Delete the images in Container Registry:

    for IMG in hello-http-image ko-example skaffold-example ; do \
        gcloud container images list-tags \
            gcr.io/$(gcloud config get-value core/project)/$IMG \
            --format 'value(digest)' | xargs -I {} gcloud container images \
            delete --force-delete-tags --quiet \
            gcr.io/$(gcloud config get-value core/project)/$IMG@sha256:{}
    done
    

What's next