Use Image streaming to pull container images


This page shows you how to use Image streaming in Google Kubernetes Engine (GKE) to pull container images by streaming the image data as your applications need it.

Overview

Image streaming is a method of pulling container images in which GKE streams data from eligible images as requested by your applications. You can use Image streaming to allow your workloads to initialize without waiting for the entire image to download, which leads to significant improvements in initialization times. The shortened pull time provides you with benefits including the following:

  • Faster autoscaling
  • Reduced latency when pulling large images
  • Faster Pod startup

GKE usually downloads the entire container image onto each node and uses it as the root filesystem for your workloads. With Image streaming, GKE uses a remote filesystem as the root filesystem for any containers that use eligible container images. GKE streams image data from the remote filesystem as needed by your workloads.

While streaming the image data, GKE downloads the entire container image onto the local disk in the background and caches it. GKE then serves future data read requests from the cached image.

When you deploy workloads that need to read specific files in the container image, the Image streaming backend serves only those requested files.

Memory reservation for Image streaming

GKE reserves memory resources for Image streaming in addition to the memory that is reserved for node system components to run. GKE does not reserve additional CPU resources for Image streaming.

For details about the memory reservations GKE makes for node components, see Standard cluster architecture.

In nodes that use Image streaming, GKE makes the following additional memory reservations:

  • No additional memory for machines with less than 1 GiB of memory
  • 10% of the first 4 GiB of memory
  • 8% of the next 4 GiB of memory (up to 8 GiB)
  • 4% of the next 8 GiB of memory (up to 16 GiB)
  • 2.4% of the next 112 GiB of memory (up to 128 GiB)
  • 0.8% of any memory above 128 GiB

Requirements

You must meet the following requirements to use Image streaming:

Limitations

  • You can't use a Secret to pull container images.
  • Container images that use the V2 Image Manifest, schema version 1 are not eligible.
  • Container images encrypted with customer-managed encryption keys (CMEK) are not eligible for Image streaming. GKE downloads these images without streaming the data. You can still use CMEK to protect attached persistent disks and custom boot disks in clusters that use Image streaming.
  • Your GKE nodes must be in the same region as the Artifact Registry repository that contains the image, or in a member region of the Artifact Registry multi-region repository.
  • If your workloads read many files in an image during initialization, you might notice increased initialization times because of the latency added by the remote file reads.
  • You might not notice the benefits of Image streaming during the first pull of an eligible image. However, after Image streaming caches the image, future image pulls on any cluster benefit from Image streaming.
  • GKE uses the cluster-level configuration to determine whether to enable Image streaming on new node pools created using node auto-provisioning. However, you cannot use workload separation to create node pools with Image streaming enabled when Image streaming is disabled at the cluster level.
  • Container images protected by VPC Service Controls are not eligible for Image streaming. GKE downloads these images without streaming the data.
  • Linux file capabilities are not supported by Image streaming. If any of your container files are assigned capabilities like CAP_NET_RAW, the capabilities will not be available when the image file is streamed, or when the image is saved to the local disk. To avoid potential disruptions, do not use Image streaming for containers with these capabilities assigned.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set up default gcloud command-line tool settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud tool to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud tool like the following: One of [--zone, --region] must be supplied: Please specify location.

Enable Image streaming on clusters

You can enable Image streaming on new or existing clusters by using the gcloud tool --enable-image-streaming flag, or using the Google Cloud Console. By default, node pools in the cluster inherit the Image streaming setting at the cluster level. You can change this behaviour by enabling or disabling Image streaming on node pools in the cluster.

On a new cluster

You can enable Image streaming on new clusters using the gcloud tool or Cloud Console.

gcloud

To create a new cluster with Image streaming enabled, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --zone=COMPUTE_ZONE \
    --image-type="COS_CONTAINERD" \
    --enable-image-streaming

Replace the following:

  • CLUSTER_NAME: the name of your new cluster.
  • COMPUTE_ZONE: the Compute Engine zone for your new cluster. For regional clusters, use the --region=COMPUTE_REGION flag instead. Ensure that the zone or region is the same region or is within the multi-region of the Artifact Registry repository that contains the image.

Console

  1. Go to the Google Kubernetes Engine page in the Cloud Console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. In the GKE Standard section, click Configure.

  4. From the navigation pane, under Cluster, click Features.

  5. In the Other section, select the Enable Image streaming checkbox.

  6. Configure the cluster as needed, and then click Create.

On an existing cluster

You can enable Image streaming on existing clusters that meet the requirements using either the gcloud tool or Cloud Console.

gcloud

To update an existing cluster to use Image streaming, run the following command using the gcloud tool:

gcloud container clusters update CLUSTER_NAME \
    --enable-image-streaming

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. On the Clusters page, in the Features section, click next to Image streaming.

  4. In the Edit Image streaming dialog box, select the Enable Image streaming checkbox.

  5. Click Save changes.

After you modify the cluster, GKE enables Image streaming on your existing node pools automatically by default. If you explicitly enabled or disabled Image streaming on individual node pools, those node pools do not inherit the changes to the cluster-level setting.

Enable Image streaming on node pools

By default, node pools inherit the Image streaming setting at the cluster level. You can enable or disable Image streaming on specific node pools using the gcloud tool.

On a new node pool

To create a new node pool with Image streaming enabled, run the following command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=COMPUTE_ZONE \
    --image-type="COS_CONTAINERD" \
    --enable-image-streaming \

Replace the following:

  • NODE_POOL_NAME: the name of your new node pool.
  • CLUSTER_NAME: the name of the cluster for the node pool.
  • COMPUTE_ZONE: the Compute Engine zone of your cluster. For regional clusters, use the --region=COMPUTE_REGION flag instead.

On an existing node pool

You can enable Image streaming on existing node pools that meet the requirements.

To update an existing node pool to use Image streaming, run the following command:

gcloud container node-pools update POOL_NAME \
    --cluster=CLUSTER_NAME \
    --enable-image-streaming

Schedule a workload using Image streaming

After you enable Image streaming on your cluster, GKE automatically uses Image streaming when pulling eligible container images from Artifact Registry without requiring further configuration.

GKE adds the cloud.google.com/gke-image-streaming: "true" label to nodes in node pools with Image streaming enabled. If you enable or disable Image streaming on specific node pools so that your cluster has a mix of nodes that use Image streaming and nodes that don't, you can use node selectors in your deployments to control whether GKE schedules your workloads on nodes that use Image streaming.

In the following example, you schedule a Deployment that uses a large container image on a cluster with Image streaming enabled. You can then optionally compare the performance to an image pull without Image streaming enabled.

  1. Create a new cluster with Image streaming enabled:

    gcloud container clusters create CLUSTER_NAME \
        --zone=COMPUTE_ZONE \
        --enable-image-streaming \
        --image-type="COS_CONTAINERD"
    
  2. Get credentials for the cluster:

    gcloud container clusters get-credentials CLUSTER_NAME \
        --zone=COMPUTE_ZONE
    
  3. Save the following manifest as frontend-deployment.yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: frontend
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: guestbook
          tier: frontend
      template:
        metadata:
          labels:
            app: guestbook
            tier: frontend
        spec:
          containers:
          - name: php-redis
            image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
            env:
            - name: GET_HOSTS_FROM
              value: "dns"
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
            ports:
            - containerPort: 80
    

    The gb-frontend container image is 327 MB in size.

  4. Apply the manifest to your cluster:

    kubectl apply -f frontend-deployment.yaml
    
  5. Verify that GKE created the Deployment:

    kubectl get pods -l app=guestbook
    

    The output is similar to the following:

    NAMESPACE    NAME                          READY    STATUS       RESTARTS    AGE
    default      frontend-64bcc69c4b-pgzgm     1/1      Completed    0           3s
    
  6. Get the Kubernetes event log to see image pull events:

    kubectl get events --all-namespaces
    

    The output is similar to the following:

    NAMESPACE  LAST SEEN  TYPE    REASON          OBJECT                                                 MESSAGE
    default    11m        Normal  Pulling         pod/frontend-64bcc69c4b-pgzgm                          Pulling image "us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5"
    default    11m        Normal  Pulled          pod/frontend-64bcc69c4b-pgzgm                          Successfully pulled image "us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5" in 1.536908032s
    default    11m        Normal  ImageStreaming  node/gke-riptide-cluster-default-pool-f1552ec4-0pjv    Image us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5 is backed by image streaming.
    ...
    

    In this output:

    • The Pulled event shows the time taken for Image streaming to pull the image.
    • The ImageStreaming event shows that the node uses Image streaming to serve the container image.

Compare performance with standard image pulls

In this optional example, you create a new cluster with Image streaming disabled and deploy the frontend Deployment to compare performance with Image streaming.

  1. Create a new cluster with Image streaming disabled:

    gcloud container clusters create CLUSTER2_NAME\
        --zone=COMPUTE_ZONE \
        --image-type="COS_CONTAINERD"
    
  2. Get credentials for the cluster:

    gcloud container clusters get-credentials CLUSTER2_NAME \
        --zone=COMPUTE_ZONE
    
  3. Deploy the frontend Deployment from the previous example:

    kubectl apply -f frontend-deployment.yaml
    
  4. Get the Kubernetes event log:

    kubectl get events --all-namespaces
    

    The output is similar to the following:

     NAMESPACE  LAST SEEN  TYPE    REASON     OBJECT                             MESSAGE
     default    87s        Normal  Pulled     pod/frontend-64bcc69c4b-qwmfp      Successfully pulled image "us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5" in 23.929723476s
    

    Notice the time GKE took to pull the entire image. In this example output, GKE needed almost 24 seconds. With Image streaming enabled, GKE only needed 1.5 seconds to pull the image data that the workload required to start.

Clean up

To avoid charges, delete the clusters you created in the previous examples:

gcloud container clusters delete CLUSTER_NAME CLUSTER2_NAME

Use Image streaming with the cluster autoscaler

The cluster autoscaler uses Image streaming to improve autoscaling performance. New nodes created during scale-up events use the images imported by Image streaming instead of downloading images from the repository. The cluster autoscaler enables Image streaming on new node pools created in clusters that have Image streaming enabled.

Disable Image streaming

By default, GKE does not enable Image streaming on new clusters. You can disable Image streaming on existing clusters and on new and existing node pools.

On an existing cluster

You can disable Image streaming on existing clusters using either the gcloud tool or Cloud Console.

gcloud

To disable Image streaming on an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --no-enable-image-streaming

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. On the Clusters page, under Features, click next to Image streaming.

  4. In the Edit Image streaming dialog box, clear the Enable Image streaming checkbox.

  5. Click Save changes.

On a new node pool

To disable Image streaming when creating a new node pool, specify the --no-enable-image-streaming flag, such as in the following command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=COMPUTE_ZONE \
    --no-enable-image-streaming

On an existing node pool

To disable Image streaming on an existing node pool, run the following command:

gcloud container node-pools update NODE_POOL_NAME \
    --cluster=CLUSTER_NAME
    --no-enable-image-streaming

Troubleshooting

GKE doesn't use the Image streaming filesystem

If your GKE event log doesn't show the Image streaming events, your image is not backed by the remote filesystem. If GKE previously pulled the image on the node, this is expected behavior because GKE uses the local cache of the image for subsequent pulls instead of using Image streaming. You can verify this by looking for Container image IMAGE_NAME already present on machine in the Message field for the Pod Pulled event.

If you don't see the Image streaming event during the first image pull on the node, ensure that you meet the requirements for Image streaming. If you meet the requirements, you can diagnose the issue by connecting to the affected node and checking the logs of the Image streaming Service (named gcfsd):

journalctl -u gcfsd

Alternatively, you can check the logs using Logs Explorer to filter for the gcfsd logs with logName="projects/PROJECT_ID/logs/gcfsd".

Try one of the following solutions based on the error message displayed.

PermissionDenied

If the gcfsd logs display an error message similar to the following, the node doesn't have the correct API scope. GKE pulls container images for workloads without using Image streaming.

level=fatal msg="Failed to create a Container File System client: rpc error:
code = PermissionDenied desc = failed to probe endpoint: rpc error: code = PermissionDenied
desc = Request had insufficient authentication scopes."

You can fix this by granting the correct scope to the node to allow it to use Image streaming. Add the devstorage.read_only scope to the cluster or node pool, similar to the following command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --zone=COMPUTE_ZONE \
    --image-type="COS_CONTAINERD" \
    --enable-image-streaming \
    --scope="https://www.googleapis.com/auth/devstorage.read_only" \

FailedPrecondition

If you notice an error message with code = FailedPrecondition, the image wasn't imported to the Image streaming remote filesystem.

You might notice this error if you tried to use Image streaming with an existing node pool. If a node in the node pool already has the container image on-disk, GKE uses the local image instead of using Image streaming to get the image.

To fix this, try the following:

  • Wait a few minutes and try to deploy your workload again.
  • Add new nodes or a new node pool and schedule the workload on those nodes.

InvalidArgument

If you notice an error message with code=InvalidArgument, the container image your workload uses is not eligible for Image streaming. Ensure that the image meets the requirements. If your image is not on Artifact Registry, try migrating to Artifact Registry.