Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver


Filesystem in Userspace (FUSE) is an interface used to export a filesystem to the Linux kernel. Cloud Storage FUSE allows you to mount Cloud Storage buckets as a file system so that applications can access the objects in a bucket using common File IO operations (e.g. open, read, write, close) rather than using cloud-specific APIs.

The Cloud Storage FUSE CSI driver lets you use the Kubernetes API to consume pre-existing Cloud Storage buckets as volumes. Your applications can upload and download objects using Cloud Storage FUSE file system semantics. The Cloud Storage FUSE CSI driver provides a fully-managed experience powered by the open source Google Cloud Storage FUSE CSI driver.

The driver natively supports the following ways for you to configure your Cloud Storage-backed volumes:

Benefits

  • The Cloud Storage FUSE CSI driver on your cluster turns on automatic deployment and management of the driver. The driver works on both Standard and Autopilot clusters.
  • The Cloud Storage FUSE CSI driver does not need privileged access that is typically required by FUSE clients. This enables a better security posture.
  • The support of CSI ephemeral volumes simplifies volume configuration and management by eliminating the need for PersistentVolumeClaim and PersistentVolume objects.
  • The Cloud Storage FUSE CSI driver supports the ReadWriteMany, ReadOnlyMany, and ReadWriteOnce access modes.
  • You can use workload identity federation for GKE to manage authentication while having granular control over how your Pods access Cloud Storage objects.
  • If you are running ML training and serving workloads with frameworks like Ray, PyTorch, Spark, and TensorFlow, the portability and simplicity provided by the Cloud Storage FUSE CSI driver allow you to run your workloads directly on your GKE clusters without additional code changes.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Limitations

Requirements

To use the Cloud Storage FUSE CSI driver, your clusters must meet the following requirements:

Enable the Cloud Storage FUSE CSI driver

To create a Standard cluster with the Cloud Storage FUSE CSI driver enabled, you can use the gcloud CLI:

gcloud container clusters create CLUSTER_NAME \
    --addons GcsFuseCsiDriver \
    --cluster-version=VERSION \
    --region=COMPUTE_REGION \
    --workload-pool=PROJECT_ID.svc.id.goog

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • VERSION: the GKE version number. You must select 1.24 or later.
  • COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.
  • PROJECT_ID: your project ID.

To enable the driver on an existing Standard cluster, use the gcloud container clusters update command:

gcloud container clusters update CLUSTER_NAME \
    --update-addons GcsFuseCsiDriver=ENABLED \
    --region=COMPUTE_REGION

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the Compute Engine region of your cluster. For zonal clusters, use --zone=COMPUTE_ZONE.

After you enable the Cloud Storage FUSE CSI driver, you can use the driver in Kubernetes volumes by specifying the driver and provisioner name: gcsfuse.csi.storage.gke.io.

Configure access to Cloud Storage buckets using GKE workload identity federation for GKE

To make your Cloud Storage buckets accessible by your GKE cluster using workload identity federation for GKE, follow these steps. See Configure applications to use workload identity federation for GKE for more information.

  1. Get credentials for your cluster:

    gcloud container clusters get-credentials CLUSTER_NAME \
        --region=COMPUTE_REGION
    

    Replace the following:

    • CLUSTER_NAME: the name of your cluster that has workload identity federation for GKE enabled.
    • COMPUTE_REGION: the Compute Engine region of your cluster.
  2. Create a namespace to use for the Kubernetes service account. You can also use the default namespace or any existing namespace.

    kubectl create namespace NAMESPACE
    

    Replace the following:

    • NAMESPACE: the name of the Kubernetes namespace for the service account.
  3. Create a Kubernetes service account for your application to use. You can also use any existing Kubernetes service account in any namespace, including the default service account.

    kubectl create serviceaccount KSA_NAME \
        --namespace NAMESPACE
    

    Replace the following:

    • KSA_NAME: the name of your new Kubernetes service account.
    • NAMESPACE: the name of the Kubernetes namespace for the service account.
  4. Create an IAM service account for your application or use an existing IAM service account instead. You can use any IAM service account in your Cloud Storage bucket's project.

    gcloud iam service-accounts create GSA_NAME \
        --project=GSA_PROJECT
    

    Replace the following:

    • GSA_NAME: the name of the new IAM service account.
    • GSA_PROJECT: the project ID of the Google Cloud project for your IAM service account.
  5. Ensure that your IAM service account has the storage roles you need.

    You can grant the role to your IAM service account to only access a specific Cloud Storage bucket using the following command:

    gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
        --member "serviceAccount:GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com" \
        --role "ROLE_NAME"
    

    Replace the following:

    • BUCKET_NAME: your Cloud Storage bucket name.
    • GSA_NAME: the name of your IAM service account.
    • GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.
    • ROLE_NAME: the IAM role to assign to your service account. For read-only workloads, use roles/storage.objectViewer. For read-write workloads, use roles/storage.objectAdmin.

    Optionally, you can grant the role to your IAM service account to access all your Cloud Storage buckets in the project using the following command:

    gcloud projects add-iam-policy-binding GSA_PROJECT \
        --member "serviceAccount:GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com" \
        --role "ROLE_NAME"
    

    Replace the following:

    • GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.
    • GSA_NAME: the name of your IAM service account.
    • ROLE_NAME: the IAM role to assign to your service account. For read-only workloads, use roles/storage.objectViewer. For read-write workloads, use roles/storage.objectAdmin.
  6. Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.

    gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \
        --role roles/iam.workloadIdentityUser \
        --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
    

    Replace the following:

    • GSA_NAME: the name of your IAM service account.
    • GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.
    • PROJECT_ID: the project ID of your GKE cluster.
    • NAMESPACE: the name of the Kubernetes namespace for the service account.
    • KSA_NAME: the name of your new Kubernetes service account.
  7. Annotate the Kubernetes service account with the email address of the IAM service account.

    kubectl annotate serviceaccount KSA_NAME \
        --namespace NAMESPACE \
        iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com
    

    Replace the following:

    • KSA_NAME: the name of your new Kubernetes service account.
    • NAMESPACE: the name of the Kubernetes namespace for the service account.
    • GSA_NAME: the name of your IAM service account.
    • GSA_PROJECT: the project ID of the Google Cloud project of your IAM service account.

Prepare to mount Cloud Storage FUSE buckets

This section covers how to prepare to mount Cloud Storage FUSE buckets on your clusters.

Specify Pod annotations

The CSI driver relies on Pod annotations to identify if your Pod uses Cloud Storage-backed volumes. If the driver detects the necessary annotations, it injects a sidecar container called gke-gcsfuse-sidecar into your workload Pod. The Cloud Storage FUSE instances run inside the sidecar container and mount the Cloud Storage buckets for your workload.

To enable the CSI driver to mount the Cloud Storage buckets, make sure you specify the annotation gke-gcsfuse/volumes: "true" in your Pod specification, under the metadata field. If you want your Cloud Storage-backed volumes to be consumed by other Kubernetes workload types (for instance, Job, Deployment, or StatefulSet), make sure you configure the annotations under the spec.template.metadata.annotations field.

Configure resources for the sidecar container

By default, the sidecar container is configured with the following resource limits:

  • 250m CPU
  • 256 Mi memory
  • 5 Gi ephemeral storage

To overwrite these values, you can optionally specify the annotation gke-gcsfuse/[cpu-limit|memory-limit|ephemeral-storage-limit] as shown in the following example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    gke-gcsfuse/volumes: "true"
    gke-gcsfuse/cpu-limit: 500m
    gke-gcsfuse/memory-limit: 1Gi
    gke-gcsfuse/ephemeral-storage-limit: 50Gi

Use the following considerations when deciding the amount of resources to allocate:

  • The resource request is enforced to be equal to the resource limit.
  • If your workload Pod consumes multiple Cloud Storage volumes, the sidecar container resources are shared by multiple Cloud Storage FUSE instances. If this applies to you, consider increasing the resource allocation for multiple Cloud Storage volumes.
  • Allocate more CPU to the sidecar container if your workloads need higher throughput. Insufficient CPU will cause Cloud Storage FUSE throttling.
  • If your workloads need to process a large number of files, and the Cloud Storage FUSE metadata caching is enabled, increase the sidecar container's memory allocation. Cloud Storage FUSE memory consumption is proportional to the number of files but not the file size, because Cloud Storage FUSE does not cache file content. Insufficient memory will cause Cloud Storage FUSE out-of-memory errors and crash the workload application.
  • For write operations, Cloud Storage FUSE stages the files in a local temporary directory before the files are uploaded to the Cloud Storage bucket. Estimate how much free space your workload needs for staging when writing large files, and increase your ephemeral storage limit accordingly. To learn more, see Read/Writes semantics in the Cloud Storage FUSE GitHub documentation.
  • You can use value "0" to unset any resource limits on Standard clusters. For example, annotation gke-gcsfuse/memory-limit: "0" leaves the sidecar container memory limit and request empty. This is useful when you cannot decide on the amount of resources Cloud Storage FUSE needs for your workloads, and want to let the Cloud Storage FUSE consume all the available resources on a node.

Configure a private image for the sidecar container

This section describes how to use the sidecar container image if you are hosting it in a private container registry. This scenario might apply if you need to use private clusters for security purposes or if your cluster has limited access to the public internet. To configure and consume the private sidecar container image, follow these steps:

  1. Refer to this page to look for a compatible public sidecar container image.

  2. Pull it to your local environment and push it to your private container registry.

  3. In the manifest, specify a container named gke-gcsfuse-sidecar with only the image field. GKE will use the specified sidecar container image to prepare for the sidecar container injection. Here is an example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  containers:
  - name: gke-gcsfuse-sidecar
    image: PRIVATE_REGISTRY/gcs-fuse-csi-driver-sidecar-mounter:PRIVATE_IMAGE_TAG
  - name: main # your main workload container.

Replace the following:

  • PRIVATE_REGISTRY: your private container registry.
  • PRIVATE_IMAGE_TAG: your private sidecar container image tag.

Configure a custom buffer volume for the sidecar container

By default, GKE uses an emptyDir volume for Cloud Storage FUSE write buffering. You can specify a volume named gke-gcsfuse-buffer to replace the default emptyDir volume for Cloud Storage FUSE to stage the files in write operations. You can specify any type of storage, such as a PVC, and GKE will use the specified volume to prepare for the sidecar container injection. This is useful if you need to write files larger than 10Gi on Autopilot clusters. Here is an example of using a predefined PVC as the buffer volume:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  containers:
  ...
  volumes:
  - name: gke-gcsfuse-buffer
    persistentVolumeClaim:
      claimName: BUFFER_VOLUME_PVC

Replace the following:

  • BUFFER_VOLUME_PVC: the predefined PVC name.

Provision your volume as a CSI ephemeral volume

CSI ephemeral volumes backed by Cloud Storage buckets are tied to the Pod lifecycle. With this provisioning approach, you don't need to maintain the PersistentVolume and PersistentVolumeClaim objects associated with the Cloud Storage buckets after Pod termination.

Consume the CSI ephemeral storage volume in a Pod

  1. Save the following YAML manifest:

    apiVersion: v1
    kind: Pod
    metadata:
      name: gcs-fuse-csi-example-ephemeral
      namespace: NAMESPACE
      annotations:
        gke-gcsfuse/volumes: "true"
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: busybox
        name: busybox
        command: ["sleep"]
        args: ["infinity"]
        volumeMounts:
        - name: gcs-fuse-csi-ephemeral
          mountPath: /data
          readOnly: true
      serviceAccountName: KSA_NAME
      volumes:
      - name: gcs-fuse-csi-ephemeral
        csi:
          driver: gcsfuse.csi.storage.gke.io
          readOnly: true
          volumeAttributes:
            bucketName: BUCKET_NAME
            mountOptions: "implicit-dirs"
    

    The previous example shows how you can specify the Cloud Storage bucket inline in the Pod manifest. The example includes the following fields:

    • metadata.annotations: the annotation gke-gcsfuse/volumes: "true" is required. See Configure resources for the sidecar container for optional annotations.
    • spec.terminationGracePeriodSeconds: optional. By default, this is set to 30. If you need to write large files to the Cloud Storage bucket, increase this value to make sure that Cloud Storage FUSE has enough time to flush the data after your application exits. To learn more, see Kubernetes best practices: Terminating with grace.
    • spec.serviceAccountName: use the same Kubernetes service account as in the Configure access to Cloud Storage buckets using GKE workload identity federation for GKE step.
    • spec.volumes[n].csi.driver: use gcsfuse.csi.storage.gke.io as the CSI driver name.
    • spec.volumes[n].csi.volumeAttributes.bucketName: specify your Cloud Storage FUSE bucket name. You can specify an underscore (_) to mount all buckets that the service account can access. To learn more, see Dynamic Mounting in the Cloud Storage FUSE documentation.
    • spec.volumes[n].csi.volumeAttributes.mountOptions: optional. Pass mount flags to Cloud Storage FUSE. Specify the flags in one string separated by commas, without spaces.
    • spec.volumes[n].csi.readOnly: optional. Specify true if all the volume mounts are read-only.
    • spec.containers[n].volumeMounts[m].readOnly: optional. Specify true if only a specific volume mount is read-only.
  2. Apply the manifest to the cluster:

    kubectl apply -f FILE_PATH
    

    Replace FILE_PATH with the path to the YAML file.

Consume the CSI ephemeral storage volume in a Job workload

  1. Save the following YAML manifest:

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: gcs-fuse-csi-job-example
      namespace: NAMESPACE
    spec:
      template:
        metadata:
          annotations:
            gke-gcsfuse/volumes: "true"
        spec:
          serviceAccountName: KSA_NAME
          containers:
          - name: writer
            image: busybox
            command:
              - "/bin/sh"
              - "-c"
              - touch /data/test && echo $(date) >> /data/test && sleep 10
            volumeMounts:
            - name: gcs-fuse-csi-ephemeral
              mountPath: /data
          - name: reader
            image: busybox
            command:
              - "/bin/sh"
              - "-c"
              - sleep 10 && cat /data/test
            volumeMounts:
            - name: gcs-fuse-csi-ephemeral
              mountPath: /data
              readOnly: true
          volumes:
          - name: gcs-fuse-csi-ephemeral
            csi:
              driver: gcsfuse.csi.storage.gke.io
              volumeAttributes:
                bucketName: BUCKET_NAME
          restartPolicy: Never
      backoffLimit: 1
    

    Replace the following:

    The manifest deploys a Job that consumes a Cloud Storage FUSE bucket through a CSI ephemeral volume.

  2. Apply the manifest to the cluster:

    kubectl apply -f FILE_PATH
    

    Replace FILE_PATH with the path to the YAML file.

If you are using the CSI driver in a Job workload, or if the Pod RestartPolicy is Never, the sidecar container will exit automatically after all the other workload containers exit.

For additional examples, see Example Applications in the GitHub project documentation.

Provision your volume using static provisioning

With static provisioning, you create one or more PersistentVolume (PV) objects containing the details of the underlying storage system. Pods in your clusters can then consume the storage through PersistentVolumeClaims (PVCs).

Create a PersistentVolume

  1. Save the following YAML manifest:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: gcs-fuse-csi-pv
    spec:
      accessModes:
      - ReadWriteMany
      capacity:
        storage: 5Gi
      storageClassName: example-storage-class
      claimRef:
        namespace: NAMESPACE
        name: gcs-fuse-csi-static-pvc
      mountOptions:
        - implicit-dirs
      csi:
        driver: gcsfuse.csi.storage.gke.io
        volumeHandle: BUCKET_NAME
        readOnly: true
    

    The example manifest shows how you can define a PersistentVolume for Cloud Storage buckets. The example includes the following fields:

    • spec.claimRef.namespace: specify the PersistentVolumeClaim namespace.
    • spec.claimRef.name: specify the PersistentVolumeClaim name.
    • spec.csi.driver: use gcsfuse.csi.storage.gke.io as the CSI driver name.
    • spec.csi.volumeHandle: specify your Cloud Storage bucket name. You can pass an underscore (_) to mount all the buckets that the service account is configured to have access to. To learn more, see Dynamic Mounting in the Cloud Storage FUSE documentation.
    • spec.mountOptions: optional. Pass mount flags to Cloud Storage FUSE.
    • spec.csi.readOnly: optional. Specify true if all the volume mounts are read-only.
  2. Apply the manifest to the cluster:

    kubectl apply -f FILE_PATH
    

    Replace FILE_PATH with the path to the YAML file.

Create a PersistentVolumeClaim

  1. Save the following YAML manifest:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: gcs-fuse-csi-static-pvc
      namespace: NAMESPACE
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
      volumeName: gcs-fuse-csi-pv
      storageClassName: example-storage-class
    

    The example manifest shows how you can define a PersistentVolumeClaim to bind the PersistentVolume. The example includes the following fields:

    • metadata.namespace: specify the PersistentVolumeClaim namespace that must be consistent with the namespace of your workload.
    • spec.volumeName: specify the PersistentVolume name.

    To bind a PersistentVolume to a PersistentVolumeClaim, make sure to follow these guidelines:

    • spec.storageClassName fields on PV and PVC manifests should match. The storageClassName does not need to refer to an existing StorageClass object. To bind the claim to a volume, you can use any name you want but it cannot be empty.
    • spec.accessModes fields on PV and PVC manifests should match.
    • spec.capacity.storage on the PersistentVolume manifest should match spec.resources.requests.storage on the PersistentVolumeClaim manifest. Since Cloud Storage buckets don't have size limits, you can put any number for capacity but it cannot be empty.
  2. Apply the manifest to the cluster:

    kubectl apply -f FILE_PATH
    

    Replace FILE_PATH with the path to the YAML file.

Consume the volume from a PersistentVolumeClaim

  1. Save the following YAML manifest:

    apiVersion: v1
    kind: Pod
    metadata:
      name: gcs-fuse-csi-example-static-pvc
      namespace: NAMESPACE
      annotations:
        gke-gcsfuse/volumes: "true"
    spec:
      containers:
      - image: busybox
        name: busybox
        command: ["sleep"]
        args: ["infinity"]
        volumeMounts:
        - name: gcs-fuse-csi-static
          mountPath: /data
          readOnly: true
      serviceAccountName: KSA_NAME
      volumes:
      - name: gcs-fuse-csi-static
        persistentVolumeClaim:
          claimName: gcs-fuse-csi-static-pvc
    

    The example shows how you can define a Pod that consumes a Cloud Storage FUSE bucket through a PersistentVolumeClaim. The example includes the following fields:

  2. Apply the manifest to the cluster:

    kubectl apply -f FILE_PATH
    

    Replace FILE_PATH with the path to the YAML file.

For additional examples, see Example Applications in the GitHub project documentation.

Configure how Cloud Storage FUSE buckets are mounted

The Cloud Storage FUSE CSI driver supports mount flags to configure how Cloud Storage buckets are mounted on your local file system. For the full list of supported mount flags, see the gcsfuse CLI documentation.

You can specify the mount flags in the following ways:

  • In the spec.mountOptions field on a PersistentVolume manifest, if you use static provisioning.
  • In the spec.volumes[n].csi.volumeAttributes.mountOptions field, if you use CSI ephemeral volumes.

Use the following considerations when configuring mounts:

  • The following flags are disallowed: app-name, temp-dir, foreground, log-file, log-format, key-file, token-url, and reuse-token-from-url.
  • Cloud Storage FUSE does not make implicit directories visible by default. To make these directories visible, you can turn on the implicit-dirs mount flag. To learn more, see Files and Directories in the Cloud Storage FUSE GitHub documentation.
  • If you use a Security Context for your Pod or container, or if your container image uses a non-root user or group, you must set the uid and gid mount flags. You also need to use the file-mode and dir-mode mount flags to set the file system permissions. Note that you cannot run chmod, chown, or chgrp commands against a Cloud Storage FUSE file system, so uid, gid, file-mode and dir-mode mount flags are necessary to provide access to a non-root user or group.
  • If you only want to mount a directory in the bucket instead of the entire bucket, pass the directory relative path by using the only-dir=relative/path/to/the/bucket/root flag.
  • To tune Cloud Storage FUSE caching behavior, refer to Caching in the Cloud Storage FUSE GitHub documentation.
  • If you need to configure the Linux kernel mount options, you can pass the options using the o flag. For example, if you don't want to permit direct execution of any binaries on the mounted file system, set the o=noexec flag. Only the following options are allowed: exec, noexec, atime, noatime, sync, async, and dirsync.
  • If you need to troubleshoot Cloud Storage FUSE issues, set the debug_fuse, debug_fs, and debug_gcs flags.

Disable the Cloud Storage FUSE CSI driver

You cannot disable the Cloud Storage FUSE CSI driver on Autopilot clusters.

You can disable the Cloud Storage FUSE CSI driver on an existing Standard cluster by using the Google Cloud CLI.

gcloud container clusters update CLUSTER_NAME \
    --update-addons GcsFuseCsiDriver=DISABLED

Replace CLUSTER_NAME with the name of your cluster.

Troubleshooting

To troubleshoot issues when using the Cloud Storage FUSE CSI driver, see Troubleshooting Guide in the GitHub project documentation.

What's next