Configure Cloud Storage volume mounts for jobs

This page shows how to mount a Cloud Storage bucket as a storage volume, using Cloud Run volume mounts.

Mounting the bucket as a volume in Cloud Run presents the bucket content as files in the container file system. After you you mount the bucket as a volume, you access the bucket as if it were a directory on your local file system, using your programming language's file system operations and libraries instead of using Google API Client Libraries.

Memory requirements

Cloud Storage volume mounts use the Cloud Run container memory for the following activities:

For all Cloud Storage FUSE caching, Cloud Run by default uses the stat cache setting with a Time-to-live (TTL) of 60 seconds. The default maximum size of the stat cache is 32 MB, the default maximum size of the type cache is 4 MB.
When reading, Cloud Storage FUSE also consumes memory other than stat and type caches, for example a 1 MiB array for every file being read and for goroutines.
When writing to Cloud Storage, the entire file is staged in Cloud Run memory before the file is written to Cloud Storage.

Limitations

Cloud Run uses Cloud Storage FUSE for this volume mount. So there are a few things to keep in mind when mounting a Cloud Storage bucket as a volume:

Cloud Storage FUSE does not provide concurrency control for multiple writes (file locking) to the same file. When multiple writes try to replace a file, the last write wins and all previous writes are lost.
Cloud Storage FUSE is not a fully POSIX-compliant file system. For more details, refer to the Cloud Storage FUSE documentation.

Before you begin

You need a Cloud Storage bucket to mount as the volume.

Required roles

To get the permissions that you need to configure Cloud Storage volume mounts, ask your administrator to grant you the following IAM roles:

Cloud Run Developer (roles/run.developer) on the Cloud Run job
Service Account User (roles/iam.serviceAccountUser) on the service identity

To get the permissions that your service identity needs to access the file and Cloud Storage bucket, ask your administrator to grant the service identity the following IAM role:

Storage Admin (roles/storage.admin)

For more details on Cloud Storage roles and permissions, see IAM for Cloud Storage.

For a list of IAM roles and permissions that are associated with Cloud Run, see Cloud Run IAM roles and Cloud Run IAM permissions. If your Cloud Run job interfaces with Google Cloud APIs, such as Cloud Client Libraries, see the service identity configuration guide. For more information about granting roles, see deployment permissions and manage access.

Disallowed paths

Cloud Run does not allow you to mount a volume at /dev, /proc and /sys, or on their subdirectories.

Mount a Cloud Storage volume

You can mount multiple buckets at different mount paths. You can also mount a volume to more than one container using the same or different mount paths across containers.

If you are using multiple containers, first specify the volumes, then specify the volume mounts for each container.

gcloud

To add a volume and mount it:
```
gcloud beta run jobs update JOB \
--add-volume name=VOLUME_NAME,type=cloud-storage,bucket=BUCKET_NAME \
--add-volume-mount volume=VOLUME_NAME,mount-path=MOUNT_PATH
```
Replace:
- JOB with the name of your job.
- MOUNT_PATH with the relative path where you are mounting the volume, for example, /mnt/my-volume.
- VOLUME_NAME with any name you want for your volume. The VOLUME_NAME value is used to map the volume to the volume mount.
- BUCKET_NAME with the name of your Cloud Storage bucket.

To mount your volume as a read-only volume:

--add-volume=name=VOLUME_NAME,type=cloud-storage,bucket=BUCKET_NAME,readonly=true

If you are using multiple containers, first specify your volume(s), then specify the volume mount(s) for each container:

gcloud beta run jobs update JOB \
--add-volume name=VOLUME_NAME,type=cloud-storage,bucket=BUCKET_NAME \
--container CONTAINER_1 \
--add-volume-mount volume=VOLUME_NAME,mount-path=MOUNT_PATH \
--container CONTAINER_2 \
--add-volume-mount volume=VOLUME_NAME,mount-path=MOUNT_PATH2

YAML

If you are creating a new job, skip this step. If you are updating an existing job, download its YAML configuration:
```
gcloud run jobs describe JOB_NAME --format export > job.yaml
```

Update the MOUNT_PATH, VOLUME_NAME, BUCKET_NAME and IS_READ_ONLY as needed.

apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: JOB_NAME
spec:
  metadata:
    annotations:
      run.googleapis.com/launch-stage: BETA
  template:
    metadata:
      annotations:
        run.googleapis.com/execution-environment: gen2
    spec:
      template:
        spec:
          containers:
          - image: IMAGE_URL
            volumeMounts:
            - mountPath: MOUNT_PATH
              name: VOLUME_NAME
          volumes:
          - name: VOLUME_NAME
            csi:
              driver: gcsfuse.run.googleapis.com
              readOnly: IS_READ_ONLY
              volumeAttributes:
                bucketName: BUCKET_NAME

Replace

IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL has the shape LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG
MOUNT_PATH with the relative path where you are mounting the volume, for example, /mnt/my-volume.
VOLUME_NAME with any name you want for your volume. The VOLUME_NAME value is used to map the volume to the volume mount.
IS_READ_ONLY with True to make the volume read-only, or False to allow writes.
BUCKET_NAME with the name of the Cloud Storage bucket.

Replace the service with its new configuration using the following command:
```
gcloud beta run jobs replace job.yaml
```

Reading and writing to a volume

If you use the Cloud Run volume mount feature, you access a mounted volume using the same libraries in your programming language that you use to read and write files on your local file system.

This is especially useful if you're using an existing container that expects data to be stored on the local file system and uses regular file system operations to access it.

The following snippets assume a volume mount with a mountPath set to /mnt/my-volume.

Nodejs

Use the File System module to create a new file or append to an existing in the volume, /mnt/my-volume:

var fs = require('fs');
fs.appendFileSync('/mnt/my-volume/sample-logfile.txt', 'Hello logs!', { flag: 'a+' });

Python

Write to a file kept in the volume, /mnt/my-volume:

f = open("/mnt/my-volume/sample-logfile.txt", "a")

Go

Use the os package to create a new file kept in the volume, /mnt/my-volume

f, err := os.Create("/mnt/my-volume/sample-logfile.txt")

Java

Use the Java.io.File class to create a log file in the volume, /mnt/my-volume:

import java.io.File;
File f = new File("/mnt/my-volume/sample-logfile.txt");

View volume mounts settings

Console

In the Google Cloud console, go to the Cloud Run jobs page:

Go to Cloud Run jobs
Click the job you are interested in to open the Job details page.
Click the Volumes tab.
Locate the volume mounts setting in the volumes detail page.

gcloud

Use the following command:
```
gcloud run jobs describe JOB_NAME
```
Locate the volume mounts setting in the returned configuration.