This page describes Cloud Storage FUSE's performance in terms of latency, bandwidth, and retries, as well as best practices when using Cloud Storage FUSE.
Reads and writes
To improve reads and writes, we recommend implementing the following best practices:
Run sequential read and write workloads when possible: Cloud Storage FUSE performs better for sequential read and write workloads than random read and write workloads. Cloud Storage FUSE uses a heuristic to detect when a file is being read sequentially, which enables Cloud Storage FUSE to issue fewer, larger read requests to Cloud Storage using the same TCP connection.
Adjust file sizes based on read type: to optimize sequential read performance, we recommend that you upload and read files between 5 MB and 200 MB in size. To optimize random read performance, we recommend that you upload and read files around 2 MB in size.
Accelerate reads by enabling parallel downloads: accelerate large file reads over 1 GB in size by enabling parallel downloads. For more information, see Improve read performance using parallel downloads.
Improve read performance using parallel downloads
You can improve read performance by enabling and configuring the parallel download feature, which uses multiple workers to download a file in parallel using the file cache directory as a prefetch buffer. We recommend using parallel downloads for single-threaded read scenarios that load large files such as model serving and checkpoint restores.
Before you enable parallel downloads, consider the following:
If your application does high read parallelism over eight threads, you might experience a slight performance degradation.
We don't recommend using parallel downloads for training workloads because of their high read parallelism.
To use parallel downloads, you must first enable and configure the file cache.
The file being read must fit within the file cache directory's available capacity which can be controlled using the
max-size-mb
property.
Configuring parallel downloads
Once you set the enable-parallel-downloads
property to true
, you can
optionally configure its supporting settings using a
Cloud Storage FUSE configuration file:
parallel-downloads-per-file
: the number of maximum workers that can be spawned per file to download the object from Cloud Storage into the file cache. The default value is16
.max-parallel-downloads
: the number of maximum workers that can be spawned at any given time across all file download jobs. The default is set to twice the number of CPU cores on your machine. To specify no limit, enter a value of-1
.download-chunk-size-mb
: the size of each read request in MiB that each worker makes to Cloud Storage when downloading the object into the file cache. The default size is 50 MiB. Note that a parallel download is only triggered if the file being read is the specified size.
Caching
Cloud Storage FUSE offers four types of caching to help increase performance and reduce cost: file caching, stat caching, type caching, and list caching. For more information about these caches, see Overview of caching.
IOPS (queries-per-second)
Filestore is a better option than Cloud Storage FUSE for workloads that require high instantaneous input/output operations per second (IOPS), also known as "queries-per-second" in Cloud Storage. Filestore is also the better option for very high IOPS on a single file system and lower latency.
Alternatively, you can also use the Cloud Storage FUSE file cache feature to build on the underlying cache media's performance characteristics if it provides high IOPS and low latency.
Latency and throughput
Cloud Storage FUSE has higher latency than a local file system. Throughput is reduced when you read or write small files one at a time, as it results in several separate API calls. Reading or writing multiple large files at a time can help increase throughput. Use the Cloud Storage FUSE file cache feature to improve performance for small and random I/Os.
Cloud Storage FUSE's file system latency affects rsync
, which reads and writes only
one file at a time. To transfer multiple files to or from your bucket in
parallel, use the Google Cloud CLI by running gcloud storage rsync
.
For more information, see the rsync
documentation.
Achieve maximum throughput
To achieve maximum throughput, use a machine with enough CPU resources to drive throughput and saturate the network interface card (NIC). Insufficient CPU resources can cause Cloud Storage FUSE throttling.
If you're using Google Kubernetes Engine, increase the CPU allocation to the Cloud Storage FUSE sidecar container if your workloads need higher throughput. You can increase the resources used by the sidecar container or allocate unlimited resources.
Rate limiting
To limit the rate of traffic that Cloud Storage FUSE sends to
Cloud Storage, you can use the following options as part of
your gcsfuse
command:
The
--limit-ops-per-sec
option controls the rate at which Cloud Storage FUSE sends requests to Cloud Storage.The
--limit-bytes-per-sec
option controls bandwidth at which Cloud Storage FUSE downloads data from Cloud Storage.
For more information on these options, see the
gcsfuse
command-line documentation.
All rate limiting is approximate and performed over an 8-hour window. By default, no rate limits are applied.
Directory semantics
Cloud Storage operates with a flat namespace, meaning that
directories don't actually exist within Cloud Storage. Rather,
directories are represented by object names that end with a slash (/
)
(for example, in the object name my-bucket/directory/file.txt
,
my-bucket/directory/
represents a directory). Directories that exist as part
of object names but don't exist as actual objects are known as
implicitly defined directories.
By default, when you mount a bucket with implicitly-defined directories, Cloud Storage FUSE cannot infer or access those directories. Cloud Storage FUSE can only infer explicitly-defined directories, meaning that the directory exists as an actual object within the Cloud Storage bucket.
For example, say you mount a bucket named my-bucket
, which contains the object
my-bucket/directory/file1.txt
. If you run ls
on the bucket mount point,
Cloud Storage FUSE cannot access the my-bucket/directory/
directory or the
file1.txt
object within it, because directory
doesn't exist as an object
in Cloud Storage. To allow Cloud Storage FUSE to infer directories and
the objects within them, you can make the directories explicitly defined by
creating them in your local file system using the mkdir
command, or by
including the --implicit-dirs
option in your gcsfuse
command.
For more information about directory semantics, including how to mount buckets
with existing prefixes, see Files and Directories in the GitHub documentation.
For more information about the --implicit-dirs
option, see the
Cloud Storage FUSE command-line documentation.
Considerations when listing objects
When you list all the objects in a mounted bucket (e.g., by running ls
),
Cloud Storage FUSE calls the Objects: list API on Cloud Storage. The
API paginates results, which means that Cloud Storage FUSE might need to issue
multiple calls, depending on how many objects are in your bucket. Note that this
can make a list operation expensive and slow.
Benchmarks
For instructions on how to perform load tests on Cloud Storage FUSE, see Performance Benchmarks in the GitHub documentation.