This page describes Cloud Storage FUSE's performance in terms of latency, bandwidth, and retries, as well as best practices when using Cloud Storage FUSE.
Reads and writes
Cloud Storage FUSE performs better for sequential read and write workloads than random read and write workloads. Cloud Storage FUSE uses a heuristic to detect when a file is being read sequentially, which enables Cloud Storage FUSE to issue fewer, larger read requests to Cloud Storage using the same TCP connection.
To optimize sequential read performance, it's recommended that you upload and read files that are between 5 MB and 200 MB in size. To optimize random read performance, it's recommended that you upload and read files around 2 MB in size.
Caching
Using Cloud Storage FUSE with file, stat, type, or all three cache types enabled can increase performance and reduce cost, but can also reduce consistency.
Stat and type cache
The stat and type caches can reduce the number of serial calls to Cloud Storage on repeat reads to the same file, which improves performance. Set stat and type caches according to the number of files that have repeat reads and might benefit from caching. You can use the number of files to represent the approximate size of the cache. We recommend the following limits for each cache type:
stat-cache-max-size-mb: use the default value of32if your workload involves up to 20,000 files. If your workload is larger than 20,000 files, increase thestat-cache-max-size-mbvalue by 10 for every additional 6,000 files, around 1,500 bytes per file.stat-cache-max-size-mbis a mount-level limit, and actual memory usage might be lower than the value you specify. Alternatively, you can setstat-cache-max-size-mbto-1to let the stat cache use as much memory as needed.type-cache-max-size-mb: use the default value of4if the maximum number of files within a single directory from the bucket you're mounting contains 20,000 files or less. If the maximum number of files within a single directory that you're mounting contains more than 20,000 files, increase thetype-cache-max-size-mbvalue by1for every 5,000 files, around 200 bytes per file.type-cache-max-size-mbis a mount-level limit, and actual memory usage might be lower than the value specified. Alternatively, you can set thetype-cache-max-size-mbvalue to-1to let the type cache use as much memory as needed.
We recommend that you run a full listing by passing ls -R on your mounted
bucket before you run your workload to improve performance on the first run by
populating the type cache ahead of time in a faster, batched method.
File caching best practices
Ensure that your entire dataset fits into the cache capacity for best performance and to avoid cache thrashing. Also, consider the maximum capacity and performance that your cache media can provide. If you hit the provisioned cache's maximum performance, capacity limit, or both, it's beneficial to read directly from Cloud Storage which has much higher limits than Cloud Storage FUSE.
IOPS (queries-per-second)
Filestore is a better option than Cloud Storage FUSE for workloads that require high instantaneous input/output operations per second (IOPS), also known as "queries-per-second" in Cloud Storage. Filestore is also the better option for very high IOPS on a single file system and lower latency.
Alternatively, you can also use the Cloud Storage FUSE file cache feature to build on the underlying cache media's performance characteristics if it provides high IOPS and low latency.
Latency and throughput
Cloud Storage FUSE has higher latency than a local file system. Throughput is reduced when you read or write small files one at a time, as it results in several separate API calls. Reading or writing multiple large files at a time can help increase throughput. Use the Cloud Storage FUSE file cache feature to improve performance for small and random I/Os.
Cloud Storage FUSE's file system latency affects rsync, which reads and writes only
one file at a time. To transfer multiple files to or from your bucket in
parallel, use the Google Cloud CLI by running gcloud storage rsync.
For more information, see the rsync documentation.
Achieve maximum throughput
To achieve maximum throughput, use a machine with enough CPU resources to drive throughput and saturate the network interface card (NIC). Insufficient CPU resources can cause Cloud Storage FUSE throttling.
If you're using Google Kubernetes Engine, increase the CPU allocation to the Cloud Storage FUSE sidecar container if your workloads need higher throughput. You can increase the resources used by the sidecar container or allocate unlimited resources.
Set the number of threads according to the number of CPU cores available. If
the number of cores or threads is higher than 100, change –max-cons-per-host
to same value. ML frameworks typically use num_workers to define the number
of threads.
Rate limiting
To limit the rate of traffic that Cloud Storage FUSE sends to
Cloud Storage, you can use the following options as part of
your gcsfuse command:
The
--limit-ops-per-secoption controls the rate at which Cloud Storage FUSE sends requests to Cloud Storage.The
--limit-bytes-per-secoption controls bandwidth at which Cloud Storage FUSE downloads data from Cloud Storage.
For more information on these options, see the
gcsfuse command-line documentation.
All rate limiting is approximate and performed over an 8-hour window. By default, no rate limits are applied.
Upload procedure control
By default, failed requests from Cloud Storage FUSE to Cloud Storage are
retried with exponential backoff up to a specified maximum backoff
duration, which has a value of 30s (30 seconds) by default. Once the backoff
duration exceeds the specified maximum duration, the retry continues with the
specified maximum duration. You can use the --max-retry-sleep option as part
of a gcsfuse call to specify the backoff duration.
For more information on the --max-retry-sleep option, see the
gcsfuse command-line documentation.
Directory semantics
Cloud Storage operates with a flat namespace, meaning that
directories don't actually exist within Cloud Storage. Rather,
directories are represented by object names that end with a slash (/)
(for example, in the object name my-bucket/directory/file.txt,
my-bucket/directory/ represents a directory). Directories that exist as part
of object names but don't exist as actual objects are known as
implicitly defined directories.
By default, when you mount a bucket with implicitly-defined directories, Cloud Storage FUSE cannot infer or access those directories. Cloud Storage FUSE can only infer explicitly-defined directories, meaning that the directory exists as an actual object within the Cloud Storage bucket.
For example, say you mount a bucket named my-bucket, which contains the object
my-bucket/directory/file1.txt. If you run ls on the bucket mount point,
Cloud Storage FUSE cannot access the my-bucket/directory/ directory or the
file1.txt object within it, because directory doesn't exist as an object
in Cloud Storage. To allow Cloud Storage FUSE to infer directories and
the objects within them, you can make the directories explicitly defined by
creating them in your local file system using the mkdir command, or by
including the --implicit-dirs option in your gcsfuse command.
For more information about directory semantics, including how to mount buckets
with existing prefixes, see Files and Directories in the GitHub documentation.
For more information about the --implicit-dirs option, see the
Cloud Storage FUSE command-line documentation.
Considerations when listing objects
When you list all the objects in a mounted bucket (e.g., by running ls),
Cloud Storage FUSE calls the Objects: list API on Cloud Storage. The
API paginates results, which means that Cloud Storage FUSE might need to issue
multiple calls, depending on how many objects are in your bucket. Note that this
can make a list operation expensive and slow.
Benchmarks
For instructions on how to perform load tests on Cloud Storage FUSE, see Performance Benchmarks in the GitHub documentation.
Logging
To configure logging for Cloud Storage FUSE activities, including log rotation,
specify the fields under the logging key in a
Cloud Storage FUSE configuration file. By default, log files are rotated and
consume around 1 GiB of space.