Optimize and test instance performance

This page describes the expected average performance and the recommended performance settings for Filestore. It also shows you how you can test the performance of your Filestore instances.

Expected performance

Each Filestore service tier provides a different level of performance. The performance of any given instance might vary from the expected numbers due to various factors, such as the use of caching, the number of client VMs, the machine type of the client VMs, and the workload tested.

The following tables show the expected performance of Filestore instances based on its service tier and configured capacity:

Performance Capacity Read and write IOPS Read and write throughput (MiB/s)
BASIC_HDD 1 TiB to 10 TiB 600/1,000 100/100
BASIC_HDD 10 TiB to 63.9 TiB 1,000/5,000 180/120
BASIC_SSD 2.5 TiB to 63.9 TiB 60,000/25,000 1,200/350
ZONAL (zonal, low capacity range) 1 TiB 9,200/2,600 260/88
ZONAL (zonal, low capacity range) 9.75 TiB 89,700/25,350 2,535/858
HIGH_SCALE_SSD (now zonal, high capacity range) 10 TiB 92,000/26,000 2,600/880
HIGH_SCALE_SSD (now zonal, high capacity range) 100 TiB 920,000/260,000 26,000/8,800
ENTERPRISE 1 TiB 12,000/4,000 120/100
ENTERPRISE 10 TiB 120,000/40,000 1,200/1,000

The previous table shows the expected performance at the minimum and maximum capacity for each service tier. Between these limits, performance scales linearly as the capacity scales. For example, if you double your enterprise instance capacity from 1 TiB to 2 TiB, the expected performance of the instance doubles from 12,000/4,000 read and write IOPS to 24,000/8,000 read and write IOPS.

In single- and few-client scenarios, you must increase the number of TCP connections with the nconnect mount option to achieve maximum NFS performance. We recommend specifying up to 7 connections for the zonal service tier and up to 2 connections for enterprise tier. In general, the larger the file share capacity and the fewer the connecting client VMs, the more performance you gain by specifying additional connections with nconnect.

Recommended client machine type

We recommend having a Compute Engine machine type, such as n2-standard-8, that provides an egress bandwidth of 16 Gbps. This egress bandwidth allows the client to achieve approximately 16 Gbps read bandwidth for cache-friendly workloads. For additional context, see Network bandwidth.

Linux client mount options

We recommend using the following NFS mount options, especially hard mount, async, and the rsize and wsize options, to achieve the best performance on Linux client VM instances. For more information on NFS mount options, see nfs.

Default option Description
hard The NFS client retries NFS requests indefinitely.
timeo=600 The NFS client waits 600 deciseconds (60 seconds) before retrying an NFS request.
retrans=3 The NFS client attempts NFS requests three times before taking further recovery action.
rsize=262144 The NFS client can receive a maximum of 262,144 bytes from the NFS server per READ request.
Note: For basic-tier instances, set the rsize value to 1048576.
wsize=1048576 The NFS client can send a maximum of 1,048,576 bytes (1 MiB) to the NFS server per WRITE request.
resvport The NFS client uses a privileged source port when communicating with the NFS server for this mount point.
async The NFS client delays sending application writes to the NFS server until certain conditions are met.
Caution: Using the sync option significantly reduces performance.

Single and multiple client VM performance

Filestore's scalable service tiers are performance optimized for multiple client VMs, not a single client VM.

For enterprise- and zonal-tier instances, at least four client VMs are needed to take advantage of full performance. This ensures that all of the VMs in the underlying Filestore cluster are fully utilized.

For added context, the smallest scalable Filestore cluster has four VMs. Each client VM communicates with just one Filestore cluster VM, regardless of the number of NFS connections per client specified using the nconnect mount option. If using a single client VM, read and write operations are only performed from a single Filestore cluster VM.

Improve performance across Google Cloud resources

Operations across multiple Google Cloud resources, such as copying data from Cloud Storage to a Filestore instance using gsutil, can be slow. To help mitigate performance issues, try the following:

  • Ensure the Cloud Storage bucket, client VM, and Filestore instance all reside in the same region.

    Dual-regions provide a maximally-performant option for data stored in Cloud Storage. If using this option, ensure the other resources reside in one of the single regions contained in the dual-region. For example, if your Cloud Storage data resides in us-central1,us-west1, ensure that your client VM and Filestore instance reside in us-central1.

  • For a point of reference, verify the performance of a PD-attached VM and compare to the performance of a Filestore instance.

    • If the PD-attached VM is similar or slower in performance when compared to the Filestore instance, this might indicate a performance bottleneck unrelated to Filestore. Try the following to improve the baseline performance of your non-Filestore resources:

      • Apply the gsutil -m option. This option allows supported operations including cp, mv, and rsync to run in parallel.

      • Apply the following gsutil configurations. Each are mutable boto configuration values set in the boto configuration file. Users can modify the file directly (recommended) or by using the -o option:

    • If the performance of the Filestore instance is notably slower than the PD-attached VM, try spreading the operation over multiple VMs.

      • This helps to improve performance of read operations from Cloud Storage.

      • For enterprise- and zonal-tier instances, at least four client VMs are needed to take advantage of full performance. This ensures that all of the VMs in the underlying Filestore cluster are fully utilized. For more information, see Single and multiple client VM performance.

Testing performance

If you are using Linux, you can use the fio tool to benchmark read and write throughput and IOPS for basic tier instances. Note that this method for benchmarking performance is not recommended for enterprise and zonal tier instances.

The examples in this section show common benchmarks you might want to run. You may need to run fio from multiple client VM instances to achieve maximum performance.

The following example benchmarks maximum write throughput:

fio --ioengine=libaio --filesize=32G --ramp_time=2s \
--runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --directory=/mnt/nfs  \
--name=write --blocksize=1m --iodepth=64 --readwrite=write

The following example benchmarks maximum write IOPS:

fio --ioengine=libaio --filesize=32G --ramp_time=2s \
--runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --directory=/mnt/nfs  \
--name=randwrite --blocksize=4k --iodepth=256 --readwrite=randwrite

The following example benchmarks maximum read throughput:

fio --ioengine=libaio --filesize=32G --ramp_time=2s \
--runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --directory=/mnt/nfs  \
--name=read --blocksize=1m --iodepth=64 --readwrite=read

The following example benchmarks maximum read IOPS:

fio --ioengine=libaio --filesize=32G --ramp_time=2s \
--runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --directory=/mnt/nfs  \
--name=randread --blocksize=4k --iodepth=256 --readwrite=randread

What's next