Benchmarking local SSD performance

Local SSD performance limits provided in the Choose a storage option section were achieved by using specific settings on the local SSD instance. If your virtual machine (VM) instance is having trouble reaching these performance limits and you have already configured the instance using the recommended local SSD settings, you can compare your performance limits against the published limits by replicating the settings used by the Compute Engine team.

Warning: The script used in this section is intended for benchmarking and performance comparisons only and is not intended to optimize your disk for performance. We strongly recommend against running this script on a VM with a local SSD where you want to keep the data because this script discards any data on your local SSD.

These instructions assume that you are using a Linux operating system with the apt package manager installed.

Note: These instructions create local SSDs connected to an instance using the NVMe interface. If you use the SCSI interface instead, replace the disk location /dev/disk/by-id/google-local-nvme-ssd-0 with /dev/disk/by-id/google-local-ssd-0 in the following commands.

Create a virtual machine with 8 vCPUs and one local SSD partition

1. Create a local SSD instance that has four or eight vCPUs for each device, depending on your workload. For example, if you want to attach four local SSD partitions to an instance, use a machine type with 16 or 32 vCPUs.

   The following command creates a virtual machine with 8 vCPUs and a single local SSD:

   gcloud compute instances create ssd-test-instance \
       --machine-type "n1-standard-8" \
       --local-ssd interface=nvme
   

2. Run the following script on your VM. The script replicates the settings used to achieve the SSD performance figures provided in the performance section. Note that the --bs parameter defines the block size, which affects the results for different types of read and write operations.

   # install tools
   sudo apt-get -y update
   sudo apt-get install -y fio util-linux
   
   # discard local SSD sectors
   sudo blkdiscard /dev/disk/by-id/google-local-nvme-ssd-0
   
   # full write pass - measures write bandwidth with 1M blocksize
   sudo fio --name=writefile --size=100G --filesize=100G \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --bs=1M --nrfiles=1 \
   --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
   --iodepth=200 --ioengine=libaio
   
   # rand read - measures max read IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=4 --rw=randread --blocksize=4k --group_reporting
   
   # rand write - measures max write IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
   

Create a virtual machine with 32 vCPUs and 24 local SSD partitions

1. If you want to attach 24 local SSD partitions to an instance, use a machine type with 32 or more vCPUs.

   The following command creates a virtual machine with 32 vCPUs and 24 local SSD partitions:

   gcloud compute instances create ssd-test-instance \
   --machine-type "n1-standard-32" \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme
   

2. Install the mdadm tool. The install process for mdadm includes a user prompt that halts scripts, so run the process manually:

   Debian and Ubuntu

   sudo apt update && sudo apt install mdadm --no-install-recommends
   

   CentOS and RHEL

   sudo yum install mdadm -y
   

   SLES and openSUSE

   sudo zypper install -y mdadm
   

3. Use the lsblk command to identify all of the local SSDs that you want to mount together:

   lsblk
   

   The output looks similar to the following:

   NAME    MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
   sda       8:0     0   10G  0 disk
   └─sda1    8:1     0   10G  0 part /
   nvme0n1 259:0     0  375G  0 disk
   nvme0n2 259:1     0  375G  0 disk
   nvme0n3 259:2     0  375G  0 disk
   nvme0n4 259:3     0  375G  0 disk
   nvme0n5 259:4     0  375G  0 disk
   nvme0n6 259:5     0  375G  0 disk
   nvme0n7 259:6     0  375G  0 disk
   nvme0n8 259:7     0  375G  0 disk
   nvme0n9 259:8     0  375G  0 disk
   nvme0n10 259:9    0  375G  0 disk
   nvme0n11 259:10   0  375G  0 disk
   nvme0n12 259:11   0  375G  0 disk
   nvme0n13 259:12   0  375G  0 disk
   nvme0n14 259:13   0  375G  0 disk
   nvme0n15 259:14   0  375G  0 disk
   nvme0n16 259:15   0  375G  0 disk
   nvme0n17 259:16   0  375G  0 disk
   nvme0n18 259:17   0  375G  0 disk
   nvme0n19 259:18   0  375G  0 disk
   nvme0n20 259:19   0  375G  0 disk
   nvme0n21 259:20   0  375G  0 disk
   nvme0n22 259:21   0  375G  0 disk
   nvme0n23 259:22   0  375G  0 disk
   nvme0n24 259:23   0  375G  0 disk
   

   Local SSDs in SCSI mode have standard names similar to sdb. Local SSDs in NVMe mode have names similar to nvme0n1.

4. Use the mdadm tool to combine multiple local SSD devices into a single array named /dev/md0. This example merges twenty four local SSD devices in NVMe mode. For local SSD devices in SCSI mode, specify the names that you obtained from the lsblk command:

   sudo mdadm --create /dev/md0 --level=0 --raid-devices=24 \
   /dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4 \
   /dev/nvme0n5 /dev/nvme0n6 /dev/nvme0n7 /dev/nvme0n8 \
   /dev/nvme0n9 /dev/nvme0n10 /dev/nvme0n11 /dev/nvme0n12 \
   /dev/nvme0n13 /dev/nvme0n14 /dev/nvme0n15 /dev/nvme0n16 \
   /dev/nvme0n17 /dev/nvme0n18 /dev/nvme0n19 /dev/nvme0n20 \
   /dev/nvme0n21 /dev/nvme0n22 /dev/nvme0n23 /dev/nvme0n24
   

   The response is similar to the following:

   mdadm: Defaulting to version 1.2 metadata
   mdadm: array /dev/md0 started.
   

5. Run the following script on your VM. The script replicates the settings used to achieve the SSD performance figures provided in the performance section. that the --bs parameter defines the block size, which affects the results for different types of read and write operations.

   # install tools
   sudo apt-get -y update
   sudo apt-get install -y fio util-linux
   
   # full write pass - measures write bandwidth with 1M blocksize
   sudo fio --name=writefile --size=100G --filesize=100G \
   --filename=/dev/md0 --bs=1M --nrfiles=1 \
   --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
   --iodepth=200 --ioengine=libaio
   
   # rand read - measures max read IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/md0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=32 --rw=randread --blocksize=4k --group_reporting --norandommap
   
   # rand write - measures max write IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/md0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=32 --rw=randwrite --blocksize=4k --group_reporting --norandommap
   

What's next

• Learn about local SSD pricing.