Optimizing local SSD performance

Optimizing Local SSDs

The performance by disk type chart describes the maximum achievable performance for local SSD partitions. To optimize your apps and VM instances to achieve these speeds, use the following best practices:

Using guest environment optimizations for Local SSDs

By default, most Compute Engine-provided Linux images automatically run an optimization script that configures the instance for peak Local SSD performance. The script enables certain queue sysfs settings that enhance the overall performance of your machine and mask interrupt requests (IRQs) to specific virtual CPUs (vCPUs). This script only optimizes performance for Compute Engine local SSD partitions.

Ubuntu, SLES, and other earlier images might not be configured to include this performance optimization. If you are using any of these images or an image that is earlier than v20141218, you can install the guest environment to enable these optimizations.

Select the best image for NVMe or SCSI interfaces

Local SSDs can expose either an NVMe or SCSI interface, and the best choice depends on the operating system you are using. Choose an interface for your local SSD partitions that works best with your boot disk image. If your instances connect to local SSDs using SCSI interfaces, you can enable multi-queue SCSI on the guest OS to achieve optimal performance over the SCSI interface.

Enable multi-queue SCSI on instances with custom images and Local SSDs

Some public images support multi-queue SCSI. If you require multi-queue SCSI capability on custom images that you import to your project, you must enable it yourself. Your imported Linux images can use multi-queue SCSI only if they include kernel version 3.19 or later.

To enable multi-queue SCSI on a custom image, import the image with the VIRTIO_SCSI_MULTIQUEUE guest OS feature enabled and add an entry to your GRUB config:

CentOS

For CentOS7 only.

1. Import your custom image using the API and include a guestOsFeatures item with a type value of VIRTIO_SCSI_MULTIQUEUE.

2. Create an instance using your custom image and attach one or more local SSDs.

3. Connect to your instance through SSH.

4. Check the value of the /sys/module/scsi_mod/parameters/use_blk_mq file

   $ cat /sys/module/scsi_mod/parameters/use_blk_mq
   

   If the value of this file is Y, then multi-queue SCSI is already enabled on your imported image. If the value of the file is N, include scsi_mod.use_blk_mq=Y in the GRUB_CMDLINE_LINUX entry in your GRUB config file and restart the system.

   1. Open the /etc/default/grub GRUB config file in a text editor.

      $ sudo vi /etc/default/grub
      

   2. Add scsi_mod.use_blk_mq=Y to the GRUB_CMDLINE_LINUX entry.

      GRUB_CMDLINE_LINUX=" vconsole.keymap=us console=ttyS0,38400n8 vconsole.font=latarcyrheb-sun16 scsi_mod.use_blk_mq=Y"
      

   3. Save the config file.

   4. Run the grub2-mkconfig command to regenerate the GRUB file and complete the configuration.

      $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      

   5. Reboot the instance.

      $ sudo reboot
      

Ubuntu

1. Import your custom image using the Compute Engine API and include a guestOsFeatures item with a type value of VIRTIO_SCSI_MULTIQUEUE.

2. Create an instance using your custom image and attach one or more local SSDs using the SCSI interface.

3. Connect to your instance through SSH.

4. Check the value of the /sys/module/scsi_mod/parameters/use_blk_mq file.

   $ cat /sys/module/scsi_mod/parameters/use_blk_mq
   

   If the value of this file is Y, then multi-queue SCSI is already enabled on your imported image. If the value of the file is N, include scsi_mod.use_blk_mq=Y in the GRUB_CMDLINE_LINUX entry in your GRUB config file and restart the system.

   1. Open the sudo nano /etc/default/grub GRUB config file in a text editor.

      $ sudo nano /etc/default/grub
      

   2. Add scsi_mod.use_blk_mq=Y to the GRUB_CMDLINE_LINUX entry.

      GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=Y"
      

   3. Save the config file.

   4. Run the update-grub command to regenerate the GRUB file and complete the configuration.

      $ sudo update-grub
      

   5. Reboot the instance.

      $ sudo reboot
      

Disable write cache flushing

File systems, databases, and other apps use cache flushing to ensure that data is committed to durable storage at various checkpoints. For most storage devices, this default makes sense. However, write cache flushes are fairly slow on local SSDs. You can increase the write performance for some apps by disabling automatic flush commands in those apps or by disabling flush options at the file system level.

Local SSDs always flush cached writes within two seconds regardless of the flush commands that you set for your file systems and apps, so temporary hardware failures can cause you to lose only two seconds of cached writes at most. Permanent hardware failures can still cause loss of all data on the device whether the data is flushed or not, so you should still back up critical data to Persistent Disks or Cloud Storage buckets.

To disable write cache flushing on ext4 file systems, include the nobarrier setting in your mount options or in your /etc/fstab entries. For example:

$ sudo mount -o discard,defaults,nobarrier /dev/[LOCAL_SSD_ID] /mnt/disks/[MNT_DIR]

where: [LOCAL_SSD_ID] is the device ID for the local SSD that you want to mount and [MNT_DIR] is the directory in which to mount it.

Benchmarking Local SSD performance

The Local SSD performance figures provided in the Performance section were achieved by using specific settings on the local SSD instance. If your instance is having trouble reaching these performance limits and you have already configured the instance using the recommended Local SSD settings, you can compare your performance limits against the published limits by replicating the settings used by the Compute Engine team.

Warning: The script used in this section is intended for benchmarking and performance comparisons only and is not intended to optimize your disk for performance. We strongly recommend against running this script on a VM with a local SSD where you want to keep the data because this script discards any data on your local SSD.

These instructions assume that you are using a Linux operating system with the apt package manager installed.

Note: These instructions create local SSDs connected to an instance using the NVMe interface. If you use the SCSI interface instead, replace the disk location /dev/disk/by-id/google-local-nvme-ssd-0 with /dev/disk/by-id/google-local-ssd-0 in the following commands.

Create a virtual machine with 8 vCPUs and one Local SSD partition

1. Create a local SSD instance that has four or eight vCPUs for each device, depending on your workload. For example, if you want to attach four local SSD partitions to an instance, use a machine type with 16 or 32 vCPUs.

   The following command creates a virtual machine with 8 vCPUs and a single local SSD:

   gcloud compute instances create ssd-test-instance \
   --machine-type "n1-standard-8" \
   --local-ssd interface=nvme
   

2. Run the following script on your VM. The script replicates the settings used to achieve the SSD performance figures provided in the performance section. Note that the --bs parameter defines the block size, which affects the results for different types of read and write operations.

   # install dependencies
   sudo apt-get update -y
   sudo apt-get install -y build-essential git libtool gettext autoconf \
   libgconf2-dev libncurses5-dev python-dev fio bison autopoint
   
   # blkdiscard
   git clone https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git
   cd util-linux/
   ./autogen.sh
   ./configure --disable-libblkid
   make
   sudo mv blkdiscard /usr/bin/
   sudo blkdiscard /dev/disk/by-id/google-local-nvme-ssd-0
   
   # full write pass - measures write bandwidth with 1M blocksize
   sudo fio --name=writefile --size=100G --filesize=100G \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --bs=1M --nrfiles=1 \
   --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
   --iodepth=200 --ioengine=libaio
   
   # rand read - measures max read IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=4 --rw=randread --blocksize=4k --group_reporting
   
   # rand write - measures max write IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
   

Create a virtual machine with 32 vCPUs and 24 Local SSD partitions

1. If you want to attach 24 local SSD partitions to an instance (available in beta), use a machine type with 32 or more vCPUs.

   The following command creates a virtual machine with 32 vCPUs and 24 local SSD partitions:

   gcloud beta compute instances create ssd-test-instance \
   --machine-type "n1-standard-32" \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme \
   --local-ssd interface=nvme
   

2. Run the following script on your VM. The script replicates the settings used to achieve the SSD performance figures provided in the performance section. Note that the --bs parameter defines the block size, which affects the results for different types of read and write operations.

   # install dependencies
   sudo apt update -y
   sudo apt install -y build-essential git libtool gettext autoconf \
   libgconf2-dev libncurses5-dev python-dev fio bison autopoint
   
   # blkdiscard
   git clone https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git
   cd util-linux/
   ./autogen.sh
   ./configure --disable-libblkid
   make
   sudo mv blkdiscard /usr/bin/
   sudo blkdiscard /dev/disk/by-id/google-local-nvme-ssd-0
   
   # full write pass - measures write bandwidth with 1M blocksize
   sudo fio --name=writefile --size=100G --filesize=100G \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --bs=1M --nrfiles=1 \
   --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
   --iodepth=200 --ioengine=libaio
   
   # rand read - measures max read IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=32 --rw=randread --blocksize=4k --group_reporting --norandommap
   
   # rand write - measures max write IOPS with 4k blocks
   sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
   --filename=/dev/disk/by-id/google-local-nvme-ssd-0 --ioengine=libaio --randrepeat=0 \
   --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
   --numjobs=32 --rw=randwrite --blocksize=4k --group_reporting --norandommap
   

What's next

• Learn about Local SSD pricing.