Create an HPC-ready VM instance

Tightly coupled high performance computing (HPC) workloads often use the Message Passing Interface (MPI) to communicate between processes and virtual machine (VM) instances. But building your own VM image that is tuned for optimal MPI performance requires systems expertise, Google Cloud knowledge, and extra time for maintenance. To quickly set up an optimized environment for your HPC workloads, use the HPC VM image.

The HPC VM image is a Rocky Linux 8 based VM image that is optimized for tightly coupled HPC workloads. It includes pre-configured kernel and network tuning parameters required to create VM instances that achieve optimal MPI performance on Google Cloud.

For best results, deploy the HPC VM image on a machine type from an HPC-optimized machine series, such as the H4D or H3 series, or a Compute-optimized machine series, such as C2 or C2D. These machine types are designed for tightly-coupled HPC applications. To use RDMA with H4D, you must use version 20241125 or later of the HPC VM image.

You can create an HPC-ready VM instance by using the following options:

Benefits

The HPC VM image provides the following benefits:

  1. VM instances ready for HPC workloads out-of-the-box. There is no need to manually tune performance, manage VM instance reboots, or stay up to date with the latest Google Cloud updates for tightly coupled HPC workloads.
  2. Networking optimizations for tightly-coupled workloads. Optimizations that reduce latency for small messages are included, which benefits applications that are heavily dependent on point-to-point and collective communications. If using the H4D machine series, the HPC VM image contains the necessary Cloud RDMA drivers.
  3. Compute optimizations for HPC workloads. Optimizations that reduce system jitter are included, which makes single-node high performance more predictable.
  4. Consistent, reproducible performance. VM image standardization gives you consistent, reproducible application-level performance.
  5. Improved application compatibility. Alignment with the node-level requirements of the Intel HPC platform specification enables a high degree of interoperability between systems.

HPC VM image features

The HPC VM image offers several features that are designed to optimize performance for high performance computing (HPC) workloads:

  • Automatic updates disabled
  • Intel MPI collective tunings
  • Pre-installed RPMs

Automatic updates disabled

Automatic updates can have a negative impact on the performance of HPC workloads. Automatic updates can be disabled when using the HPC VM images by setting the google_disable_automatic_updates metadata entry to TRUE when creating a VM instance. How this metadata entry is set during instance creation depends on the tool that you use to create the instance.

For example, when using the gcloud compute instances create command to create a VM instance, provide the --metadata argument. For more information, see About VM metadata.

Intel MPI Library

Google recommends that you use the Intel MPI 2021 library for running MPI jobs on Google Cloud. For more information, see the following sections:

Pre-installed RPMs

The HPC VM image comes with the following RPM packages pre-installed:

  • daos-client
  • gcc-gfortran
  • gcc-toolset-12
  • Lmod
  • dkms
  • htop
  • hwloc
  • hwloc-devel
  • infiniband-diags
  • kernel-devel
  • kmod-idpf-irdma
  • libfabric
  • librdmacm-utils
  • libibverbs-utils
  • libXt
  • ltrace
  • nfs-utils
  • numactl
  • numactl-devel
  • papi
  • pciutils
  • pdsh
  • perf
  • perftest
  • rdma-core
  • redhat-lsb-core
  • redhat-lsb-cxx
  • rsh
  • screen
  • strace
  • wget
  • zsh
  • "Development Tools" package group

Quickstarts

These tutorials describe the steps to set up an HPC-optimized VM instance. The procedures describe how to:

  • Create an HPC VM instance (without configuring Cloud RDMA)
  • Specify a compact placement policy when creating HPC VM instances
  • Create an HPC VM instance that uses Cloud RDMA (Preview)

You can't add Cloud RDMA to an existing HPC VM instance, so follow the appropriate steps when creating your instance.

Before you begin

  1. To use the Google Cloud CLI for this quickstart, you must first install and initialize the Google Cloud CLI:
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Create an HPC VM instance

We strongly recommend choosing a machine type from the following machine series:

These VMs have fixed virtual-to-physical core mapping and expose NUMA cell architecture to guest OS, both of which are critical for performance of tightly coupled HPC applications.

Console

  1. In the Google Cloud console, go to the HPC VM Cloud Marketplace page. Go to the HPC VM Cloud Marketplace page

  2. Click Get started.

  3. On the HPC VM deployment page, enter a Deployment name. This name becomes the root of your VM name. Compute Engine appends -vm to this name when naming your instance.

  4. Choose a Zone and Machine type. For this quickstart, you can leave all settings as they are or change them.

  5. Leave the Boot disk type, Boot disk size, and Network interface at their default settings.

  6. Click Deploy.

After the VM instance creation completes, the Cloud Deployment Manager opens, where you can manage your HPC VM and other deployments.

gcloud

Create an HPC VM by using the instances create command. If you are creating multiple, interconnected HPC VM instances, then create HPC VMs with compact placement policies to achieve low network latency.

To create an HPC VM instance without a placement policy, use a command similar to the following:

gcloud compute instances create INSTANCE_NAME \
        --zone=ZONE \
        --image-family=IMAGE_FAMILY \
        --image-project=cloud-hpc-image-public \
        --maintenance-policy=TERMINATE \
        --machine-type=MACHINE_TYPE

Replace the following:

  • INSTANCE_NAME: a name for the HPC VM instance.
  • ZONE: the zone to create the instance in.
  • IMAGE_FAMILY: The image family of the image to use when creating the VM instances. Use hpc-rocky-linux-8 for the latest Rocky Linux 8 based image.
  • MACHINE_TYPE: the machine type to use when creating the VM instance.

After some time, the VM instance creation completes. To verify the instance configuration and to see its status, run the following command:

gcloud compute instances describe INSTANCE_NAME

Create HPC VM instances with compact placement policies

You can reduce the latency between VM instances by creating a compact placement policy. A compact placement policy ensures that instances in the same availability zone are located close to each other.

If you need more VM instances than can fit in a single compact placement policy, then divide your instances into multiple placement policies. Use the minimum number of placement policies that can fit all your instances.

To create HPC VM instances that specify a compact placement policy, follow these steps:

  1. Create a compact placement policy.

  2. Do one of the following:

Create a HPC VM instance that uses Cloud RDMA

To create a HPC VM instance that uses Cloud RDMA, you must first create at least one regular VPC network and one Falcon VPC network. The Falcon VPC network uses an RDMA network profile that enables RDMA traffic between compute instances. This network is separate from the regular VPC network that carries non-RDMA traffic to other Google Cloud services or the internet.

To create a HPC VM instance that uses Cloud RDMA, the tasks to complete are as follows:

  1. Identify or create at least two VPC networks:

    • A regular VPC network for the traffic that goes through the gVNIC network interface
    • A Falcon VPC network for the RDMA traffic

  2. Create a HPC VM instance.

    1. For the source image, use the HPC VM image. This image includes the drivers needed for Cloud RDMA.

    2. During the instance creation, configure at least two network interfaces—one that uses the gVNIC driver and one that uses the IRDMA driver.

      For a more complete explanation, see Create an instance that uses Cloud RDMA.

  3. If you plan to run MPI applications on the HPC VM instances that use Cloud RDMA, then following the MPI configuration steps in Set up and scale MPI applications on H4D VMs with Cloud RDMA.

Access the HPC VM instance

After you create the HPC VM instance, it starts automatically. To access the instance, do one of the following:

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Click the name of your VM instance.

  3. In the Remote Access section, click the first drop-down list and choose how you want to access the instance.

Compute Engine propagates your SSH keys and creates your user. For more information, see Connecting to Linux VMs.

gcloud

To access the instance using SSH, use the gcloud compute ssh command:

gcloud compute ssh INSTANCE_NAME

Compute Engine propagates your SSH keys and creates your user. For more information, see Connecting to instances.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in these quickstarts, delete any HPC VM instances that you created.

Console

  1. In the Google Cloud console, go to the Deployments page.

    Go to Deployments

  2. Select the checkbox next to the HPC VM deployment.

  3. Click Delete.

gcloud

Use the instances delete command:

gcloud compute instances delete INSTANCE_NAME

Configure your HPC VM instance according to best practices

To get better and more predictable performance for your HPC VM instance, we recommend that you use the following best practices.

Disable simultaneous multithreading

The HPC VM image enables simultaneous multithreading (SMT), also known as Hyper-Threading on Intel processors, by default. Disabling SMT can make your performance more predictable and can decrease job times.

You can use the following methods to disable SMT:

  • To disable SMT while creating a new HPC VM, follow the steps to create an HPC VM and include the flag --threads-per-core=1.

  • To disable SMT on an existing HPC VM, connect to the VM and run the following command from the VM:

    sudo google_mpi_tuning --nosmt

For more information, see Set the number of threads per core.

Configure gVNIC as the network interface type for C2 and C2D instances

The HPC VM image supports both Virtio-net and Google Virtual NIC (gVNIC) as virtual network interfaces. Using gVNIC instead of Virtio-net can improve the scalability of MPI applications by providing better communication performance and higher throughput. Additionally, gVNIC is a prerequisite for per VM Tier_1 networking performance, which provides higher bandwidth and allows for higher throughput.

If you create a new C2 or C2D instance, then, by default, Virtio-net is used for the virtual network interface. To use gVNIC, follow the steps to create an HPC VM and do one of the following:

Console

To set gVNIC as the network interface, when creating the instance, in the navigation menu, click Networking. In the Networking pane that appears, do the following:

  1. Go to the Network interfaces section
  2. For Network interface card, select gVNIC.

gcloud

Include the --network-interface=nic-type=GVNIC flag in the gcloud compute instances create command.

The HPC VM image includes the gVNIC driver as a Dynamic Kernel Module Support (DKMS). For more information, see Using Google Virtual NIC.

Turn off Meltdown and Spectre mitigations

The HPC VM image enables the Meltdown and Spectre mitigations by default. In some cases, these mitigations might result in workload-specific performance degradation. To disable these mitigations and incur the associated security risks, do the following:

  1. Run the following command on your HPC instance:

    sudo google_mpi_tuning --nomitigation

  2. Reboot the instance.

Improve network performance

To improve the network performance of your instance, set up one or more of the following configurations:

  • Configure a higher bandwidth. To configure Tier_1 networking for C2 or C2D VMs, use the gcloud compute instances create command to create the VM and specify the --network-performance-configs flag. For more information, see Creating a VM with high-bandwidth configuration.

  • Use jumbo frames. To help minimize the processing overhead for network packets, we recommend using a larger packet size. You need to validate larger packet sizes for the specifics of your application. For information about the use of jumbo frames and packet sizes, see Maximum transmission unit guide.

  • Increase the TCP memory limits. Higher bandwidth requires larger TCP memory. Follow the steps to increase tcp_*mem settings.

  • Use the network-latency profile. Evaluate your application's latency and enable busy polling that reduces latency in the network receive path. Adjust the net.core.busy_poll and net.core.busy_read settings in /etc/sysctl.conf, or use tuned-adm.

Use Intel MPI 2021

Google recommends to use the Intel MPI 2021 library for running MPI jobs on Google Cloud.

MPI implementations have many internal configuration parameters that can affect communication performance. These parameters are especially relevant for MPI Collective communication, which lets you specify algorithms and configuration parameters that can perform very differently in the Google Cloud environment.

The HPC VM image includes a utility, google-hpc-compute, that installs the recommended MPI libraries and uses Google Cloud tailored libfabric providers over the TCP transport.

Limitations

The benefits of tuning vary from application to application. In some cases, a particular tuning might have a negative effect on performance. Consider benchmarking your applications to find the most efficient or cost-effective configuration.

Use google-hpc-compute utility for Intel MPI 2021 support

The google_install_intelmpi script is the MPI related tool in the Google-hpc-compute utility. It helps to install and configure Intel MPI.

The google-hpc-compute utility is included in the HPC VM image.

Install Intel MPI 2021

To install the Intel MPI library while creating a new HPC VM, follow the steps to create an HPC VM and include the following when creating the VM instance:

--metadata=google_install_intelmpi="--impi_2021"

To install the library on an existing HPC VM, run the following command on that VM:

sudo google_install_intelmpi --impi_2021 --install_dir=PATH_INSTALL_MPI

The default location for install_dir is set to /opt/intel.

Intel MPI Library 2018 and MPI collective tunings

The HPC VM image includes Intel MPI collective tunings performed on c2-standard-60 and c2d-standard-112 instances using compact placement policies. These tunings are available in the directory /usr/share/google-hpc-compute/mpitune-configs/intelmpi-2018.

To apply these tuning configurations on an installed MPI library, source the mpivars shell script to set up the proper environment. You can then install the tunings directly with the following command (use the --sudo option if you need root access to the directory): google_install_mpitune

You can use mpitune to manually specify the algorithms and configuration parameters for MPI Collective communication and generate configuration files.

For example, to tune for 22 VM instances and 30 vCPUs per instance, source the mpivars shell script to set up the proper environment, and then run the following command. You must have write access to the directory or run the command as the root user.

mpitune -hf hostfile -fl 'shm:tcp' -pr 30:30 -hr 22:22

This command generates a configuration file in the Intel MPI directory which can be used later to run applications. To make use of the tuning configuration for an application, add the -tune option to mpirun command line, for example:

mpirun -tune -hostfile HOSTFILE -genv I_MPI_FABRICS 'shm:tcp' -np 660 -ppn 30 ./APPLICATION_NAME

Replace the following:

  • HOSTFILE: The name of a file in the guest OS that lists the network names of the VM instances
  • APPLICATION_NAME: The name of the application file to run

Create a custom image using the HPC VM image

To implement the best practices at scale, create a custom OS image to use with your HPC VM instances. Complete the following tasks to create a custom image to use when creating HPC VM instances:

  1. Create a VM that uses the HPC VM image.

  2. Customize the VM with MPI tunings.

  3. Create a custom image using the boot disk of your HPC VM image as the source disk. You can do so using the Google Cloud console or the Google Cloud CLI.

Console

  1. In the Google Cloud console, go to the Images page.

    Go to Images

  2. Click Create image.

  3. Specify a Name for your image.

  4. Under Source disk, select the name of the boot disk on your HPC VM.

  5. Choose other remaining properties for your image.

  6. Click Create.

gcloud

Create the custom image by using the images create command.

gcloud compute images create IMAGE_NAME \
         --source-disk=INSTANCE_NAME \
         --source-disk-zone=VM_ZONE \
         --family=IMAGE_FAMILY \
         --storage-location=LOCATION

Replace the following:

  • IMAGE_NAME: name for the custom image.
  • INSTANCE_NAME: name of your HPC VM.
  • INSTANCE_ZONE: zone where your HPC VM is located.
  • IMAGE_FAMILY: optional. The image family this image belongs to.
  • LOCATION: optional. Region in which to store the custom image. The default location is the multi-region closest to the location of the source disk.

Pricing

The HPC VM image is available at no additional cost. Because the HPC VM image runs on Compute Engine, you might incur charges for Compute Engine resources such as vCPUs, disks, and memory. To learn more, see Compute Engine pricing.

What's next