Introduction
Tightly coupled high performance computing (HPC) workloads often use the Message Passing Interface (MPI) to communicate between processes and virtual machine (VM) instances. But building your own VM image that is tuned for optimal MPI performance requires systems expertise, Google Cloud knowledge, and extra time for maintenance. To quickly create VM instances for your HPC workloads, you can use the HPC VM image.
The HPC VM image is a CentOS 7 based VM image that is optimized for tightly coupled HPC workloads. It includes pre-configured kernel and network tuning parameters required to create VM instances that achieve optimal MPI performance on Google Cloud.
Benefits
The HPC VM image provides the following benefits:
- VMs ready for HPC out-of-the-box: There is no need to manually tune performance, manage VM reboots, or stay up to date with the latest Google Cloud updates for tightly coupled HPC workloads.
- Networking optimizations for tightly-coupled workloads: Optimizations that reduce latency for small messages are included, which benefits applications that are heavily dependent on point-to-point and collective communications.
- Compute optimizations for HPC workloads: Optimizations that reduce system jitter are included, which makes single-node high performance more predictable.
- Consistent, reproducible performance: VM image standardization gives you consistent, reproducible application-level performance.
Features
Google recommends the following optimizations for HPC-ready VMs. The HPC VM image already includes these optimizations, so you don't have to configure them yourself:
- Disabled hyper-threading
- Adjusted user limits
- Increased tcp_*mem settings
- Network-latency profile in use
- Disabled Linux firewalls
- Disabled SELinux
The HPC VM image also includes Intel MPI collective tunings performed on
c2-standard-60
instances using compact placement policies.
Quickstarts
You can create an HPC-ready VM by using the gcloud
command-line tool
or the Google Cloud Console. In the console, the image is available through Cloud
Marketplace.
Before you begin
- To use the
gcloud
command-line tool for this quickstart, you must first install and initialize the Cloud SDK: - In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
Creating an HPC VM instance
Create the VM
Console
In the Google Cloud Console, go to the HPC VM Cloud Marketplace page. Go to the HPC VM Cloud Marketplace page
Click Launch.
On the HPC VM deployment page, enter a Deployment name. This name becomes the root of your VM name. Compute Engine appends
-vm
to this name when naming your instance.Choose a Zone and Machine type. For this quickstart, you can leave all settings as they are or change them. We strongly recommend choosing a C2 machine type. To learn why, read Use compute-optimized instances.
Leave the Boot disk type, Boot disk size, and Network interface at their default settings.
Click Deploy.
After the VM instance creation completes, the Cloud Deployment Manager opens, where you can manage your HPC VM and other deployments.
gcloud
Create an HPC VM by using the instances create
command.
We strongly recommend that you create HPC VMs using compact placement
policies
to achieve low network latency.
gcloud compute instances create INSTANCE_NAME \ --zone=ZONE \ --image-family=hpc-centos-7 \ --image-project=cloud-hpc-image-public \ --maintenance-policy=TERMINATE \ --machine-type=MACHINE_TYPE
Replace the following:
INSTANCE_NAME
: Name of the HPC VM to create.ZONE
: Zone in which to create the VM.MACHINE_TYPE
: Machine type for the new VM. We strongly recommend choosing a C2 machine type, such asc2-standard-60
. To learn why, read Use compute-optimized instances.
After some time, the VM instance creation completes. To verify the instance and see its status, run the following command:
gcloud compute instances describe INSTANCE_NAME
Access the VM
Console
After you create your HPC VM instance, it starts automatically. To access it, do the following:
In the Google Cloud Console, go to the VM instances page.
Click the name of your VM instance.
In the Remote Access section, click the first drop-down list and choose how you want to access the instance.
Compute Engine propagates your SSH keys and creates your user. For more information, see Connecting to Linux VMs.
gcloud
After you create your HPC VM instance, it starts automatically. To access
it using SSH, use the compute ssh
command:
gcloud compute ssh INSTANCE_NAME
Compute Engine propagates your SSH keys and creates your user. For more information, see Connecting to instances.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this quickstart, delete the HPC VM instance that you created.
Console
In the Google Cloud Console, go to the Deployments page.
Select the checkbox next to the HPC VM deployment.
Click Delete.
gcloud
Use the instances delete
command:
gcloud compute instances delete INSTANCE_NAME
Using Intel MPI collective tunings for your applications
MPI implementations such as Intel MPI and OpenMPI have many internal configuration parameters that can affect communication performance. These parameters are especially relevant for MPI collective communication, which lets you specify algorithms and configuration parameters that can perform very differently in the Google Cloud environment.
We strongly recommend that you tune configuration parameters based on the characteristics of your applications. We also strongly recommend that you enable VM placement policies when generating and using the tuning configuration files.
Tuning configurations included with the HPC VM image
The HPC VM image includes output configurations from Intel MPI collective tunings performed on c2-standard-60 instances with compact placement policies. These tuning files are available in the following directory:
/usr/share/google-hpc-compute/mpitune-configs/intelmpi-2018
The HPC image contains several tuning configurations to support the following scenarios:
- Number of VMs: 1 to 22
- Number of MPI ranks (processes) per VM: 1, 2, 6, 10, and 30
To use these tuning configurations, do the following:
Install the Intel MPI library 2018.
Execute the following bash script to set up the environment:
source MPI_INSTALL_DIR/mpivars.sh
Replace
MPI_INSTALL_DIR
with the path to the directory where you installed the Intel MPI library.Install the tunings included in the HPC VM image with the following command. Use the
--sudo
option when you need root access to the directory:google_install_mpitune
Generating custom tuning configurations using mpitune
You can use mpitune
to manually specify the algorithms and configuration
parameters for MPI collective communication and to generate configuration files.
For example, to tune for 22 VMs and 30 processes per VM, source the
mpivars.sh
script to set up the proper environment, then run the following
command. You must have write access to the directory or run the command as root.
mpitune -hf hostfile -fl 'shm:tcp' -pr 30:30 -hr 22:22
This generates a configuration file in the Intel MPI directory, which you can
use to run applications. To make use of the tuning configuration for an
application, add the -tune
option to the following command:
mpirun -tune -hostfile hostfile -genv I_MPI_FABRICS 'shm:tcp' -np 660 -ppn 30 ./app
Turning off Meltdown and Spectre mitigations
The HPC VM image enables the Meltdown and Spectre mitigations by default. In some cases, these mitigations might result in workload-specific performance degradation. To disable these mitigations and incur the associated security risks, do the following:
- Create an HPC VM instance.
On that VM instance, run the following command:
sudo google_mpi_tuning --nomitigation
Reboot the VM.
Pricing
The HPC VM image is available at no additional cost. Because the HPC VM image runs on Compute Engine, you might incur charges for Compute Engine resources such as C2 vCPUs and memory. To learn more, see Compute Engine pricing.
Limitations
The benefits of tuning vary from application to application. In some cases, a particular tuning might have a negative effect on performance. Consider benchmarking your applications to find the most efficient or cost-effective configuration.
What's next
- Learn more about high performance computing on Google Cloud.
- If you have feedback or require support, email hpc-image-feedback@google.com.