Cluster deployment overview

This document provides an overview of how to use Cloud HPC Toolkit to deploy an HPC cluster on Google Cloud.

Pro Tip: For a full step-by-step example walkthrough on the setup, deployment, and submission of jobs on an HPC cluster using Cloud HPC Toolkit, see Deploy a Slurm cluster.

Before you begin

1. If you are using a Linux or macOS workstation to deploy your cluster, install dependencies.
2. From either your workstation or Cloud Shell, configure your environment.
3. Ensure that you have created an HPC blueprint or selected one from the Cloud HPC Toolkit examples. See Prepare an HPC blueprint.

Overview

To deploy a cluster, you need to complete the following steps:

1. Create the HPC deployment folder from the HPC blueprint file using the ghpc create command.
2. Deploy the cluster from the HPC deployment folder using the ghpc deploy command.

Create the HPC deployment folder

To create an HPC deployment folder, use the ghpc create command command. Replace PATH_TO_BLUEPRINT with the location of your HPC blueprint file.

./ghpc create PATH_TO_BLUEPRINT

Note: Some blueprints might need additional APIs to be enabled, the ghpc create command will let you know if this is required.

Set deployment variables at the command line

The example HPC blueprints included with the Cloud HPC Toolkit don't set the project ID. You must supply a valid project ID by using one of the following options:

• Using the --vars flag.

  ./ghpc create --vars project_id=PROJECT_ID examples/hpc-slurm.yaml
  

• Using a deployment file with the --deployment-file (or -d) flag:

  ./ghpc create -d DEPLOYMENT_FILE examples/hpc-slurm.yaml
  

  If the --deployment-file and --vars flags are used at the same time, then the variables specified with --vars take precedence. If the --vars flag is specified multiple times, then later invocations take precedence.

The preceding example commands assume that you are in the main working directory and you want to use the example hpc-slurm.yaml blueprint that is located in the hpc-toolkit/examples/ directory.

Get help at the command line

For a full list of flags that you can use, run the --help flag on ghpc or on any of the sub-commands.

./ghpc --help
./ghpc create --help

Deploy the cluster

To deploy the cluster, run the ghpc deploy command as shown in the output of the ./ghpc create command. For example:

1. Run the ghpc deploy command to begin automatic deployment of your cluster:

   ./ghpc deploy hpc-slurm

2. ghpc reports proposed changes for your cluster. Optionally, you may review the proposed changes by typing d and pressing enter. To deploy the cluster, accept the proposed changes by typing a and pressing enter.

   Summary of proposed changes: Plan: 37 to add, 0 to change, 0 to destroy.
   (D)isplay full proposed changes,
   (A)pply proposed changes,
   (S)top and exit,
   (C)ontinue without applying
   Please select an option [d,a,s,c]:
   

3. After accepting the changes, ghpc runs terraform apply automatically. This takes approximately 5 minutes while it displays progress. If the run is successful, the output is similar to the following:

   Apply complete! Resources: 37 added, 0 changed, 0 destroyed.
   

You are now ready to submit jobs to your HPC cluster.