Deploy a cluster using Cloud HPC Toolkit

This document provides an overview of how to use Cloud HPC Toolkit to deploy an HPC cluster on Google Cloud.

Before you begin

  1. If you are using a Linux or macOS workstation to deploy your cluster, install dependencies.
  2. From either your workstation or Cloud Shell, configure your environment.
  3. Ensure that you have created an HPC blueprint or selected one from the Cloud HPC Toolkit examples. See Prepare an HPC blueprint.

Overview

To deploy a cluster, you need to complete the following steps:

  1. Create the HPC deployment folder using the ghpc command. The ghpc command uses the HPC blueprint file to create a deployment folder.
  2. Deploy the cluster using Terraform.

Create the HPC deployment folder

To create an HPC deployment folder, use the ghpc command. Replace PATH_TO_BLUEPRINT with the location of your blueprint file.

./ghpc create PATH_TO_BLUEPRINT

For example, if you are in the main working directory and you want to use the example hpc-cluster-small.yaml blueprint that is located in the hpc-toolkit/examples/ directory, run the following command:

./ghpc create examples/hpc-cluster-small.yaml

The output is similar to the following:

terraform -chdir=hpc-small/primary init
terraform -chdir=hpc-small/primary validate
terraform -chdir=hpc-small/primary apply

Set deployment variables at deployment time

You can overwrite or specify deployment variables at deployment time by using the --vars flag.

For example, if you didn't set the project_id variable in your blueprint file, you can specify the project ID at deployment time by using the --vars flag. Replace PROJECT_ID with your project ID.

./ghpc create examples/hpc-cluster-small.yaml \
    --vars project_id=PROJECT_ID

For a full list of flags that you can use, see ghpc command.

Deploy the cluster using Terraform

To deploy the cluster, run the Terraform commands that are returned as an output from the ./ghpc create command. For example, using the instructions from the output in the previous section, complete the following steps:

  1. Set up the Terraform deployment by running the terraform init command:

    terraform -chdir=hpc-small/primary init
  2. Validate the Terraform deployment by running the terraform validate command:

    terraform -chdir=hpc-small/primary validate
  3. Generate a plan that describes the Google Cloud resources that you want deploy by running the terraform apply command:

    terraform -chdir=hpc-small/primary apply
  4. Review the plan and then start the deployment by typing yes and pressing enter.

    This deployment takes a few minutes. You will see regular status updates in the terminal.

    If the terraform apply is successful, a message similar to the following is displayed:

    Apply complete! Resources: xx added, 0 changed, 0 destroyed.
    

You are now ready to submit jobs to your HPC cluster.