Deploying and managing an Elastifile Cloud File System cluster with Terraform

By: Yannick Guillerm, Director Solution Partners, Elastifile

This tutorial demonstrates how to deploy and manage an Elastifile Cloud File System (ECFS) cluster on Google Cloud with Terraform. By using Terraform, you can deploy an initial Elastifile cluster, add storage capacity to the cluster, remove storage nodes, and finally, delete the cluster.

Diagram of components used during tutorial

This diagram shows the different components you use during the tutorial and also shows the different Terraform actions you apply on the ECFS cluster:

  • Deploy an ECFS cluster.
  • Add and remove storage nodes.
  • Delete the ECFS cluster.

The following Google Cloud components are used throughout the tutorial:

  • Compute Engine instances.
  • Terraform command-line interface (CLI).

Objectives

  • Deploy an initial ECFS cluster with 3 storage nodes.
  • Add 2 storage nodes to add capacity to the ECFS cluster.
  • Connect to the ECFS Dashboard and verify the deployment.
  • Remove 2 storage nodes from the ECFS cluster.
  • Delete the ECFS cluster.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  3. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  4. In the Google Cloud Console, add firewall rules to authorize TCP ports 80, 22, and 443. For more information, read Using firewall rules.
    Go to the Firewall Rules page
  5. In the Cloud Console, add the following roles to the service account you are using for this tutorial. For more information, read Granting, changing, and revoking access to resources.
    Go to the IAM page
    • Compute Instance Admin (v1)
    • Compute Network Admin
    • Compute Security Admin
    • Service Account User
    • Storage Admin
    • Storage Object Admin
  6. In the Cloud Console, open Cloud Shell.

    OPEN Cloud Shell

    You run all of the commands in Cloud Shell and the Code Editor.

Installing and configuring Terraform

Terraform is a stand-alone binary that you install and use on Linux to manage infrastructure by using configuration files and Terraform commands. For more information, read about installing Terraform.

  1. In Cloud Shell, download the Terraform package for Linux:

    wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip
    
  2. Unzip the Terraform package and move the binary to the /usr/local/bin folder:

    unzip terraform_0.11.11_linux_amd64.zip
    sudo mv terraform /usr/local/bin
    
  3. Verify the installation:

    terraform --version
    

    The output is similar to the following:

    Terraform v0.11.11
    + provider.google v2.1.0
    + provider.null v2.1.0
    

Deploying an initial ECFS cluster

In this step, you configure the Terraform tfvars file to deploy an initial ECFS cluster that consists of:

  • An ECFS management instance (EMS)
  • 3 ECFS storage nodes

Install and configure Terraform

  1. In Cloud Shell, clone the Elastifile GitHub Terraform package. For more information about the content of the Elastifile GitHub Terraform package, read the GitHub page about automation.

    git clone https://github.com/Elastifile/gcp-automation.git
    
  2. Display the files in the gcp-automation package:

    ls -l
    

    The output lists a number of files. During this tutorial, you only use the terraform.tfvars file, which is the main file to dictate actions on the ECFS cluster. All the other files are part of the Terraform automation package.

  3. Get the credential JSON file associated with your service account. See Creating and managing service account keys.

  4. To upload a file, in Cloud Shell, click More more_vert and then click Upload File. Navigate to the JSON file you just downloaded. Select the file and click Open to upload the file.

  5. When the upload is complete, move the file to the gcp-automation/ folder:

    mv elastifile-show-041ae354d086.json gcp-automation/
    
  6. In Cloud Shell, click Launch Code Editor create. When the Code Editor opens, it displays the contents of the Terraform package.

    Terraform package in Code Editor

  7. Go to /home/[USER_NAME]/gcp-automation/terraform.tfvars. The contents of the file are displayed in the right panel allowing you to edit it. For a description of the Elastifile Terraform variables, read the GitHub page.

    Tfvars file in Code Editor

  8. Edit the value of the following variables with values corresponding to your environment. The variables need to reflect your environment.

    # Company name - No spaces allowed
    COMPANY_NAME = "[COMPANY_NAME]"
    # Contact person name - No spaces allowed
    CONTACT_PERSON_NAME = "[PERSON_NAME]"
    # Contact person email address
    EMAIL_ADDRESS = "[EMAIL_ADDRESS]"
    # "small" "medium" "large" "standard" "small standard" "local" "small local" "custom"
    TEMPLATE_TYPE = "[CLUSTER_SIZE_TEMPLATE]"
    # instance prefix
    CLUSTER_NAME = "[CLUSTER_NAME]"
    # GCP region
    REGION = "[REGION]"
    # GCP zone
    EMS_ZONE = "[ZONE]"
    # GCP project
    PROJECT = "[PROJECT_NAME]"
    # GCP project subnetwork
    SUBNETWORK = "[SUBNETWORK_NAME]"
    # GCP project network
    NETWORK = "[NETWORK_NAME]"
    # GCP service account credential filename
    CREDENTIALS = "[JSON_CREDENTIAL_FILE_NAME]"
    SERVICE_EMAIL = "[SERVICE_ACCOUNT_NAME]"
    

    Where:

    • [COMPANY_NAME] represents the name of your company, for example, ExampleOrganization.
    • [PERSON_NAME] represents your name, for example, Elizabeth Bennet.
    • [EMAIL_ADDRESS] represents your email address, for example, liz@gmail.com.
    • [CLUSTER_SIZE_TEMPLATE] represents the size of the initial Elastifile cluster to be deployed. For the purpose of this tutorial use small.
    • [CLUSTER_NAME] represents the name of the Elastifile cluster, for example, elastifile-liz.
    • [REGION] represents theGoogle Cloud region in which you want to deploy the cluster, for example, us-central1.
    • [ZONE] represents the Google Cloud zone in which you want to deploy the cluster, for example, us-central1-f.
    • [PROJECT_NAME] represents the unique Google Cloud project you want to use to deploy the cluster, for example, elastifile-show.
    • [SUBNETWORK_NAME] represents the name of the Google Cloud subnet in the project you want to use to deploy the cluster, for example, default.
    • [NETWORK_NAME] represents the name of the Google Cloud network in the project you want to use to deploy the cluster, for example, default.
    • [JSON_CREDENTIAL_FILE_NAME] represents the name of the JSON credential file you downloaded. The JSON file is in the gcp-automation folder, for example, elastifile-show-6daae7905a83.json.
    • [SERVICE_ACCOUNT_NAME] represents the name of the service account associated with the JSON credential file, for example, liz@elastifile-show.iam.gserviceaccount.com.
  9. To save your changes, click File and then click Save.

Deploy the ECFS cluster with Terraform

  1. Initialize the Terraform package:

    cd gcp-automation
    terraform init
    

    The output is similar to the following:

    Initializing provider plugins...
    The following providers do not have any version constraints in configuration,
    so the latest version was installed.
    To prevent automatic upgrades to new major versions that may contain breaking
    changes, it is recommended to add version = "..." constraints to the
    corresponding provider blocks in configuration, with the constraint strings
    suggested below.
    * provider.google: version = "~> 2.1"
    * provider.null: version = "~> 2.1"
    Terraform has been successfully initialized!
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.
    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    
  2. Apply the configuration to deploy the ECFS cluster:

    terraform apply
    
  3. When prompted, enter Y and then press Enter. It might take a few minutes to deploy the ECFS cluster.

    Output while deploying the cluster

    If you encounter any errors, correct the issue, and repeat the terraform apply command to deploy the cluster again. Wait until the output displays Apply complete, indicating that the ECFS cluster was successfully deployed. The deployment can take up to 10 minutes to complete.

    null_resource.cluster (local-exec): + curl -k -D session.txt -H 'Content-Type: application/json' -X POST -d '{"user":
    {"login":"admin","password":"changeme"}}' https://35.222.67.172/api/sessions
    null_resource.cluster (local-exec): + enable_async_dr
    null_resource.cluster (local-exec): + [[ false == \t\r\u\e ]]
    null_resource.cluster: Creation complete after 7m3s (ID: 8890963652689317749)
    
    Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
    

    The ECFS cluster is fully deployed and operational.

  4. In the Cloud Console, go to the VM Instances page.

    GO TO VM INSTANCES PAGE

    There are 4 Compute Engine instances listed, corresponding to 1 ECFS EMS instance and 3 ECFS storage node instances.

    VM Instances page with 4 instances deployed

  5. To connect to the Elastifile Dashboard, click the External IP Address of the Elastifile management instance. Enter the username and password that you configured in the terraform.tfvars configuration file.

    The ECFS Dashboard looks similar to this:

    ECFS Dashboard

To learn more about the ECFS Dashboard and how to mount the data container (NFS share) on client VM instances, use the Deploying the Elastifile Cross-Cloud Data Fabric tutorial.

Adding capacity to the ECFS cluster

In this step, you edit the terraform.tfvars file to add 2 additional storage nodes to the initial ECFS cluster you deployed.

  1. In the Code Editor, go to /home/[USER_NAME]/gcp-automation/terraform.tfvars. Edit the following two variables with values:

    # number of vheads exclusive of EMS
    NUM_OF_VMS = "5"
    # setup complete - false for initial deployment, true for add/remove nodes
    SETUP_COMPLETE = "true"
    
  2. To save your changes, click File and then click Save.

  3. To modify the ECFS cluster and add the storage nodes, you need to apply the new Terraform configuration:

    terraform apply
    

    A message displays to tell you that Terraform is changing the ECFS cluster configuration.

    Terraform will perform the following actions:
    ~ google_compute_instance.Elastifile-EMS-Public
    metadata.cluster_size:   "3" => "5"
    metadata.setup_complete: "false" => "true"
    tags.#:                  "2" => "1"
    tags.1387189632:       "elastifile-management-node-elastifile-yannickg" => ""
    tags.1936433573:         "https-server" => "https-server"
    + null_resource.update_cluster
    id:                      <computed>
    triggers.%:              "1"
    triggers.num_of_vms:     "5"
    
  4. When prompted, type Yes and press Enter to accept the changes. When the configuration change is successfully applied, the output displays:

    null_resource.update_cluster: Creation complete after 2m20s (ID: 748067365815546592)
    Apply complete! Resources: 1 added, 1 changed, 0 destroyed.yannick_guillerm@cloudshell:~/gcp-automation (elastifile-show)$
    
  5. In the Cloud Console, go to the VM Instances page.

    GO TO VM INSTANCES PAGE

    This page now displays the 2 ECFS storage nodes you added.

    VM instances page with 7 instances

Removing capacity from the ECFS cluster

You can remove storage nodes from the ECFS cluster by modifying the value of the NUM_OF_VMSvariableand running the Terraform command again.

  1. In Cloud Shell, repeat the steps from the previous section. Change the value of the variable to 3 and rerun the terraform apply command. This step lets you shrink the ECFS cluster down to a minimum number of nodes, for example, after a major scale-out for a compute burst.

    # number of vheads exlusive of EMS
    NUM_OF_VMS = "3"
    # setup complete - false for initial deployment, true for add/remove nodes
    SETUP_COMPLETE = "true"
    
  2. To save your changes, click File and then click Save.

  3. To modify the ECFS cluster and remove the storage nodes, apply the new Terraform configuration:

    terraform apply
    
  4. When prompted, type Yes and press Enter to accept the changes. When the configuration change is successfully applied, the output displays:

    null_resource.update_cluster: Creation complete after 2m20s (ID: 748067365815546592)
    Apply complete! Resources: 1 added, 1 changed, 0 destroyed.yannick_guillerm@cloudshell:~/gcp-automation (elastifile-show)$
    
  5. In the Cloud Console, go to the VM Instances page.

    GO TO VM INSTANCES PAGE

    The two instances you added in the previous section are now deleted.

    VM Instances page with 4 instances again

Deleting the ECFS cluster

In this step, you use the Terraform destroy command to delete the ECFS cluster that you deployed and modified in the previous step.

  1. In Cloud Shell, enter the following command:

    terraform destroy
    

    A message displays to tell you that Terraform is changing the ECFS cluster configuration:

    Terraform will perform the following actions:
      - google_compute_instance.Elastifile-EMS-Public
      - null_resource.cluster
      - null_resource.update_cluster
    
  2. When prompted, type Yes and press Enter to accept the changes. It might take a few minutes to delete the cluster. When the configuration change is successfully applied, the output displays:

    Destroy complete! Resources: 3 destroyed.
    
  3. In the Cloud Console, go to the VM Instances page.

    GO TO VM INSTANCES PAGE

    All of the ECFS instances are deleted

    VM Instances page with 0 instances

Cleaning up

In this tutorial, you deploy, expand, and delete an Elastifile Cloud File System cluster by using a Terraform script. The last step of this tutorial deleted all Google Cloud resources that were used in this tutorial except for the Google Cloud project.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project you want to delete and click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

To try Elastifile as a fully-managed storage service on Google Cloud, visit Elastifile Cloud File Service on Google Cloud Marketplace.

In addition, you can expand your knowledge about Elastifile and how it is used in Google Cloud using the following resources:

  • Getting Started With Elastifile. Use this tutorial to learn how to deploy Elastifile from Google Cloud Marketplace, mount the ECFS NFS share on a client instance, and run a performance test.
  • Rendering in Google Cloud with Elastifile. Use this tutorial to learn about a use case of Elastifile in Google Cloud: VFX and animation rendering. In this tutorial, you deploy a render farm in Google Cloud by using Blender and Elastifile. You render 10 frames of a 3D sequence and watch the animated result.

  • Learn more about the different Elastifile Cloud Management software products.

  • Try out other Google Cloud features for yourself. Have a look at our tutorials.