Deploying and managing an Elastifile Cloud File System cluster with Terraform

---

This tutorial demonstrates how to deploy and manage an Elastifile Cloud File System (ECFS) cluster on Google Cloud with Terraform. By using Terraform, you can deploy an initial Elastifile cluster, add storage capacity to the cluster, remove storage nodes, and finally, delete the cluster.

This diagram shows the different components you use during the tutorial and also shows the different Terraform actions you apply on the ECFS cluster:

• Deploy an ECFS cluster.
• Add and remove storage nodes.
• Delete the ECFS cluster.

The following Google Cloud components are used throughout the tutorial:

• Compute Engine instances.
• Terraform command-line interface (CLI).

Objectives

• Deploy an initial ECFS cluster with 3 storage nodes.
• Add 2 storage nodes to add capacity to the ECFS cluster.
• Connect to the ECFS Dashboard and verify the deployment.
• Remove 2 storage nodes from the ECFS cluster.
• Delete the ECFS cluster.

Costs

This tutorial uses the following billable components of Google Cloud:

• Compute Engine
• Cloud Storage

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

6. In the Google Cloud console, add firewall rules to authorize TCP ports 80, 22, and 443. For more information, read Using firewall rules.
   Go to the Firewall Rules page
7. In the Google Cloud console, add the following roles to the service account you are using for this tutorial. For more information, read Granting, changing, and revoking access to resources.
   Go to the IAM page
   
   • Compute Instance Admin (v1)
   • Compute Network Admin
   • Compute Security Admin
   • Service Account User
   • Storage Admin
   • Storage Object Admin
8. In the Google Cloud console, open Cloud Shell.
   OPEN Cloud Shell
   

   You run all of the commands in Cloud Shell and the Code Editor.

Installing and configuring Terraform

Terraform is a stand-alone binary that you install and use on Linux to manage infrastructure by using configuration files and Terraform commands. For more information, read about installing Terraform.

1. In Cloud Shell, download the Terraform package for Linux:

   wget https://releases.hashicorp.com/terraform/0.11.15/terraform_0.11.15_linux_amd64.zip
   

2. Unzip the Terraform package and move the binary to the /usr/local/bin folder:

   unzip terraform_0.11.15_linux_amd64.zip
   sudo mv terraform /usr/local/bin
   

3. Verify the installation:

   terraform --version
   

   The output is similar to the following:

   Terraform v0.11.15
   + provider.google v2.5.1
   + provider.null v2.1.2
   

Deploying an initial ECFS cluster

In this step, you configure the Terraform tfvars file to deploy an initial ECFS cluster that consists of:

• An ECFS management instance (EMS)
• 3 ECFS storage nodes

Install and configure Terraform

1. In Cloud Shell, clone the Elastifile GitHub Terraform package. For more information about the content of the Elastifile GitHub Terraform package, read the GitHub page about automation.

   git clone https://github.com/Elastifile/gcp-automation.git
   

2. Display the files in the gcp-automation package:

   ls -l
   

   The output lists a number of files. During this tutorial, you only use the terraform.tfvars file, which is the main file to dictate actions on the ECFS cluster. All the other files are part of the Terraform automation package.

3. Get the credential JSON file associated with your service account. See Creating and managing service account keys.

4. To upload a file, in Cloud Shell, click More and then click Upload File. Navigate to the JSON file you just downloaded. Select the file and click Open to upload the file.

5. When the upload is complete, move the file to the gcp-automation/ folder:

   mv elastifile-show-041ae354d086.json gcp-automation/
   

6. In Cloud Shell, click Launch Code Editor . When the Code Editor opens, it displays the contents of the Terraform package.

   

7. Go to /home/[USER_NAME]/gcp-automation/terraform.tfvars. The contents of the file are displayed in the right panel allowing you to edit it. For a description of the Elastifile Terraform variables, read the GitHub page.

   

8. Edit the value of the following variables with values corresponding to your environment. The variables need to reflect your environment.

   # Company name - No spaces allowed
   COMPANY_NAME = "[COMPANY_NAME]"
   # Contact person name - No spaces allowed
   CONTACT_PERSON_NAME = "[PERSON_NAME]"
   # Contact person email address
   EMAIL_ADDRESS = "[EMAIL_ADDRESS]"
   # "small" "medium" "large" "standard" "small standard" "local" "small local" "custom"
   TEMPLATE_TYPE = "[CLUSTER_SIZE_TEMPLATE]"
   # instance prefix
   CLUSTER_NAME = "[CLUSTER_NAME]"
   # GCP region
   REGION = "[REGION]"
   # GCP zone
   EMS_ZONE = "[ZONE]"
   # GCP project
   PROJECT = "[PROJECT_NAME]"
   # GCP project subnetwork
   SUBNETWORK = "[SUBNETWORK_NAME]"
   # GCP project network
   NETWORK = "[NETWORK_NAME]"
   # GCP service account credential filename
   CREDENTIALS = "[JSON_CREDENTIAL_FILE_NAME]"
   SERVICE_EMAIL = "[SERVICE_ACCOUNT_NAME]"
   

   Where:

   • [COMPANY_NAME] represents the name of your company, for example, ExampleOrganization.
   • [PERSON_NAME] represents your name, for example, Elizabeth Bennet.
   • [EMAIL_ADDRESS] represents your email address, for example, [e-mail].
   • [CLUSTER_SIZE_TEMPLATE] represents the size of the initial Elastifile cluster to be deployed. For the purpose of this tutorial use small.
   • [CLUSTER_NAME] represents the name of the Elastifile cluster, for example, elastifile-liz.
   • [REGION] represents theGoogle Cloud region in which you want to deploy the cluster, for example, us-central1.
   • [ZONE] represents the Google Cloud zone in which you want to deploy the cluster, for example, us-central1-f.
   • [PROJECT_NAME] represents the unique Google Cloud project you want to use to deploy the cluster, for example, elastifile-show.
   • [SUBNETWORK_NAME] represents the name of the Google Cloud subnet in the project you want to use to deploy the cluster, for example, default.
   • [NETWORK_NAME] represents the name of the Google Cloud network in the project you want to use to deploy the cluster, for example, default.
   • [JSON_CREDENTIAL_FILE_NAME] represents the name of the JSON credential file you downloaded. The JSON file is in the gcp-automation folder, for example, elastifile-show-6daae7905a83.json.
   • [SERVICE_ACCOUNT_NAME] represents the name of the service account associated with the JSON credential file, for example, [e-mail].

9. To save your changes, click File and then click Save.

Deploy the ECFS cluster with Terraform

1. Initialize the Terraform package:

   cd gcp-automation
   terraform init
   

   The output is similar to the following:

   Initializing provider plugins...
   The following providers do not have any version constraints in configuration,
   so the latest version was installed.
   To prevent automatic upgrades to new major versions that may contain breaking
   changes, it is recommended to add version = "..." constraints to the
   corresponding provider blocks in configuration, with the constraint strings
   suggested below.
   * provider.google: version = "~> 2.1"
   * provider.null: version = "~> 2.1"
   Terraform has been successfully initialized!
   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.
   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.
   

2. Apply the configuration to deploy the ECFS cluster:

   terraform apply
   

3. When prompted, enter Y and then press Enter. It might take a few minutes to deploy the ECFS cluster.

   

   If you encounter any errors, correct the issue, and repeat the terraform apply command to deploy the cluster again. Wait until the output displays Apply complete, indicating that the ECFS cluster was successfully deployed. The deployment can take up to 10 minutes to complete.

   null_resource.cluster (local-exec): + curl -k -D session.txt -H 'Content-Type: application/json' -X POST -d '{"user":
   {"login":"admin","password":"changeme"}}' https://35.222.67.172/api/sessions
   null_resource.cluster (local-exec): + enable_async_dr
   null_resource.cluster (local-exec): + [[ false == \t\r\u\e ]]
   null_resource.cluster: Creation complete after 7m3s (ID: 8890963652689317749)
   
   Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
   

   The ECFS cluster is fully deployed and operational.

4. In the Google Cloud console, go to the VM Instances page.

   GO TO VM INSTANCES PAGE

   There are 4 Compute Engine instances listed, corresponding to 1 ECFS EMS instance and 3 ECFS storage node instances.

   

5. To connect to the Elastifile Dashboard, click the External IP Address of the Elastifile management instance. Enter the username and password that you configured in the terraform.tfvars configuration file.

   The ECFS Dashboard looks similar to this:

   

To learn more about the ECFS Dashboard and how to mount the data container (NFS share) on client VM instances, use the Deploying the Elastifile Cross-Cloud Data Fabric tutorial.

Adding capacity to the ECFS cluster

In this step, you edit the terraform.tfvars file to add 2 additional storage nodes to the initial ECFS cluster you deployed.

1. In the Code Editor, go to /home/[USER_NAME]/gcp-automation/terraform.tfvars. Edit the following two variables with values:

   # number of vheads exclusive of EMS
   NUM_OF_VMS = "5"
   # setup complete - false for initial deployment, true for add/remove nodes
   SETUP_COMPLETE = "true"
   

2. To save your changes, click File and then click Save.

3. To modify the ECFS cluster and add the storage nodes, you need to apply the new Terraform configuration:

   terraform apply
   

   A message displays to tell you that Terraform is changing the ECFS cluster configuration.

   Terraform will perform the following actions:
   ~ google_compute_instance.Elastifile-EMS-Public
   metadata.cluster_size:   "3" => "5"
   metadata.setup_complete: "false" => "true"
   tags.#:                  "2" => "1"
   tags.1387189632:       "elastifile-management-node-elastifile-yannickg" => ""
   tags.1936433573:         "https-server" => "https-server"
   + null_resource.update_cluster
   id:                      <computed>
   triggers.%:              "1"
   triggers.num_of_vms:     "5"
   

4. When prompted, type Yes and press Enter to accept the changes. When the configuration change is successfully applied, the output displays:

   null_resource.update_cluster: Creation complete after 2m20s (ID: 748067365815546592)
   Apply complete! Resources: 1 added, 1 changed, 0 destroyed.yannick_guillerm@cloudshell:~/gcp-automation (elastifile-show)$
   

5. In the Google Cloud console, go to the VM Instances page.

   GO TO VM INSTANCES PAGE

   This page now displays the 2 ECFS storage nodes you added.

   

Removing capacity from the ECFS cluster

You can remove storage nodes from the ECFS cluster by modifying the value of the NUM_OF_VMSvariableand running the Terraform command again.

1. In Cloud Shell, repeat the steps from the previous section. Change the value of the variable to 3 and rerun the terraform apply command. This step lets you shrink the ECFS cluster down to a minimum number of nodes, for example, after a major scale-out for a compute burst.

   # number of vheads exclusive of EMS
   NUM_OF_VMS = "3"
   # setup complete - false for initial deployment, true for add/remove nodes
   SETUP_COMPLETE = "true"
   

2. To save your changes, click File and then click Save.

3. To modify the ECFS cluster and remove the storage nodes, apply the new Terraform configuration:

   terraform apply
   

4. When prompted, type Yes and press Enter to accept the changes. When the configuration change is successfully applied, the output displays:

   null_resource.update_cluster: Creation complete after 2m20s (ID: 748067365815546592)
   Apply complete! Resources: 1 added, 1 changed, 0 destroyed.yannick_guillerm@cloudshell:~/gcp-automation (elastifile-show)$
   

5. In the Google Cloud console, go to the VM Instances page.

   GO TO VM INSTANCES PAGE

   The two instances you added in the previous section are now deleted.

   

Deleting the ECFS cluster

In this step, you use the Terraform destroy command to delete the ECFS cluster that you deployed and modified in the previous step.

1. In Cloud Shell, enter the following command:

   terraform destroy
   

   A message displays to tell you that Terraform is changing the ECFS cluster configuration:

   Terraform will perform the following actions:
     - google_compute_instance.Elastifile-EMS-Public
     - null_resource.cluster
     - null_resource.update_cluster
   

2. When prompted, type Yes and press Enter to accept the changes. It might take a few minutes to delete the cluster. When the configuration change is successfully applied, the output displays:

   Destroy complete! Resources: 3 destroyed.
   

3. In the Google Cloud console, go to the VM Instances page.

   GO TO VM INSTANCES PAGE

   All of the ECFS instances are deleted

   

Clean up

In this tutorial, you deploy, expand, and delete an Elastifile Cloud File System cluster by using a Terraform script. The last step of this tutorial deleted all Google Cloud resources that were used in this tutorial except for the Google Cloud project.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

To try Elastifile as a fully-managed storage service on Google Cloud, visit Elastifile Cloud File Service on Google Cloud Marketplace.

In addition, you can expand your knowledge about Elastifile and how it is used in Google Cloud using the following resources:

• Getting Started With Elastifile. Use this tutorial to learn how to deploy Elastifile from Google Cloud Marketplace, mount the ECFS NFS share on a client instance, and run a performance test.

• Rendering in Google Cloud with Elastifile. Use this tutorial to learn about a use case of Elastifile in Google Cloud: VFX and animation rendering. In this tutorial, you deploy a render farm in Google Cloud by using Blender and Elastifile. You render 10 frames of a 3D sequence and watch the animated result.

• Learn more about the different Elastifile Cloud Management software products.

• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.