Managing infrastructure as code with Terraform, Jenkins, and GitOps

This tutorial explains how to manage infrastructure as code with Terraform and Jenkins using the popular GitOps methodology. The tutorial is for developers and operators who are looking for best practices to manage infrastructure the way they manage software applications. The article assumes you are familiar with Terraform, Jenkins, GitHub, Google Kubernetes Engine (GKE), and Google Cloud.

Architecture

The architecture used in this tutorial uses GitHub branches—dev and prod—to represent actual development and production environments. These environments are defined by Virtual Private Cloud (VPC) networks—dev and prod—in a Google Cloud project.

Note: For simplicity, this tutorial implements only dev and prod environments using VPCs. You can extend this behavior to deploy to more environments and to create projects under your organization hierarchy as needed.

Infrastructure proposal

As the following architecture diagram shows, the process starts when a developer or operator makes an infrastructure proposal to a GitHub non-protected branch, usually a feature branch. When appropriate, this feature branch can be promoted to the development environment through a pull request (PR) to the dev branch. Jenkins then automatically triggers a job to execute the validation pipeline. This job runs the terraform plan command and reports the validation result back to GitHub and includes a link to a detailed infrastructure change report. This step is critical because it lets you review potential changes with collaborators and add follow-up commits before changes are merged into the dev branch.

Dev deployment

If the validation process succeeds and you approve the proposed infrastructure changes, you can merge the pull request into the dev branch. The following diagram shows this process.

When the merge is complete, Jenkins triggers another job to execute the deployment pipeline. In this scenario, the job applies Terraform manifests in the development environment to achieve the state you want. This step is important because it lets you test the Terraform code before promoting it to production.

Prod deployment

After you have tested and are ready to promote the changes to production, you must merge the dev branch into the prod branch to trigger the infrastructure installation to the production environment. The following diagram shows this process.

The same validation process executes when you create the pull request. This process lets your operations team review and approve the proposed changes to production.

Objectives

• Set up your GitHub repository.
• Create Terraform remote state storage.
• Create a GKE cluster and install Jenkins.
• Change your environment configuration in a feature branch.
• Promote changes to the development environment.
• Promote changes to the production environment.

Costs

This tutorial uses the following billable components of Google Cloud:

• GKE
• Cloud Storage
• Compute Engine
• Persistent Disk

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

4. In the Cloud Console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

5. In Cloud Shell, configure your project ID and set your GitHub user name and email address:
   

   PROJECT_ID=PROJECT_ID
   GITHUB_USER=YOUR_GITHUB_USER
   GITHUB_EMAIL=YOUR_EMAIL_ADDRESS
   gcloud config set project $PROJECT_ID
   

   If you haven't accessed GitHub from Cloud Shell before, you can configure it with your username and email address:

   git config --global user.email "$GITHUB_EMAIL"
   git config --global user.name "$GITHUB_USER"
   

   GitHub uses this information to identify you as the author of the commits that you create in Cloud Shell.

Setting up your GitHub repository

In this tutorial, you use a single GitHub repository to define your cloud infrastructure. You orchestrate this infrastructure by having different branches that correspond to different environments:

• The dev branch contains the latest changes that are applied to the development environment.
• The prod branch contains the latest changes that are applied to the production environment.

With this infrastructure, you can always reference the repository to know what configuration is expected in each environment and to propose new changes by first merging them into the dev environment. You then promote the changes by merging the dev branch into the prod branch.

To get started, you must fork the solutions-terraform-jenkins-gitops repository.

1. In GitHub, go to the solutions-terraform-jenkins-gitops repository.

2. Click Fork.

   

   Now you have a copy of the solutions-terraform-jenkins-gitops repository with source files.

3. In Cloud Shell, clone this forked repository:

   cd ~
   git clone https://github.com/$GITHUB_USER/solutions-terraform-jenkins-gitops.git
   cd ~/solutions-terraform-jenkins-gitops
   

   The code in this repository is structured as follows:

   • example-pipelines/ folder: contains subfolders with the example pipeline used in this tutorial.
   • example-create/: contains Terraform code for creating a virtual machine in your environment.
   • environments/: contains dev and prod environment folders with backend configurations and links to files from the example-create/ folder.
   • jenkins-gke/ folder: contains scripts required to deploy Jenkins in a new GKE cluster.
   • tf-gke/: contains the Terraform code for deploying to GKE, and installing Jenkins and its dependent resources.

Creating Terraform remote state storage

Terraform state is stored locally by default. However, we recommend that you store the state in remote central storage that you can access from any system. This approach helps you avoid creating multiple copies on different systems, which can lead to mismatched infrastructure configuration and states.

In this section, you configure a Cloud Storage bucket that stores Terraform's remote state.

1. In Cloud Shell, create a Cloud Storage bucket:

   gsutil mb gs://${PROJECT_ID}-tfstate
   

2. Enable object versioning to keep the history of your states:

   gsutil versioning set on gs://${PROJECT_ID}-tfstate
   

   Note: Enabling object versioning increases storage costs, which you can mitigate by configuring Object Lifecycle Management to delete old state versions.

3. Replace the PROJECT_ID placeholder with your project ID in both terraform.tfvars and backend.tf files:

   sed -i.bak "s/PROJECT_ID/${PROJECT_ID}/g" ./example-pipelines/environments/*/terraform.tfvars
   sed -i.bak "s/PROJECT_ID/${PROJECT_ID}/g" ./example-pipelines/environments/*/backend.tf
   
   sed -i.bak "s/PROJECT_ID/${PROJECT_ID}/g" ./jenkins-gke/tf-gke/terraform.tfvars
   sed -i.bak "s/PROJECT_ID/${PROJECT_ID}/g" ./jenkins-gke/tf-gke/backend.tf
   

4. Check whether all files were updated:

   git status
   

   The output looks like the following:

   On branch dev
   Your branch is up-to-date with 'origin/dev'.
   Changes not staged for commit:
     (use "git add <file>..." to update what will be committed)
     (use "git checkout -- <file>..." to discard changes in working directory)
   
           modified:   example-pipelines/environments/dev/backend.tf
           modified:   example-pipelines/environments/dev/terraform.tfvars
           modified:   example-pipelines/environments/prod/backend.tf
           modified:   example-pipelines/environments/prod/terraform.tfvars
           modified:   jenkins-gke/tf-gke/backend.tf
           modified:   jenkins-gke/tf-gke/terraform.tfvars
   

5. Commit and push your changes:

   git add --all
   git commit -m "Update project IDs and buckets"
   git push origin dev
   

Depending on your GitHub configuration, you might have to authenticate in order to push the preceding changes.

Creating a GKE cluster and installing Jenkins

In this section, you use Terraform and Helm to set up your environment for managing infrastructure as code. First, you use Terraform and Cloud Foundations Toolkit to configure a Virtual Private Cloud, a GKE cluster and a Workload Identity. Then, you use helm to install Jenkins on top of this environment.

Before you start running Terraform commands, you must create a GitHub personal access token. This token is required in order to allow Jenkins to access your forked repository.

Create a GitHub personal access token

1. Sign in to GitHub.
2. Click your profile photo, and then click Settings.
3. Click Developer settings, and then click Personal access tokens.
4. Click Generate new token and give the token a description, in the Note field, and select the repo scope.

5. Click Generate token and copy the newly created token to the clipboard.

   

6. In Cloud Shell, save the token in the GITHUB_TOKEN variable. This variable content is stored later as a Secret in your GKE cluster.

   GITHUB_TOKEN="NEWLY_CREATED_TOKEN"
   

Create a GKE cluster and install Jenkins

Now you create your GKE cluster. This cluster includes a Workload Identity (jenkins-wi-jenkins@PROJECT_ID.iam.gserviceaccount.com), which lets you give Jenkins the permission it needs in the Service Accounts page in the Cloud Console. Because of its improved security properties and manageability, we recommend using Workload Identity to access Google Cloud services from within GKE.

To manage Google Cloud infrastructure as code, Jenkins must authenticate to use Google Cloud APIs. In the following steps, Terraform configures the Kubernetes service account (KSA) used by Jenkins to act as a Google service account (GSA). This configuration allows Jenkins to automatically authenticate as the GSA when accessing Google Cloud APIs.

For simplicity, you grant project editor access in this tutorial. However, because this role has a broad set of permissions, in production environments, you must follow your company's IT security best practices, which usually means providing least-privileged access.

1. In Cloud Shell, install Terraform:

   wget https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
   unzip terraform_0.12.24_linux_amd64.zip
   sudo mv terraform /usr/local/bin/
   rm terraform_0.12.24_linux_amd64.zip
   

2. Create the GKE cluster and install Jenkins:

   cd jenkins-gke/tf-gke/
   terraform init
   terraform plan --var "github_username=$GITHUB_USER" --var "github_token=$GITHUB_TOKEN"
   terraform apply --auto-approve --var "github_username=$GITHUB_USER" --var "github_token=$GITHUB_TOKEN"
   

   This process might take a few minutes to finish. The output is similar to the following:

   Apply complete! Resources: 28 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   ca_certificate = LS0tLS1CRU..
   client_token = <sensitive>
   cluster_name = jenkins
   gcp_service_account_email = jenkins-wi-jenkins@PROJECT_ID.iam.gserviceaccount.com
   jenkins_k8s_config_secrets = jenkins-k8s-config
   jenkins_project_id = PROJECT_ID
   k8s_service_account_name = jenkins-wi-jenkins
   kubernetes_endpoint = <sensitive>
   service_account = tf-gke-jenkins-k253@PROJECT_ID.iam.gserviceaccount.com
   zone = us-east4-a
   

   When Jenkins is deployed to the newly created GKE cluster, the Jenkins Home directory is stored on a persistent volume according to the Helm chart documentation. This deployment also comes with a multi-branch pipeline that's preconfigured with example-pipelines/environments/Jenkinsfile, which is triggered at pull requests and merges to the dev and prod branches.

3. Go back to the main folder:

   cd ../..
   

4. Retrieve the cluster credentials that you just created:

   gcloud container clusters get-credentials jenkins --zone=us-east4-a --project=${PROJECT_ID}
   

5. Retrieve the Jenkins URL and credentials:

   JENKINS_IP=$(kubectl get service jenkins -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   JENKINS_PASSWORD=$(kubectl get secret jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
   printf "Jenkins url: http://$JENKINS_IP\nJenkins user: admin\nJenkins password: $JENKINS_PASSWORD\n"
   

6. Log in to Jenkins using the output information from the previous step.

   Note: Jenkins tries to deploy your infrastructure in the dev and prod environments. The prod branch will fail because you prepared only the dev branch. Later in this tutorial, you promote those changes to prod.

7. Configure the Jenkins location so that GitHub can create links that go directly to your builds. Click Manage Jenkins > Configure System, and in the Jenkins URL field, set your Jenkins URL.

Changing your environment configuration in a new feature branch

Most of your environment is configured now. So it's time to make some code changes.

1. In Cloud Shell, create a new feature branch where you can work without affecting others in your team:

   git checkout -b change-vm-name
   

2. Change the virtual machine name:

   cd example-pipelines/example-create
   sed -i.bak "s/\${var.environment}-001/\${var.environment}-new/g" main.tf
   

   You are changing the main.tf file in the example-create folder. This file is linked by the dev and prod environment folders, which means that your change is propagated to both environments.

3. Push the code change to the GitHub feature branch:

   git commit -am "change vm name"
   git push --set-upstream origin change-vm-name
   

4. In GitHub, go to the main page of your forked repository.

5. Click the Pull requests tab for your repository, and then click New pull request.

6. For the base repository, select your forked repository.

   

7. For base, select dev, and for compare, select change-vm-name.

   

8. Click Create pull request.

9. When your pull request is open, a Jenkins job is automatically initiated (Jenkins can take a minute or so to acknowledge the new pull request). Click Show all checks and wait for the check to become green.

   

10. Click Details to see more information, including the output of the terraform plan command.

    

The Jenkins job ran the pipeline defined at Jenkinsfile. This pipeline has different behaviors depending on the branch being fetched. The build checks whether the TARGET_ENV variable matches any environment folder. If it does match, Jenkins executes terraform plan for that environment. Otherwise, Jenkins executes terraform plan for all environments to make sure that the proposed change holds for them all. If any of these plans fail to execute, the build fails.

The reason terraform plan runs for all example-pipelines/environments subfolders is to ensure that the changes being proposed apply to every environment. This way, before merging the pull request, you can review the plans to make sure that you're not granting access to an unauthorized entity, for example.

Here's the terraform plan code in the JENKINSFILE:

example-pipelines/environments/Jenkinsfile

View on GitHub

stage('TF plan') {
  when { anyOf {branch "prod";branch "dev";changeRequest() } }
  steps {
    container('terraform') {
      sh '''
      if [[ $CHANGE_TARGET ]]; then
        TARGET_ENV=$CHANGE_TARGET
      else
        TARGET_ENV=$BRANCH_NAME
      fi

      if [ -d "example-pipelines/environments/${TARGET_ENV}/" ]; then
        cd example-pipelines/environments/${TARGET_ENV}
        terraform plan
      else
        for dir in example-pipelines/environments/*/
        do 
          cd ${dir}
          env=${dir%*/}
          env=${env#*/}
          echo ""
          echo "*************** TERRAFOM PLAN ******************"
          echo "******* At environment: ${env} ********"
          echo "*************************************************"
          terraform plan || exit 1
          cd ../../../
        done
      fi'''
    }
  }
}

Similarly, the terraform apply command runs for environment branches, but it is ignored in any other case. In this section, you submitted a code change to a new branch, so no infrastructure deployments were applied to your Cloud project.

Here's the terraform apply code in the JENKINSFILE:

example-pipelines/environments/Jenkinsfile

View on GitHub

stage('TF Apply') {
  when { anyOf {branch "prod";branch "dev" } }
  steps {
    container('terraform') {
      sh '''
      TARGET_ENV=$BRANCH_NAME

      if [ -d "example-pipelines/environments/${TARGET_ENV}/" ]; then
        cd example-pipelines/environments/${TARGET_ENV}
        terraform apply -input=false -auto-approve
      else
        echo "*************** SKIPPING APPLY ******************"
        echo "Branch '$TARGET_ENV' does not represent an official environment."
        echo "*************************************************"
      fi'''
    }
  }
}

Enforcing Jenkins execution success before merging branches

You can make sure merges are applied only when Jenkins job executions are successful.

1. In GitHub, go to the main page of your forked repository.
2. For your repository name, click Settings.
3. Click Branches.
4. For Branch protection rules, click Add rule.
5. In Branch name pattern, enter dev.

6. In Protect matching branches, select Require status checks to pass before merging, and then select continuous-integration/jenkins/pr-merge.

   (Optional) Consider enabling Require pull request reviews before merging and Include administrators options for avoiding unreviewed and unauthorized pull requests to be merged to production.

7. Click Create.

8. Repeat steps 4–7, setting Branch name pattern to prod.

This configuration is important to protect the dev and prod branches, which means that you must first push commits to another branch, and only then merge them to the protected branch. In this tutorial, the protection requires the Jenkins job execution to be successful for the merge to be allowed. You can see if your configuration was applied in the newly created pull request. Look for the green check marks.

Promoting changes to the development environment

You have a pull request waiting to be merged. Now you can apply the state that you want to your dev environment.

1. In GitHub, go to the main page of your forked repository.
2. For your repository name, click Pull requests.
3. Click the pull request that you created.

4. Click Merge pull request, and then click Confirm merge.

   

5. In Jenkins, click Open Blue Ocean. Then in the terraform-jenkins-create-demo multi-branch project, in the Branches tab, check the Status icon to see if a new dev job has been triggered. It might take a minute or so to start.

   

6. In the Cloud Console, go to the VM instances page and check whether you have the VM with the new name.

   Go to VM instances

   

Promoting changes to the production environment

Now that you have tested your development environment, you can promote your infrastructure code to production.

1. In GitHub, go to the main page of your forked repository.
2. Click New pull request.

3. For the base repository, select the repository that you forked.

   

4. For base, select prod, and for compare, select dev

   

5. Click Create pull request.

6. Enter a title, such as Promoting vm name change, and then click Create pull request.

7. Wait for the checker to turn green (it might take a minute or two), and then click the Details link next to continuous-integration/jenkins/pr-merge.

   

8. In Jenkins, select TF Plan and review the proposed changes in the logs.

   

9. If the proposed changes look correct, in GitHub click Merge pull request and then click Confirm merge.

10. In the Cloud Console, open the VM instances page and check whether your production VM was deployed.

    VM instances

    

You have configured an infrastructure-as-code pipeline in Jenkins. In the future, you might want to try the following:

• Add deployments for separate use cases.
• Create additional environments that reflect your needs.
• Use a project per environment instead of a VPC per environment.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• To learn more about deploying and authorizing secure access to Google Cloud from in-house Jenkins, read Jenkins on Google Cloud.
• Check out Introducing Workload Identity: Better authentication for your GKE applications.
• Learn how to manage infrastructure as code in Managing infrastructure as code with Terraform and Cloud Build.
• Consider using Cloud Foundation Toolkit templates to quickly build a repeatable enterprise-ready foundation in Google Cloud.
• Check out GitOps-style continuous delivery with Cloud Build.
• Try more advanced Cloud Build features:
  • Configuring the order of build steps
  • Building, testing, and deploying artifacts
  • Creating custom build steps
• Try out other Google Cloud features for yourself. Have a look at our tutorials.