Don’t run all code, run only what’s changed: Optimizing IaC deployment with Cloud Build

We often use infrastructure-as-code (IaC) to deploy cloud resources at scale and store this code in source control repositories. Multi-folder repositories can be used to combine similar IaC into a single repository with following benefits:

• Reduced overhead of managing multiple CI/CD pipelines

• Better code visibility

• Reduced overhead of managing multiple ACLs for similar code

We also often use CI/CD pipelines to deploy the IaC within these repositories. In this post, we will cover a method of optimizing IaC pipelines by deploying only what has changed from the last run of the pipeline, resulting in improved performance and reduced cost.

An example of multi-folder IaC repository:

Business Impact

The approach described in this post  is expected to result in the following benefits:

• Faster builds: By only running what has changed.

• Increased developer productivity: You can achieve faster feedback cycles from your IaC pipelines which can improve developer agility.

• Cost optimization: You will be able to reduce the cost of your IaC pipelines by reducing the build times.

Getting started

• You will need a cloud source repository (or any other source control repositories) containing multiple folders of IaC like Terraform.

• You will also need a Cloud Build pipeline with the push to branch event based trigger.

General approach used today

In a multi-folder IaC repository, you will need to iterate over all the folders to deploy the IaC. For the repository example shown above, one of the steps in the Cloud Build pipeline would look like the following:

In this approach, you will need to run code in all the folders of the repository, even if the latest commit change affected only a single folder. This approach has the following disadvantages:

• Slower feedback of code deployment status impacting developer agility

• Longer build times, resulting in higher operational costs of running the IaC pipelines

Selective deployment

In this approach, you will only run IaC which was changed after the last successful deployment of an IaC pipeline.

Solution design 

The following steps are the high level solution design of selective deployment:

• Last successful build: you will need to find the last successful Cloud Build run.

• Compute delta: you will need to find what folders are affected after the last successful deployment of your pipeline.

• Execute: finally, you can deploy IaC code in folders from the compute delta step.

Implementation steps

Step 1: Find the commit associated with your last successful build:

• In this step, you will find the last successful build using the gcloud command `gcloud builds list`. Notice the filters in the example code below are only fetching successful commits for a single Cloud Build trigger.
  If you use an event based Cloud Build trigger, where the event is pushing off a code into the repository, you will have a commit associated with this build. Thus, you can use the `gcloud builds describe` command to get the commit associated with a given Cloud Build run.

Step 2: Find the folders changed after the last successful commit

• You can use the `git diff` command to find the difference between the commit associated with the last successful build (from step 1) and the commit associated with the current build run.

• The diff output can be stored in a log file to be used in the next step. For audit purposes, you can also store this log file in a cloud storage bucket after the build completion.

Step 3: Iterate over changed folders

• You can now iterate over folders from git diff output from step 2 and run the code.

Important points/Edge cases

Including the repository history in a build

To build your source on a Git repo, Cloud Build performs a shallow clone of the repo. This means that only the single commit that started the build is checked out in the workspace to build. This will prevent you from performing the `git diff` operation needed to find the folders changed. You will need to include the repository build history by following the steps defined here.

Last successful build does not exist

You need to have at least one successful build in your build history. You can execute the pipeline without selective deployment to get the first successful build.

Manual commit as input

You might need to manually pass a specific commit to calculate the `git diff`. This feature can be useful for running the last couple of builds again to recover from an error.

Running all folders or a subset of folders when the centralized module is changed

There might be a centralized folder like a Terraform module in your repository. If a change is made at the centralized folder level, you will need to run all folders.

Canonical example