Speeding up your Builds

This page explains some of the ways by which you can speed up your builds using Cloud Build.

Using a cached Docker image

The easiest way to increase the speed of your Docker image build is by specifying a cached image that can be used for subsequent builds. You can specify the cached image by adding the --cache-from argument in your build config file, which will instruct Docker to build using that image as a cache source.

Each Docker image is made up of stacked layers. Using --cache-from rebuilds all the layers from the changed layer until the end of the build; therefore using --cache-from is not beneficial if you change a layer in the earlier stages of your Docker build.

It is recommended that you always use --cache-from for your builds, but keep the following caveats in mind:

  • You need a previously built Docker image to cache from.
  • You can use --cache-from only for Docker builds; you cannot use it for builders that create other kind of artifacts.
  • The cached image must be retrieved from a registry, which may add to the time it takes to build.

The following steps explain how to build using a previously cached image:

YAML

  1. In your build config, add instructions to:

    • Pull the cached image from Container Registry.
    • Add a --cache-from argument to use that image for rebuilds.

      steps:
      - name: 'gcr.io/cloud-builders/docker'
        args: ['pull', 'gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest']
      - name: 'gcr.io/cloud-builders/docker'
        args: [
                  'build',
                  '-t', 'gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest',
                  '--cache-from', 'gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest',
                  '.'
              ]
      images: ['gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest']
      

      where [IMAGE_NAME] is the name of your image.

  2. Build your image using the above build config:

    gcloud builds submit --config cloudbuild.yaml .
    

JSON

  1. In your build config, add instructions to:

    • Pull the cached image from Container Registry.
    • Add a --cache-from argument to use that image for rebuilds.

      {
          "steps": [
          {
              "name": "gcr.io/cloud-builders/docker",
              "args": ["pull", "gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest"]
          },
          {
              "name": "gcr.io/cloud-builders/docker",
              "args": [
                  "build",
                  "-t",
                  "gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest",
                  "-cache-from",
                  "gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest",
                  "."
              ]
          }
          ],
          "images": ["gcr.io/$PROJECT_ID/[IMAGE_NAME]:latest"]
      }
      

      where [IMAGE_NAME] is the name of your image.

  2. Build your image using the above build config:

    gcloud builds submit --config cloudbuild.json .
    

Caching directories with Google Cloud Storage

To increase the speed of a build, reuse the results from a previous build. You can copy the results of a previous build to a Google Cloud Storage bucket, use the results for faster calculation, and then copy the new results back to the bucket. Use this method when your build takes a long time and produces a small number of files that does not take time to copy to and from Google Cloud Storage.

Unlike --cache-from, which is only for Docker builds, Google Cloud Storage caching can be used for any builder supported by Cloud Build.

Use the following steps to cache directories using Google Cloud Storage:

YAML

  1. In your build config file, add instructions to:

    • Copy the results of a previous build from the Google Cloud Storage bucket.
    • Use the results for the current build.
    • Copy the new results back into the bucket.

      steps:
      - name: gcr.io/cloud-builders/gsutil
        args: ['cp', 'gs://mybucket/results.zip', 'previous_results.zip']
      # operations that use previous_results.zip and produce new_results.zip
      - name: gcr.io/cloud-builders/gsutil
        args: ['cp', 'new_results.zip', 'gs://mybucket/results.zip']
      
  2. Build your code using the above build config:

    gcloud builds submit --config cloudbuild.yaml .
    

JSON

  1. In your build config file, add instructions to:

    • Copy the results of a previous build from the Google Cloud Storage bucket.
    • Use the results for the current build.
    • Copy the new results back into the bucket.

      {
          "steps": [
          {
              "name": "gcr.io/cloud-builders/gsutil",
              "args": ["cp", "gs://mybucket/results.zip", "previous_results.zip"]
          },
          {
              // operations that use previous_results.zip and produce new_results.zip
          },
          {
              "name": "gcr.io/cloud-builders/gsutil",
              "args": ["cp", "new_results.zip", "gs://mybucket/results.zip"]
          }
          ]
      }
      
  2. Build your code using the above build config:

    gcloud builds submit --config cloudbuild.json .
    

Using custom virtual machine sizes

In addition to the standard machine type, Cloud Build provides two high-CPU virtual machine types to run your builds. To increase the speed of your build, select a virtual machine with a higher CPU. Requesting a high-CPU machine may increase the startup time of your build as Cloud Build only starts these machines on demand.

For information on selecting custom disk sizes, see diskSize.

The following steps explain how to specify a custom VM size for a build:

gcloud

To specify a custom VM size, use the --machine-type argument:

gcloud builds submit --config=cloudbuild.yaml \
    --machine-type=n1-highcpu-8 .

YAML

  1. Specify the VM size in your build config file:

    steps:
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build', '-t', 'gcr.io/my-project/image1', '.']
      # operations that take a long time to build
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build', '-t', 'gcr.io/my-project/image2', '.']
    options:
      machineType: 'N1_HIGHCPU_8'
    
  2. Build using the above build config:

    gcloud builds submit --config cloudbuild.yaml .
    

JSON

  1. Specify the VM size in your build config file:

    {
        "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": ["build", "-t", "gcr.io/my-project/image1", "."]
        },
        // operations that take a long time to build
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": ["build", "-t", "gcr.io/my-project/image2", "."]
        }
        ],
        "options": {
            "machineType": "N1_HIGHCPU_8"
        }
    }
    
  2. Build using the above build config:

    gcloud builds submit --config cloudbuild.json .
    

Building leaner containers

Often times, building an application requires a lot of files such as build-time dependencies and intermediate files, which are not needed at runtime. But, when you containerize the application, these files get included in the container, causing unnecessary bloat. Over time, this bloat costs you both time and money because of storing and moving bits between your Docker registry and your container runtime.

To make sure your container is as small as possible, separate the building of the application (and the tools needed to build it) from the assembly of the runtime container.

Cloud Build provides a series of Docker containers with common developer tools such as Git, Docker, and the gcloud command-line interface. Use these tools to define a build config file with one step to build the application, and another step to assemble its final runtime environment.

For example, if you're building a Java application, which requires files such as the source code, application libraries, build systems, build system dependencies, and the JDK. You might have a Dockerfile that looks like the following:

FROM java:8

COPY . workdir/

WORKDIR workdir

RUN GRADLE_USER_HOME=cache ./gradlew buildDeb -x test

RUN dpkg -i ./gate-web/build/distributions/*.deb

CMD ["/opt/gate/bin/gate"]

In the above example, Gradle, which is used to build the package, downloads a large number of libraries in order to function. These libraries are essential to the building of the package, but are not needed at runtime. All of the runtime dependencies are bundled up in the package.

Each command in the Dockerfile creates a new layer in the container. If data is generated in that layer and is not deleted in the same command, that space cannot be recovered. In this case Gradle is downloading hundreds of megabytes of libraries to the cache directory in order to perform the build, but the libraries are not deleted.

A more efficient way to perform the build is to use Cloud Build to separate building the application from building its runtime layer.

The following example separates the step for building the Java application from the step for assembling the runtime container:

YAML

  1. Build the application: In cloudbuild.yaml, add a step to build the application.

    The following code adds a step named java:8 for building the Java code.

    steps:
    
    - name: 'java:8'
      env: ['GRADLE_USER_HOME=cache']
      entrypoint: 'bash'
      args: ['-c', './gradlew gate-web:installDist -x test']
    
    
  2. Assemble the runtime container: In cloudbuild.yaml, add a step to assemble the runtime container.

    The following code adds a step named gcr.io/cloud-builders/docker that assembles the runtime container. It defines the runtime container in a separate file named Dockerfile.slim.

    The example uses the Alpine Linux base layer openjdk:8u111-jre-alpine, which is incredibly lean. Also, it includes the JRE, instead of the bulkier JDK that was necessary to build the application.

    cloudbuild.yaml
    
    steps:
    - name: 'java:8'
      env: ['GRADLE_USER_HOME=cache']
      entrypoint: 'bash'
      args: ['-c',
             './gradlew gate-web:installDist -x test']
    
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build',
             '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA', 
             '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:latest',
             '-f', 'Dockerfile.slim',
             '.'
      ]
    
    
    Dockerfile.slim
    
    FROM openjdk:8u111-jre-alpine
    
    COPY ./gate-web/build/install/gate /opt/gate
    
    CMD ["/opt/gate/bin/gate"]
    
  3. Create the Docker images: In cloudbuild.yaml, add a step to create the images.
    steps:
    - name: 'java:8'
      env: ['GRADLE_USER_HOME=cache']
      entrypoint: 'bash'
      args: ['-c', './gradlew gate-web:installDist -x test']
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build',
             '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA',
             '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:latest',
             '-f', 'Dockerfile.slim', '.']
    images:
    - 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA'
    - 'gcr.io/$PROJECT_ID/$REPO_NAME:latest'
    

JSON

  1. Build the application: In cloudbuild.json, add a step to build the application.

    The following code adds a step named java:8 for building the Java code.

    {
        "steps": [
        {
            "name": "java:8",
            "env": [
                "GRADLE_USER_HOME=cache"
            ],
            "entrypoint": "bash",
            "args": [
                "-c",
                "./gradlew gate-web:installDist -x test"
            ]
        },
    }
    
  2. Assemble the runtime container: In cloudbuild.json, add a step to assemble the runtime container.

    The following code adds a step named gcr.io/cloud-builders/docker that assembles the runtime container. It defines the runtime container in a separate file named Dockerfile.slim.

    The example uses the Alpine Linux base layer openjdk:8u111-jre-alpine, which is incredibly lean. Also, it includes the JRE, instead of the bulkier JDK that was necessary to build the application.

    cloudbuild.json:
    
    {
        "steps": [
        {
            "name": "java:8",
            "env": [
                "GRADLE_USER_HOME=cache"
            ],
            "entrypoint": "bash",
            "args": [
                "-c",
                "./gradlew gate-web:installDist -x test"
            ]
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA",
                "-t",
                "gcr.io/$PROJECT_ID/$REPO_NAME:latest",
                "-f",
                "Dockerfile.slim",
                "."
            ]
        }
        ],
    }
    
    Dockerfile.slim:
    
    FROM openjdk:8u111-jre-alpine
    
    COPY ./gate-web/build/install/gate /opt/gate
    
    CMD ["/opt/gate/bin/gate"]
    
  3. Create the Docker images: In cloudbuild.json, add a step to create the images.
    {
        "steps": [
        {
            "name": "java:8",
            "env": [
                "GRADLE_USER_HOME=cache"
            ],
            "entrypoint": "bash",
            "args": [
                "-c",
                "./gradlew gate-web:installDist -x test"
            ]
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA",
                "-t",
                "gcr.io/$PROJECT_ID/$REPO_NAME:latest",
                "-f",
                "Dockerfile.slim",
                "."
            ]
        }
        ],
        "images": [
            "gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA",
            "gcr.io/$PROJECT_ID/$REPO_NAME:latest"
        ]
    }
    
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Build