Edit on GitHub
Report issue
Page history

Use webhooks to trigger a central Cloud Build pipeline from multiple Git repositories

Author(s): @gmogr @MrTrustor ,   Published: 2022-01-21

Grigory Movsesyan | Cloud Engineer | Google

Théo Chamley | Cloud Solutions Architect | Google

Contributed by Google employees.

In this tutorial, you build a centrally managed Cloud Build pipeline step by step. Using webhook triggers, you also trigger builds from a source code repository, which is not natively supported by Cloud Build. One example of why a setup like this is useful is a Terraform automation pipeline. Terraform typically uses powerful service accounts to apply changes to the infrastructure. In this scenario, you might want to control exactly which commands are executed in the pipeline, and therefore to keep the automation pipeline code separate from the Terraform code.

To achieve this result, you need a pipeline that follows these steps:

  1. When a developer pushes a code change to the repository, it triggers the build pipeline with a webhook.
  2. The build pipeline receives the Git repository URL from the webhook event.
  3. The build pipeline clones the Git repository.
  4. The build pipeline applies the code changes.

pipeline_schema

You use GitLab in this tutorial, but you can apply the same principles to other source code repositories.

Costs

This tutorial uses the following billable components of Google Cloud:

  • Cloud Build
  • Cloud Storage
  • Secret Manager

To generate a cost estimate based on your projected usage, use the pricing calculator.

When you finish this tutorial, you can avoid continued billing by deleting the resources that you created. For details, see the "Cleaning up" section at the end of this tutorial.

Prerequisites

This tutorial uses a sample Git repository on GitLab. If you don’t already have an account on GitLab, you can create an account for free on GitLab.

Before you begin

  1. Select or create a Google Cloud project.

    To select or create a project, go to the Manage resources page in the Google Cloud Console.

  2. Enable billing for your project.

  3. Open Cloud Shell.

  4. In Cloud Shell, set the working project with the following command:

    gcloud config set project [PROJECT_ID]
    

    Replace [PROJECT_ID] with your project ID.

  5. Enable the APIs needed for this tutorial:

    gcloud services enable \
      cloudbuild.googleapis.com \
      secretmanager.googleapis.com 
    
  6. Sign in to GitLab.

  7. In GitLab, fork the sample repository:

    1. Click the Fork button.
    2. Select your personal namespace in the Project URL field.
    3. Select Private as the visibility level.
    4. Click Fork project.

Create a webhook Cloud Build trigger

  1. In the Cloud Console, go to the Triggers page.

  2. Click Create Trigger.

  3. Enter webhook-trigger in the Name field.

  4. Choose Webhook event for the Event field.

  5. Choose Create new in the Secret field, and click Create Secret.

  6. Enter webhook-trigger-secret in the Secret name field, and click Create Secret.

  7. Click Create.

Configure the GitLab repository webhook

  1. Go to the Triggers page.

  2. Click the webhook-trigger trigger.

  3. Click Show URL preview and copy the webhook URL.

  4. On GitLab, click Settings in your forked GitLab repository.

  5. Click webhooks.

  6. Insert the URL that you copied in step 3 in the URL field.

  7. Click Add webhook.

You can learn more about creating webhooks in the GitLab documentation.

Test the webhook

  1. In your GitLab repository, click settings, and then click webhooks.

  2. At the bottom of the page, find the Project Hooks block with the new webhook.

  3. Next to your webhook, click Test and choose Push events.

    This triggers the Cloud Build job.

  4. Go to the Cloud Build history page to see the Cloud Build job running.

  5. Find the build with webhook-trigger as Trigger name that is running or ran recently and click it. You should see “hello world” in the build log.

Get the repository URL from the push event

To be able to fetch the source code from the repository, you first need to get the repository URL from the event payload.

  1. Go to the Triggers page for your Google Cloud project.

  2. Click the webhook-trigger trigger.

  3. In the Configuration block, click Open editor.

  4. In the right panel, modify the inline build config like the following:

    steps:
      - name: ubuntu
        args:
          - echo
          - ${_GIT_REPO}
    substitutions:
      _GIT_REPO: $(body)
    
  5. Click Done, and then click Save.

  6. Trigger the test push event from the GitLab repository again.

  7. Go to the Cloud Build history page to see your new job running.

  8. Click the new job.

    Instead of the “hello world” line, you should see a JSON object with the content of the event payload. There are two fields of particular interest: project.git\_http\_url and project.git\_ssh\_url. This tutorial uses a token for the authorization, so it focuses on the project.git\_http\_url field.

  9. In the Cloud Build trigger configuration, modify the inline build config again to display only the necessary field from the event payload:

    steps: 
      - name: ubuntu 
        args: 
          - echo 
          - '${_GIT_REPO}' 
    substitutions: 
      _GIT_REPO: $(body.project.git_http_url)
    
  10. Trigger the test push event from GitLab again.

    On the Cloud Build history page, you should see the line with the GitLab HTTP URL.

Clone the GitLab repository

You can clone a GitLab repository using tokens or using an SSH key. In this tutorial, you clone the repository using a token. For details of the SSH option, see Building repositories from GitLab.

To be able to clone the GitLab repository, you first need to create the GitLab deploy token.

  1. In GitLab, click Settings and choose Repository.
  2. Click Expand next to Deploy tokens.
  3. Enter cloudbuild in the Name field.
  4. Enter gitlab-token in the Username field.
  5. Select read_repository in Scopes.
  6. Click Create deploy token.
  7. Copy the token value.

    For more information, see Deploy tokens in the GitLab documentation.

  8. Modify the Cloud Build trigger inline config again, replacing [GITLAB_TOKEN] with actual value of the token in the line with the git clone command.

    steps: 
      - name: gcr.io/cloud-builders/git 
        args: 
          - '-c' 
          - 'git clone https://gitlab-token:[GITLAB_TOKEN]@${_REPO_URL}'
        entrypoint: bash 
    substitutions: 
      _GIT_REPO: $(body.project.git_http_url) 
      _REPO_URL: '${_GIT_REPO##https://}'
    

    The parameter project.git\_http\_url from the event payload already contains the protocol: https://<git\_url>. You need to insert the token after the protocol but before the actual URL. To achieve this, you remove the protocol from the string and then build the URL again using bash parameter extension.

  9. To test the new trigger config, trigger the test push event from GitLab.

    You should see the repository successfully cloned in the build logs on the Cloud Build history page.

Move the GitLab token to Secret Manager

For security reasons, you should not store the GitLab token in plaintext in the inline Cloud Build config. In this section, you move the token to Google Cloud Secret Manager.

  1. Go to the Secret Manager page.

  2. Click Create secret.

  3. Enter gitlab-token in the Name field.

  4. Copy the GitLab token into the Secret value field.

  5. Click Create secret to create the secret.

Give the Cloud Build service account access to the secret value

To access the secret value, the Cloud Build service account needs the secretmanager.versions.access role.

  1. Go to the IAM & Admin page in the Cloud Console.

  2. Find the Cloud Build service account with the name <project\_number>@cloudbuild.gserviceaccount.com and click Edit member.

  3. Click Add another role.

  4. Add the Secret Manager Secret Accessor role.

  5. Click Save.

  6. Modify the Cloud Build inline config to read the token value from Secret Manager:

    steps: 
      - name: gcr.io/cloud-builders/git 
        args: 
          - '-c' 
          - 'git clone https://gitlab-token:$$GITLAB_TOKEN@${_REPO_URL} repo'
        entrypoint: bash
        secretEnv:
         - GITLAB_TOKEN
    substitutions:
      _GIT_REPO: $(body.project.git_http_url)
      _REPO_URL: '${_GIT_REPO##https://}'
    availableSecrets:
      secretManager:
        - versionName: 'projects/${PROJECT_ID}/secrets/gitlab-token/versions/latest'
          env: GITLAB_TOKEN
    
  7. Test the new Cloud Build config by triggering a test push event from GitLab.

Apply the Terraform config

In this section, you execute the Terraform code from the Git repository.

  1. Create a Cloud Storage bucket to store Terraform state:

    1. Go to the Cloud Storage page in the Cloud Console.
    2. Click Create bucket.
    3. In the Name your bucket field enter tf-state-[PROJECT_ID], replacing [PROJECT_ID] with your actual project ID.
    4. Click Create.
  2. Modify the inline Cloud Build config as follows:

    steps:
      - name: gcr.io/cloud-builders/git
        args:
          - '-c'
          - 'git clone https://gitlab-token:$$GITLAB_TOKEN@${_REPO_URL} repo'
        entrypoint: bash
        secretEnv:
          - GITLAB_TOKEN
      - name: hashicorp/terraform
        args:
          - init
          - '-backend-config=bucket=${_TF_BACKEND_BUCKET}'
          - '-backend-config=prefix=${_TF_BACKEND_PREFIX}'
        dir: repo
      - name: hashicorp/terraform
        args:
          - plan
          - '-var=project=${PROJECT_ID}'
          - '-out=/workspace/tfplan-$BUILD_ID'
        dir: repo
      - name: hashicorp/terraform
        args:
          - apply
          - '-auto-approve'
          - /workspace/tfplan-$BUILD_ID
        dir: repo
    substitutions:
      _GIT_REPO: $(body.project.git_http_url)
      _REPO_URL: '${_GIT_REPO##https://}'
      _TF_BACKEND_BUCKET: 'tf-state-${PROJECT_ID}'
      _TF_BACKEND_PREFIX: tf-state-prefix
    availableSecrets:
      secretManager:
        - versionName: 'projects/${PROJECT_ID}/secrets/gitlab-token/versions/latest'
          env: GITLAB_TOKEN
    
  3. Trigger a test push event in your GitLab repository.

  4. Go to the Cloud Storage page and observe that there is a bucket called test-bucket-[PROJECT_ID].

After you have followed these steps, whenever you push changes to the Terraform repository, you also trigger the Cloud Build pipeline, which deploys changes to your project. If you decide to add another Terraform project to the same pipeline, you only need to add a webhook to the repository.

Cleaning up

You can clean up all of the resources created in this tutorial by shutting down the project. However, if you used an existing project that contains resources other than those created in this tutorial, you can remove the resources created in this tutorial instead:

  1. In Cloud Shell set the PROJECT_ID environment variable:

    export PROJECT_ID=$(gcloud config get-value project)
    
  2. Delete the Cloud Build trigger:

    gcloud beta builds triggers delete webhook-trigger
    
  3. Delete the secret and token from Secret Manager:

    gcloud secrets delete webhook-trigger-secret
    gcloud secrets delete gitlab-token
    
  4. Delete the IAM policy:

    gcloud projects remove-iam-policy-binding ${PROJECT_ID} \
      --member=serviceAccount:${PROJECT_ID}@cloudbuild.gserviceaccount.com \
      --role=roles/secretmanager.secretAccessor
    
  5. Delete the Cloud Storage buckets:

    gsutil rm -r gs://test-bucket-${PROJECT_ID}
    gsutil rm -r gs://tf-state-${PROJECT_ID}
    
  6. Delete the deploy token key from your repository:

    1. In GitLab, click Settings and choose Repository.
    2. Click Expand next to Deploy tokens.
    3. In the Active deploy tokens block, find the cloudbuild token and click Revoke.
  7. Delete the GitLab repository:

    1. In your GitLab repository, click Settings, and then click General.
    2. Click Expand next to Advanced.
    3. At the bottom of the page, click Delete project.
    4. Confirm deletion by typing the repository name and clicking Yes, delete project.

Submit a tutorial

Share step-by-step guides

Submit a tutorial

Request a tutorial

Ask for community help

Submit a request

View tutorials

Search Google Cloud tutorials

View tutorials

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.