Resource deployment

Last reviewed 2022-12-16 UTC

Resources in the reference architecture can be grouped into one of two categories: foundation or workload components. Foundation resources need to be tightly secured, governed, and audited to help avoid exposing the enterprise to any security or compliance risks. Foundation resources include resources in the hierarchy such as organization policies, folders, projects, APIs, identities (for example, service accounts), role bindings, custom role definitions, Shared VPC networks, subnets, routes (dynamic and static), firewall rules, and Dedicated Interconnect connections. Workload components include resources such as Cloud SQL databases and Google Kubernetes Engine clusters.

Resources within Google Cloud can be deployed from the Google Cloud console web interface, using the gcloud command-line tool, using API calls directly, or using an infrastructure-as-code (IaC) tool. For creating the foundation elements of your enterprise's environment, you should minimize the amount of manual configuration so that you limit the possibility of human error. In the reference architecture, Terraform is used for IaC, deployed through a pipeline that's implemented using Cloud Build.

The combination of Terraform and Cloud Build allows the foundation elements of to be deployed in a consistent and controllable manner. The consistency and controllability of this approach helps enable governance and policy controls across the Google Cloud environment. The reference pipeline architecture is designed to be consistent with most enterprises' security controls.

CICD and seed projects

The organization uses the Cloud Foundation Toolkit to create the basic resources necessary to stand up an IaC environment within Google Cloud. This process creates a bootstrap folder at the root of the organization that contains a CICD project and a seed Terraform project.

The CICD project is a tightly controlled project within the organization hierarchy that's used to host the Cloud Build deployment pipeline.The CICD project is created through a scripted process. It has direct connectivity to the on-premises environment, separate from the connectivity described in Hub-and-spoke transitivity. The CICD project also contains a Cloud Storage bucket for storing build logs.

The seed project includes two Cloud Storage buckets that contain the Terraform state of the foundation infrastructure, five highly privileged service accounts that can create new infrastructure, and the encryption configuration that protects the state buckets. In addition to being encrypted, the Terraform state is protected through storage Object Versioning. When the CI/CD pipeline runs, it impersonates the associated service account in the seed project. The reason for having independent CICD and Terraform seed projects is due to separation of concerns. Although Terraform is used as the IaC tool and has its own requirements, the deployment of that IaC is the responsibility of the CI/CD pipeline.

Deployment pipeline architecture

The following diagram shows the foundation deployment pipeline for Terraform code that defines the infrastructure is stored in an on-premises Git repository. The Cloud Build deployment pipeline connects to the on-premises Git repository through the use of private pools. Private pools are a fully-managed resource that are hosted in the Google-owned service producer network. In order to communicate with the on-premises Git repository, the private pool in the reference architecture uses a VPC peering connection to connect with the CICD network described in CICD and seed projects. This peered connectivity allows code changes in the on-premises repository to trigger the Cloud Build pipeline without having to expose the repository through a public IP address.

Cloud Build provides built-in on-premises trigger integration with several Git repositories, including GitHub Enterprise, GitLab Enterprise, and Bitbucket Data Center.

The foundation deployment pipeline with Cloud Build.

The following table details the security and governance controls that are integrated with the pipeline.

Control Description
Pull request (PR) Code that's merged with the main branch needs to have an approved PR.
Policy checks Policy checks are enforced by the pipeline against the Terraform code using Terraform Validator and the policy library included in the blueprint.
Deployment approval An optional manual approval stage is included in the pipeline for deploying code.
Multiple Cloud Build configurations The Cloud Build deployment pipeline uses multiple configuration files.

Do not hard-code policy directly into Terraform modules or deployment pipelines. Use services such as Terraform Validator or Open Policy Agent to handle policy.

The security foundation is built in five stages: 0-bootstrap, 1-org, 2-environment, 3-network, and 4-project. Each stage consists of two Cloud Build configurations, with each configuration having a separate Cloud Build trigger. One trigger for a stage causes the configuration to create a Terraform Plan, and the other trigger applies the Terraform plan. The security foundation uses Cloud Build custom builders to run Terraform in the private pools. The private pools that Cloud Build uses are a regional resource and you must associate the triggers with the same region.

Each stage in the security foundation build uses the same service account for both the Terraform Plan and Terraform Apply triggers. The Cloud Build service agent is used to impersonate the service accounts for the stage being run. The service accounts used by the Cloud Build service agent are in the seed project. As described in the table below, the service account associated with each stage has only the roles necessary to build out the organizational resources associated with that stage. Because the service accounts include very sensitive permissions, access to the CICD and seed projects is highly restricted.

Cloud Build stage Service account Roles
0-bootstrap Foundation Bootstrap SA
1-org Foundation Organization SA
2-environments Foundation Environment SA
3-networks Foundation Network SA
4-projects Foundation Projects SA

Project deployment

Projects in the organization are deployed through the deployment pipeline. This section describes project attributes that are assigned when the project is created.

Project labels and tags

Project labels are key-value pairs that are included with the billing export into BigQuery, enabling enhanced analysis on billing and resource usage. The table below describes the project metadata that's added to each project in the deployment.

Label Description
business-code A 4-character code that describes which business unit owns the project. The code abcd is used for projects that are not explicitly tied to a business unit.
billing-code A code that's used to provide chargeback information.
primary-contact The primary email contact for the project.
secondary-contact The secondary email contact for the project.
environment A value that identifies the type of environment, such as nonprod or prod.

Tags are key-value pairs similar to labels. However, tags have the following features that make them distinct from labels:

  • Tags are their own Google Cloud resource and are access-controlled through IAM permissions.

  • Tags can be used to conditionally enforce policy.

  • Tags can be inherited.

You can use tags to perform chargebacks, audits, and other cost allocation analysis such as exporting Cloud Billing cost data to BigQuery. See Policy engine services for a list of policy engines that support tags. See Supported service resources for a list of services that you can attach tags to. The reference architecture replicates all the project label metadata as tags and also uses tags with global network firewall policies.

IAM permissions

IAM permissions are defined and created on a per-project basis as part of project deployment.

Google Cloud APIs

Google Cloud APIs are enabled on a per-project basis, and the pipeline has policy checks in place to ensure that only approved APIs can be enabled using an allow/deny list.

Billing account

New projects are linked to the primary billing account. Chargeback to the appropriate business unit is enabled through the use of project labels or tags, as described in Billing exports and chargeback.


Project networking structures such as VPC networks, subnets, firewalls, and routes are enabled through the deployment pipeline.

Project editor

There are two project editors associated with a project. One is the custom service account that's used by the deployment pipeline in the seed project. The other project editor is a firecall account that can be used if automation breaks down or in an emergency, as described in Privileged identities.

Repository structure

The code repository is distributed as a combined single repository to make it easy for you to fork, copy, and use. However, the code has been constructed in such a way that each step in the code is executed through a separate Cloud Build configuration and repository. The top-level folders and the contents of each folder are shown in the table below. You can find the code for in the terraform-example-foundation GitHub repository.

Folder Description components
0-bootstrap This is where initial projects and IAM permissions are deployed for subsequent IaC stages (1-4). bootstrap folder
  • seed project
  • CICD project
    • Cloud Build pipeline
    • Terraform Validator
1-org This is for organization-wide concerns such as policy, log exports, and IAM.

organization policy

organization-wide IAM policy

common folder
  • base_network_hub
  • billing export project
  • log export project
  • interconnect project
  • org-wide secrets project
  • DNS project
  • restricted_network_hub
  • SCC notifications project
2-environments This is for modular creation of new top-level environments, including required projects and the top-level folder. dev folder
  • base Shared VPC host projects
  • restricted Shared VPC host projects
  • environment secrets projects
  • environment monitoring project
nonprod folder
  • (same projects as dev folder)
prod folder
  • (same projects as dev folder)
3-networks This is for modular creation and management of VPC networks.

VPC networks

firewall rules

Cloud Router instances


4-projects This is for creation of projects for different teams or business units, with an application workload focus. application projects

Foundation creation and branching strategy

To build out the foundation, you start by creating a fork of the repository and then create separate repositories for each of the folders in the repository. After you've created separate repositories, you manually deploy the code that's in the 0-bootstrap repository. The code in this repository creates the initial projects and the foundation deployment pipeline. After the code from the 0-bootstrap folder has run, the code in the 1-org folder runs by using the foundation deployment pipeline to create organizational-wide settings and to create the common folder.

After the code has been deployed in the 0-bootstrap and 1-org folders, the remainder of the foundation is built by deploying the code from the 2-environments, 3-networks, and 4-projects folders. The code uses a persistent branch strategy to deploy code through the foundation deployment pipeline to the appropriate environment.

As shown in the diagram below, the organization uses three branches (development, non-production, and production) that reflect the corresponding environments. You should protect each branch through a PR process. Development happens on feature branches that branch off development.

The deployment branching strategy

When a feature or bug fix is complete, you can open a PR that targets the development branch. Submitting the PR triggers the foundation pipeline to perform a plan and to validate the changes against all environments. After you've validated the changes to the code, you can merge the feature or bug fix into the development branch. The merge process triggers the foundation pipeline to apply the latest changes in the development branch to the development environment. After the changes have been validated in the development environment, changes can be promoted to non-production by opening a PR that targets the non-production branch and merging those changes. Similarly, changes can be promoted from non-production to production.

The foundation pipeline and workloads

The pipeline architecture that's described in CICD and seed projects through Repository structure deploys the foundation layer of the Google Cloud organization. You should not use the pipeline for deploying higher-level services or applications. Furthermore, as shown in the diagram below, the access pattern to Google Cloud through deployment pipelines is only one potential access pattern. You might need to evaluate the access pattern and the controls on the workload for each workload individually. The diagram below shows the operator and user personas for the pipelines.

Access patterns for

The foundation provides you with an infrastructure pipeline, detailed in The infrastructure pipeline, that you can use to deploy infrastructure components such as a Cloud SQL instance or a Google Kubernetes Engine (GKE) cluster. The secured foundation also includes an example application pipeline, described in The application pipeline, that you can use to deploy containers to GKE clusters. The application pipeline is maintained in a separate repository from the secured foundation.

The infrastructure pipeline

The infrastructure pipeline that comes with the foundation builds off the Cloud Build-based code of the foundation pipeline. The infrastructure pipeline manages the lifecycle of the infrastructure components independently of the foundation components. The service account that's associated with the infrastructure pipeline has a more limited set of permissions compared to the service account that's associated with the foundation pipeline.

When the foundation pipeline creates a project, it creates a service account that has a controlled set of permissions. The service account that's associated with the infrastructure pipeline is allowed to impersonate the project service account and to perform only those actions that are permitted to the project service account. This strategy allows you to create a clear separation of duties between the people who deploy foundation components and those who deploy infrastructure components.

The infrastructure pipeline is created by the foundation pipeline and deployed in the prj-bu1-c-infra-pipelineandprj-bu2-c-infra-pipeline projects in the common folder, rather than in the CICD project in the seed folder where the foundation pipeline is deployed. To deploy infrastructure components, you create a separate repository to define those components, such as 5-infrastructure. You can then deploy components to your various environments using the same branching strategy that the foundation pipeline uses.

If your organization has multiple business units and you want each business unit to be able to deploy infrastructure components independently, as seen in the diagram below, you can create multiple infrastructure pipelines and repositories. Each of the infrastructure pipelines can have a service account with permission to impersonate only the service accounts of the projects that are associated with that business unit.

Multiple infrastructure pipelines

The application pipeline

The application pipeline that's used with the security foundation is a Cloud Build-based implementation of the secured shift-left pipeline that enables you to deploy applications to a Kubernetes cluster. The application pipeline consists of a continuous integration (CI) pipeline and a continuous deployment (CD) pipeline, as shown in the diagram below.

Application deployment pipeline

The application pipeline uses immutable container images across your environments. This means that the same image is deployed across all environments and will not be modified while it's running. If you must update the application code or apply a patch, you build a new image and redeploy it. The use of immutable container images requires you to externalize your container configuration so that configuration information is read during runtime.

Continuous integration

The application CI pipeline starts when you commit your application code to the release branch; this operation triggers the Cloud Build pipeline. Cloud Build creates a container image, and then Cloud Build pushes the container image to Container Registry, creating an image digest. When you build your application, you should follow best practices for building containers.

After the container has been pushed to the Container Registry, the container is analyzed using the Container Structure Tests framework. This framework performs command tests, file existence tests, file content tests, and metadata tests. The container image then goes through vulnerability scanning to identify vulnerabilities against a vulnerability database that's maintained by Google Cloud. To help avoid compromised artifacts being introduced, IAM policies are used so that only the Cloud Build service account can contribute to the repository.

After the container has successfully gone through vulnerability scanning, the application pipeline uses Binary Authorization to sign an image. Binary Authorization is a service on Google Cloud that provides software supply-chain security for container-based applications by using policies, rules, notes, attestations, attestors, and signers. At deployment time, the Binary Authorization policy enforcer ensures the provenance of the container before allowing the container to deploy.

Continuous deployment

The application CD pipeline uses Cloud Build and Anthos Config Management to enable you to deploy container images to your development, non-production, and production environments. Deployments to your environments are controlled through a persistent branching strategy. To start your deployment, you submit a Kubernetes manifest into a dry repository on the development branch. The manifest contains the image digest of the container or containers that you want to deploy. The initial submission into the dry repository triggers Cloud Build to place the manifest into the wet repository.

The CD pipeline uses Anthos Config Management to monitor the wet repository and to deploy the container or containers that are specified in the manifest file or files to the GKE cluster that corresponds to the Git branch. Anthos Config Management synchronizes the states of your clusters with your Git repository using Config Sync. If Config Sync Operator fails to apply changes to a resource, the resource is left in the last known good state.

After you validate the container image in the development environment, you can promote changes to the non-production environment by opening a PR that targets the non-production branch and then merging the changes. Changes can be promoted from the non-production branch to the production branch in a similar way. In each promotion, the CD pipeline uses the Binary Authorization framework to create a new signed attestation to ensure that the container image has the appropriate providence. If you need to roll back, you can revert to the last known good commit.

What's next