The enterprise application blueprint is deployed through a series of automated systems and pipelines. Each pipeline deploys a particular aspect of the blueprint. Pipelines provide a controllable, auditable, and repeatable mechanism for building out the blueprint. The following diagram shows the interaction of the various pipelines, repositories, and personas.
The blueprint uses the following pipelines:
- The foundation infrastructure pipeline (part of the enterprise foundations blueprint) deploys the application factory, the multi-tenant infrastructure pipeline, and the fleet-scope pipeline.
- The multi-tenant infrastructure pipeline deploys the GKE clusters, and the other managed services that the enterprise application blueprint relies on.
- The fleet-scope pipeline configures fleet scopes, namespaces, and RBAC roles and bindings.
- The application factory provides a mechanism to deploy new application pipelines through a templated process.
- The application CI/CD pipeline provides a CI/CD pipeline to deploy services into GKE clusters.
- Config Sync deploys and maintains additional Kubernetes configurations, including Policy Controller constraints.
Repositories, repository contributors, and repository change approvers
The blueprint pipelines are triggered through changes to Git repositories. The following table describes the repositories that are used throughout the blueprint, who contributes to the repository, who approves changes to the repository, which pipeline uses the repository, and the description of what the repository contains.
| Repository | Repository contributor | Repository change approver | Pipeline | Description |
|---|---|---|---|---|
|
Developer platform developer |
Developer platform administrator |
Foundation infrastructure pipeline |
Repository that contains the code to deploy the multi-tenant infrastructure pipeline, the application, and the fleet-scope pipeline |
|
Developer platform developer |
Developer platform administrator |
Multi-tenant infrastructure |
The Terraform modules that are used by developer platform teams when they create the infrastructure |
|
Developer platform developer |
Developer platform administrator |
Fleet-scope |
The repository that defines the fleet team scopes and namespaces in the fleet |
|
Developer platform developer |
Developer platform administrator |
Application factory |
The code that defines the application repository and references the
modules in the |
|
Application developer |
Application operator |
Application factory |
The base code that is placed in the |
|
Developer platform developer |
Developer platform administrator |
Application factory Multi-tenant infrastructure |
The Terraform modules that define the application and the infrastructure |
|
Application developer |
Application operator |
Application CI/CD |
The application code that is deployed into the GKE clusters |
|
Developer platform developer |
Developer platform administrator |
Config Sync |
The policies that are used by the GKE clusters to maintain their configurations |
Automated pipelines help build security, auditability, traceability, repeatability, controllability, and compliance into the deployment process. By using different systems that have different permissions and putting different people into different operating groups, you create separation of responsibilities and follow the principle of least privilege.
Foundation infrastructure pipeline
The foundation infrastructure pipeline is described in the enterprise foundations blueprint and is used as a generic entrypoint for further resource deployments. The following table describes the components that the pipeline creates.
| Component | Description |
|---|---|
Creates the shared infrastructure that is used by all tenants of the developer platform. |
|
Creates namespaces and RBAC role bindings. |
|
Creates the application CI/CD pipelines that are used to deploy the services. |
Multi-tenant infrastructure pipeline
The multi-tenant infrastructure infrastructure pipeline deploys fleets, GKE clusters, and related shared resources. The following diagram shows the components of the multi-tenant infrastructure pipeline.
The following table describes the components that the multi-tenant infrastructure pipeline builds.
| Component | Description |
|---|---|
GKE clusters |
Provides hosting for the services of the containerized application. |
Policy Controller |
Provides policies that help ensure the proper configuration of the GKE clusters and services. |
Config Sync |
Applies the Policy Controller policies to clusters and maintains consistant appplication of the policies. |
Creates the encryption key that is based on the customer-managed encryption key (CMEK) for GKE, AlloyDB for PostgreSQL, and Secret Manager. |
|
Secret Manager secret |
Provides a secret store for the RSA key pair that's used for user authentication with JSON Web Tokens (JWT). |
Google Cloud Armor security policy |
Provides the policy that's used by the Google Cloud Armor web-application firewall. |
Fleet-scope pipeline
The fleet-scope pipeline is responsible for configuring the namespaces and RBAC bindings in the fleet's GKE clusters. The following table describes the components that the fleet-scope pipeline builds.
| Component | Description |
|---|---|
Defines the logical clusters within the physical cluster. |
|
Defines the authorization that a Kubernetes service account has at the cluster level or namespace level. |
Application factory
The application factory is deployed by the foundation infrastructure pipeline and is used to create infrastructure for each new application. This infrastructure includes a Google Cloud project which holds the application CI/CD pipeline.
As engineering organizations scale, the application team can onboard new applications using the application factory. Scaling enables growth by adding discrete application CI/CD pipelines and supporting the infrastructure for deploying new applications within the multi-tenant architecture. The following diagram shows the application factory.
The application factory has the following components:
- Application factory repository: A Git repository that stores the declarative application definition.
- Pipelines to create applications: Pipelines that require
Cloud Build to complete the following:
- Create a declarative application definition and store it in the application catalog.
- Apply the declarative application definition to create the application resources.
- Starter application template repository: Templates for creating a simple application (for example, a Python, Golang, or Java microservice).
- Shared modules: Terraform modules that are created with standard practices and that are used for multiple purposes, including application onboarding and deployment.
The following table lists the components that the application factory creates for each application.
| Component | Description |
|---|---|
Application source code repository |
Contains source code and related configuration used for building and deploying a specific application. |
Application CI/CD pipeline |
A Cloud Build based pipeline that is used to connect to the source code repository and provides a CI/CD pipeline for deploying application services. |
Application CI/CD pipeline
The application CI/CD pipeline enables automated build and deployment of container-based applications. The pipeline consists of continuous integration (CI) and continuous deployment (CD) steps. The pipeline architecture is based on the Secure CI/CD blueprint.
The application CI/CD pipeline uses immutable container images across your environments. Immutable container images help ensure that the same image is deployed across all environments and isn't modified while the container is running. If you must update the application code or apply a patch, you build a new image and redeploy it. The use of immutable container images requires you to externalize your container configuration so that configuration information is read during run time.
To reach GKE clusters over a private network path and manage kubeconfig
authentication, the application CI/CD pipeline interacts with the
GKE clusters through the Connect gateway. The pipeline also uses
private pools
for the CI/CD environment.
Each application source code repository includes Kubernetes configurations. These configurations enable applications to successfully run as Kubernetes services on GKE. The following table describes the types of Kubernetes configurations that the application CI/CD pipeline applies.
| Component | Description |
|---|---|
Defines a scaled set of pods (containers). |
|
Makes a deployment reachable over the cluster network. |
|
Makes a service part of the service mesh. |
|
Defines how peers on the service mesh should reach a virtual service. Used in the blueprint to configure locality load balancing for east-west traffic. |
|
Sets access control between workloads in the service mesh. |
|
Defines the identity that's used by a Kubernetes service. Workload Identity Federation for GKE defines the Google Cloud service account that's used to access Google Cloud resources. |
|
Allows external ingress traffic to reach a service. The gateway is only required by deployments that receive external traffic. |
|
Configure SSL, Google Cloud Armor, session affinity, and connection draining for deployments that receive external traffic. GCPBackendPolicy is used only by deployments that receive external traffic. |
|
Configures collection of Prometheus metrics exported by an application. |
Continuous integration
The following diagram shows the continuous integration process.
The process is the following:
- A developer commits application code to the application source repository. This operation triggers Cloud Build to begin the integration pipeline.
- Cloud Build creates a container image, pushes the container image to Artifact Registry, and creates an image digest.
- Cloud Build performs automated tests for the application. Depending on the application language, different testing packages may be performed.
- Cloud Build performs the following scans on the container image:
- Cloud Build analyzes the container using the Container Structure Tests framework. This framework performs command tests, file existence tests, file content tests, and metadata tests.
- Cloud Build uses vulnerability scanning to identify any vulnerabilities in the container image against a vulnerability database that's maintained by Google Cloud.
- Cloud Build approves the image to continue in the pipeline after successful scan results.
- Binary Authorization signs the image. Binary Authorization is a service on Google Cloud that provides software supply-chain security for container-based applications by using policies, rules, notes, attestations, attestors, and signers. At deployment time, the Binary Authorization policy enforcer helps ensure the provenance of the container before allowing the container to deploy.
- Cloud Build creates a release in Cloud Deploy to begin the deployment process.
To see the security information for a build, go to the Security insights panel. These insights include vulnerabilities that were detected using Artifact Analysis, and the build's level of security assurance denoted by SLSA guidelines.
Continuous deployment
The following diagram shows the continuous deployment process.
The process is the following:
- At the end of the build process, the application CI/CD pipeline creates a new Cloud Deploy release to launch the newly built container images progressively to each environment.
- Cloud Deploy initiates a rollout to the first environment of the deployment pipeline, which is usually development. Each deployment stage is configured to require manual approval.
- The Cloud Deploy pipelines uses sequential deployment to deploy images to each cluster in an environment in order.
- At the end of each deployment stage, Cloud Deploy verifies the functionality of the deployed containers. These steps are configurable within the Skaffold configuration for the applications.
Deploying a new application
The following diagram shows how the application factory and application CI/CD pipeline work together to create and deploy a new application.
The process for defining a new application is the following:
- An application operator defines a new application within their tenant by executing a Cloud Build trigger to generate the application definition.
- The trigger adds a new entry for the application in Terraform and commits the change to the application factory repository.
- The committed change triggers the creation of application-specific repositories and projects.
- Cloud Build completes the following:
- Creates two new Git repositories to host the application's source code and IaC.
- Pushes the Kubernetes manifests for network policies, and Workload Identity Federation for GKE to the Configuration management repository.
- Creates the application's CI/CD project and the Cloud Build IaC trigger.
- The Cloud Build IaC trigger for the application creates the application CI/CD pipeline and the Artifact Registry repository in the application's CI/CD project.
- Config Sync deploys the network policies and Workload Identity Federation for GKE configurations to the multi-tenant GKE clusters.
- The fleet scope creation pipeline creates the fleet scope and namespace for the application on multi-tenant GKE clusters.
- The application's CI/CD pipeline performs the initial deployment of the application to the GKE clusters.
- Optionally, the application team uses the Cloud Build IaC trigger to deploy projects and additional resources (for example, databases and other managed services) to dedicated single-tenant projects, one for each environment.
GKE Enterprise configuration and policy management
In the blueprint, developer platform administrators use Config Sync to create cluster-level configurations in each environment. Config Sync connects to a Git repository which serves as the source of truth for the chosen state of the cluster configuration. Config Sync continuously monitors the actual state of the configuration in the clusters and reconciles any discrepancies by applying updates to ensure adherence to the chosen state, despite manual changes. Configs are applied to the environments (development, non-production, and production) by using a branching strategy on the repository.
In this blueprint, Config Sync applies Policy Controller constraints. These configurations define security and compliance controls as defined by developer platform administrators for the organization. This blueprint relies on other pipelines to apply other configurations: the application CI/CD pipelines apply application-specific configuration, and the fleet-scope pipeline creates namespaces and associated role bindings.
What's next
- Read about the Cymbal Bank application architecture (next document in this series).