Creating and deploying secured applications

Stay organized with collections Save and categorize content based on your preferences.

The Bank of Anthos secured application example is a container-based web application that lets you simulate an online bank. The example provides you with a reference cloud technology stack that includes the secured foundation, the Google Cloud services that are necessary in order to support the Bank of Anthos, and the Bank of Anthos application code.

The Bank of Anthos secured application example is deployed using the pipelines (foundation pipeline, infrastructure pipeline, and application pipeline) that are described in Resource deployment and using Anthos Config Management.

The example architecture is based on design patterns that are defined in the Hardening your cluster's security guide, the GKE safer cluster repository, the Anthos Multicloud Workshop repository, the Anthos security blueprint, and the Bank of Anthos application. The complete code for the Bank of Anthos secured application example can be found in terraform-example-foundation-app GitHub repository.

The Bank of Anthos secured application platform architecture

Figure 2.12.1 shows how the Bank of Anthos secured application example builds on the secured foundation for its deployment. In this example, the Bank of Anthos is deployed across the three foundation environments: development, production, and non-production. The hub-and-spoke VPC networking topology that's detailed in Hub-and-spoke is used by the application for networking. The application uses Google Cloud projects as a logical security boundary.

Figure 2.12.1 Bank of Anthos project structure on the
secured foundation

Figure 2.12.1 Bank of Anthos project structure on the secured foundation

The Bank of Anthos application adds several projects to the projects that are already defined for the foundation. The additional projects are for deploying resources, configurations, and applications, which are all necessary to support the Bank of Anthos, as shown in Table 2.12.1.

Folder Project Description
common infra-cicd Contains the infrastructure pipeline that's described in The infrastructure pipeline. This project also contains the Anthos Config Management policy repository that's used for GKE cluster configuration.
app-cicd Contains the application pipeline that's described in The infrastructure pipeline.
development, non-production,production boa-gke Contains the Bank of Anthos clusters clusters and Multi Cluster Ingress GKE cluster. Also used as the environ host project for Anthos.
boa-sql Contains the Cloud SQL for PostgreSQL databases that are used by the Bank of Anthos application.
secret Contains Secret Manager and KMS instances for secrets that are specific to the Bank of Anthos application.
monitoring Used for storing environment logs as well as monitoring the environment instance of the Bank of Anthos application.

Table 2.12.1 Additional Bank of Anthos projects

As shown in Figure 2.12.2, a total of three Google Kubernetes Engine (GKE) clusters are created in each environment to deploy the Bank of Anthos. Two clusters (gke-boa-region1-01 and gke-boa-region2) act as identical GKE private clusters in two different regions to provide multi-region resiliency. The third cluster (gke-mci-region1-01) acts as the config cluster for Multi Cluster Ingress.

Figure 2.12.2 Detailed Bank of Anthos architecture

Figure 2.12.2 Detailed Bank of Anthos architecture

Bank of Anthos application components

The Bank of Anthos application example lets users create login accounts, log in to their account, see their transaction history, make deposits, and transfer money to other users' accounts. The application is composed of six services that are described in Table 2.12.2. The services run as containers that connect to each other over REST APIs and gRPC APIs.

Service Language Description
frontend Python Exposes an HTTP server to serve the website. Contains a login page, a signup page, and a home page.
ledger-writer Java Accepts and validates incoming transactions before writing them to the ledger.
balance-reader Java Provides an efficient readable cache of user balances, as read from ledger-db.
transaction-history Java Provides an efficient readable cache of past transactions, as read from ledger-db.
user-service Python Manages user accounts and authentication. The service signs JWTs that are used for authentication by other services.
contacts Python Stores a list of additional accounts that are associated with a user. These accounts are listed in the application's Send Payment and Deposit forms.

Table 2.12.2 Bank of Anthos software components

The application runs on Anthos, Google's managed hybrid multi-cloud application platform. This platform allows you to build, deliver, and manage the lifecycle of your applications. Table 2.12.3 details the Anthos components that are used in the deployment of the Bank of Anthos application.

Anthos component Use
Anthos clusters Container management
Anthos Configuration Management Policy management and enforcement
Anthos Service Mesh Service management
Cloud Operations Observability and platform management
Binary Authorization Container image attestation
Multi Cluster Ingress Multi Cluster Ingress controller for Anthos clusters clusters

Table 2.12.3 Anthos components deployed in the Bank of Anthos application

Distributed services and Anthos Service Mesh

In the Bank of Anthos application, Anthos Service Mesh adds security by providing encryption, mutual authentication, and authorization for all communications between workloads. For this deployment, the Anthos Service Mesh uses its own managed private certificate authority (Mesh CA) for issuing TLS certificates to authenticate peers and to help ensure that only authorized clients can access a service. Using mutual TLS (mTLS) for authentication also helps ensure that all TCP communications are encrypted in transit. For service ingress traffic into the mesh, the Bank of Anthos uses an Istio ingress gateway (istio-ingressgateway).

The Bank of Anthos application runs as a distributed service, which means that it runs as a Kubernetes Service on two or more Kubernetes clusters. The Kubernetes Service runs in the same namespace on each cluster and acts as a single logical service. A distributed service continues running even if one or more GKE clusters are down, as long as the healthy clusters are able to serve the load. In order to create a distributed service across clusters, Anthos Service Mesh provides L4/L7 connectivity between the services running on each cluster and enables them to act as a single logical service.

Bank of Anthos cluster protection

The secured application uses private GKE clusters to increase its security posture; neither the cluster nodes nor the control plane has a public endpoint. The clusters in the secured application are protected by VPC firewall rules and hierarchical firewall policies. As part of the secured application deployment, one Cloud NAT instance is used for each environment to give Pods a mechanism to access resources that have public IPs. Further protection of the cluster is provided by GKE Sandbox, which helps protect the host kernel of the nodes. In addition, the cluster uses Shielded GKE Nodes to limit the ability of an attacker to impersonate a node, and the nodes run Container-Optimized OS to limit their attack surface.

The web frontend of the Bank of Anthos application is exposed to the internet by using Multi Cluster Ingress. Multi Cluster Ingress creates a load balancer for the Bank of Anthos application using Cloud Load Balancing and also creates and manages Network Endpoint Groups (NEGs). The load balancer has the istio-ingressgateway service in both clusters as backends and the NEGs dynamically track the service endpoints of the two istio-ingressgateway services to ensure the load balancer has healthy backends. Cloud Load Balancing uses Google's network infrastructure to provide the Bank of Anthos frontend with an anycast IP address that enables low-latency access to the Bank of Anthos application and helps protect the application frontend from DDoS attacks. The web interface to the Bank of Anthos application is further protected against attacks through the use of Google Cloud Armor.

Cloud Domains is used to register the public domain (boaongcp.cloud) on which the secured application example operates. Cloud DNS is used to provide low-latency and high-availability DNS zone serving.

Bank of Anthos namespaces

The GKE clusters that are used by the Bank of Anthos application are segregated into four namespaces in order to provide separation of services. The traffic flows between namespaces are restricted through network policies. The namespaces are as follows:

  • istio-system, which contains the Ingress gateway.
  • frontend, which contains the Bank of Anthos frontend service.
  • accounts, which contains Bank of Anthos user services.
  • transactions, which contains Bank of Anthos transaction services.

All namespaces that participate in the service mesh are injected with the istio.io/rev=REVISION label. The label allows the resource controller to automatically inject the sidecar Envoy proxy into every Pod within the labeled namespace.

Bank of Anthos identity and access control

The services that constitute the Bank of Anthos application run in Pods that use Workload Identity, which provides each Pod with a unique identity that has only the minimal permissions necessary for the service to operate. Workload Identity lets a Kubernetes service account act as a Google service account by creating a secure mapping between the Kubernetes service account and the Google service account. Pods that run as the Kubernetes service account automatically authenticate as the Google service account when they access Google Cloud APIs because Workload Identity allows Identity and Access Management (IAM) to trust Kubernetes service account credentials.

Access to each GKE control plane is enabled through a bastion host, with one host in each environment. Each bastion is protected by Identity-Aware Proxy. Access to the GKE clusters is controlled by Google Groups for GKE-based Kubernetes role-based access control (RBAC). Groups let you control identities using a central identity management system that's controlled by identity administrators. Instead of updating the RBAC configuration whenever a change is needed for permissions, an administrator can modify group membership.

Bank of Anthos database structure

The Bank of Anthos application uses two PostgreSQL databases. One database (ledger-db) is used for transactions, and the other database (accounts-db) is used for user accounts. The databases are deployed using Google's managed database service, Cloud SQL for PostgreSQL. The databases are configured with cross-region replicas for disaster recovery, and they are encrypted using customer-managed encryption keys. Cloud SQL proxy is used to connect Bank of Anthos microservices to Cloud SQL for PostgreSQL by using service account authentication.

Deployment roles for the Bank of Anthos secured application

The complete application stack of the Bank of Anthos secured application, from foundation to software application, is deployed through a series of automated systems that are described in Resource deployment. These systems are intended to be operated by different groups, as shown in the model in Figure 2.5.3. Table 2.12.4 lists the groups, their roles, the deployment mechanism they use, and the resources that they are responsible for deploying.

Operator team Role Deployment systems Resources deployed
Platform team Responsible for centrally creating and managing the resources that are associated with the security foundation. Foundation pipeline Table 2.12.6
Infrastructure service team Responsible for deploying managed services and configuring the application platform where applications are deployed. Infrastructure pipeline, Anthos Config Management Table 2.12.7, Table 2.12.8, Table 2.12.9
Application team Responsible for developing application code. Application pipeline Table 2.12.2

Table 2.12.4 Operator teams, roles, and the corresponding deployment systems

Using automated pipelines to deploy the secured application stacks makes it possible to build security, auditability, traceability, repeatability, controllability, and compliance into the deployment process. Using different systems that have different permissions and putting different people into different operating groups creates a separation of responsibilities. This lets you follow the principle of least privilege.

Anthos Config Management

The secured application uses Anthos Config Management to manage GKE workload policies and resources. Anthos Config Management uses a Git repository that acts as the single source of truth for declared policies that are stored in a config. Configs are applied to the environments (development, production, and non-production) by using a branching strategy on the Git repository that stores the configs.

Anthos Config Management uses a declarative approach to policy and resource management. It continuously checks cluster state and applies the state that's declared in the config in order to enforce policies.

Anthos Config Management works in conjunction with Policy Controller. Policy Controller is a Kubernetes dynamic admission controller that checks, audits, and enforces your clusters' compliance with policies related to security, regulations, and business rules. Policy Controller enforces your clusters' compliance by using policies called constraints. Policy Controller is built from the Gatekeeper open source project.

Logging and monitoring

Cloud Operations for GKE is used to provide Cloud Logging and Cloud Monitoring services for the Bank of Anthos application. Cloud Operations for GKE provides predefined monitoring dashboards for GKE. Cloud Operations for GKE also allows you to collect system and application logs into central log buckets.

The secured application has a project in each environment folder that's used for storing logs and for a monitoring workspace. The security foundation has a separate logging project where the aggregate Cloud Audit Logs logs from across the entire Google Cloud organization are exported. The logging mechanism described in this section is specific to the secured application.

Security Command Center provides insight into the overall security posture of the secured application. Security Command Center provides the secured application with Container Threat Detection. This service continuously monitors the state of container images, evaluating all changes and remote access attempts to help detect runtime attacks in near real time. Configuration of Security Command Center and Container Threat Detection is described in Security Command Center.

The secured application uses Web Security Scanner to detect vulnerabilities in the Bank of Anthos application. It does this by crawling the application, following all links starting at the base URL (boaongcp.cloud). It then attempts to exercise as many user inputs and event handlers as possible.

Mapping BeyondProd security principles to the secured application

Google pioneered container-based architectures over 15 years ago when we moved to containers and container orchestration to achieve higher resource utilization, to build highly available applications, and to simplify work for Google developers. This container-based architecture required a different security mode.This security model is termed BeyondProd and encompasses several key security principles that map to the Bank of Anthos application architecture as shown in Table 2.12.5.

Security principle Mapping to secured application architecture
Protection of the network at the edge Cloud Load Balancing
Google Cloud Armor
VPC with private GKE clusters
Firewall policy
No inherent mutual trust between services Anthos Service Mesh
Workload Identity
Network Policy
Trusted machines running code with known provenance Binary Authorization
Shielded GKE Nodes
Choke points for consistent policy enforcement across services Anthos Config Management
Policy Controller
Simple, automated, and standardized change rollout Foundation pipeline
Infrastructure pipeline
Application pipeline
Anthos Config Management
Isolation between workloads that share an operating system GKE Sandbox
Container-Optimized OS

Table 2.12.5 Mapping BeyondProd principles to the secured application

Pipelines used to deploy Bank of Anthos architectural components

As mentioned at the beginning of Creating and deploying secured applications, the Bank of Anthos application is deployed using a combination of the foundation pipeline, infrastructure pipeline, manual processes, and Anthos Config Management. Table 2.12.6, Table 2.12.7, Table 2.12.8, and Table 2.12.9 show which components are deployed using which methods.

Architectural component Purpose
Project Provides a logical boundary for Google Cloud resources and services. The boundary is used to segment the Bank of Anthos deployment.
Virtual Private Cloud network Provides networking services to Compute Engine and GKE resources through regional subnets that define the IP address ranges for Bank of Anthos clusters clusters.
VPC firewall rules Defines allow and deny rules that are applied to networking traffic to control access to the Bank of Anthos clusters clusters.
IAM roles Defines permissions that are used within the projects by the Bank of Anthos.
Cloud APIs Enables APIs to support the Bank of Anthos deployment.
Cloud DNS Publishes the Bank of Anthos domain name to the global DNS.

Table 2.12.6 Components deployed by the foundation pipeline

Architectural component Purpose
GKE Provides hosting for the microservices of the containerized Bank of Anthos application.
Cloud SQL for PostgreSQL Provides hosting for the data backend for the Bank of Anthos application.
Cloud Key Management Provides a key encryption key for encryption based on customer-managed encryption keys (CMEK) for Cloud SQL for PostgreSQL, GKE, and Secret Manager.
Secret Manager Provides a secret store for the RSA key pair that's used for JWT-based user authentication.
Compute Engine Provides a bastion host to access the GKE control plane (API server) to bootstrap Anthos Config Management and Anthos Service Mesh.
Static external IP address Provides a reserved IP address that Multi Cluster Ingress binds to a load balancer.
Google Cloud Armor Provides a web-application firewall and DDoS protection.

Table 2.12.7 Components deployed by the infrastructure pipeline

Architectural component Purpose
Anthos Config Management Provides configuration management for the Bank of Anthos application. Anthos Config Management version 1.1 and higher include Policy Controller as one of its components.
Anthos Service Mesh Provides service mesh capability to secure the communication (using mTLS) between microservices of the Bank of Anthos application.

Table 2.12.8 Components deployed through a manual bootstrap process from the bastion host

Architectural component Purpose Configuration area
VirtualService Provides configuration that enables name-based routing and canary rollouts. Istio custom resource
DestinationRule Defines policies, load balancing, mTLS, and circuit breakers.
AuthorizationPolicy Defines access control on workloads in the service mesh.
Service Defines the virtual IP address/DNS name that's used to access the logical set of Pods. Kubernetes workload definitions
Deployment Provides a declarative template for Pods and replica sets.
RBAC (Roles and Bindings) Defines what authorization a Kubernetes service account has at the cluster level or namespace level. Kubernetes identity and authorization
Workload Identity Defines the Google Cloud service account that's used to access Google Cloud resources.
Kubernetes Service Account Defines an identity that's used by a Kubernetes Service.
Namespace Defines the logical clusters within the physical cluster.
Policy Controller Defines constraints that are used to enforce compliance on Kubernetes clusters. Kubernetes hardening

Table 2.12.9 Components deployed by Anthos Config Management

Bank of Anthos resource IP address ranges

The Bank of Anthos secured application example requires multiple IP address ranges to be assigned in the development, non-production, and production environments. Each GKE cluster that's used by the Bank of Anthos needs separate IP address ranges allocated for the nodes, Pods, Services, and control plane. Both the Cloud SQL instances and bastion hosts also require separate IP address ranges. If you need to design your own IP address allocation scheme, you should ensure that you follow the GKE guides and recommendations.

The following tables show the spoke VPC subnets and IP address ranges that are used in the different environments to deploy the Bank of Anthos secured application example:

Resource/subnet Primary IP range Pod IP range Service IP range GKE control plane IP range
Multi Cluster Ingress GKE cluster subnet-usw1-01 10.0.64.0/29 100.64.64.0/22 100.64.68.0/26 100.64.70.0/28
Application GKE cluster region 1 subnet-usw1-02 10.0.65.0/29 100.64.72.0/22 100.64.76.0/26 100.64.78.0/28
Bastion host subnet-usw1-03 10.0.66.0/29
development application GKE cluster region 2 subnet-use4-01 10.1.64.0/29 100.65.64.0/22 100.65.68.0/26 100.65.70.0/28
Cloud SQL 10.16.64.0/24

Table 2.12.10 Bank of Anthos resource IP address ranges for the development environment

Resource/subnet Primary IP range Pod IP range Service IP range GKE control plane IP range
Multi Cluster Ingress GKE cluster subnet-usw1-01 10.0.128.0/29 100.64.128.0/22 100.64.132.0/26 100.64.134.0/28
non-production application GKE cluster region 1 subnet-usw1-02 10.0.129.0/29 100.64.136.0/22 100.64.140.0/26 100.64.142.0/28
non-production bastion host / subnet-usw1-03 10.0.130.0/29
non-production application GKE cluster region 2 subnet-use4-01 10.1.128.0/29 100.65.128.0/22 100.65.132.0/26 100.65.134.0/28
non-production Cloud SQL 10.16.128.0/24

Table 2.12.11 Bank of Anthos resource IP address ranges for the non-production environment

Resource/subnet Primary IP range Pod IP range Service IP range GKE control plane IP range
production Multi Cluster Ingress GKE cluster / subnet-usw1-01 10.0.192.0/29 100.64.192.0/22 100.64.196.0/26 100.64.198.0/28
production App GKE cluster region 1 subnet-usw1-02 10.0.193.0/29 100.64.200.0/22 100.64.204.0/26 100.64.206.0/28
Bastion host subnet-usw1-03 10.0.194.0/29
App GKE cluster region 2 subnet-use4-01 10.1.192.0/29 100.65.192.0/22 100.65.196.0/26 100.65.198.0/28
Cloud SQL instance 10.16.192.0/24

Table 2.12.12 Bank of Anthos resource IP address ranges the production environment