Using the Terraform example

Stay organized with collections Save and categorize content based on your preferences.

This section takes you step by step through building a Google Cloud deployment with a secured foundation that you can use to run workloads securely in the cloud. Figure 2.1.1 shows the high-level architecture of example.com, the reference organization used in this guide. The diagram shows a hybrid organization. Some components run on premises, while others are deployed in Google Cloud, with dedicated high-speed connections between the components. The Google Cloud resources are deployed using infrastructure as code (IaC) cloud foundation processes that help make the deployment faster and more repeatable than manual processes.

Figure 2.1.1 The example.com high-level architecture

Figure 2.1.1 The example.com high-level architecture

This guide achieves security for example.com by using Google's opinionated principles in areas including the following:

  • Security
  • Governance
  • Compliance
  • Auditing
  • Provisioning
  • Networking
  • High availability
  • Scaling
  • Monitoring
  • Alerting
  • Chargeback support

The security of example.com takes a unified approach to governance, security objectives, compliance, identity, connectivity, and workloads. This unified approach is used throughout this guide. You can find scripts, code, and other deployment artifacts for the example.com organization in the terraform-example-foundation GitHub repository.

Google Cloud foundation security model

Foundation security in Google Cloud is enabled through a combination of preventative controls and detective controls. Preventative controls are realized through policy and architecture. Policy is defined as a series of programmatic constraints that protect the organization from threats. Architecture is defined as how infrastructure constructs can be used to protect the organization.

Detective controls are defined as monitoring capabilities that look for anomalous or malicious behavior within the organization. Figure 2.2.1 depicts how these three pillars of the security model come together to form the example.com reference architecture.

Figure 2.2.1 The example.com security model

Figure 2.2.1 The example.com security model

Deployment best practices, as detailed in Resource deployment, are implemented through codifying the Google Cloud infrastructure into Terraform modules, putting the modules under source code control, and deploying those modules through an automated pipeline that has policy validation and approval stages. The architectural constructs, as detailed in Authentication and authorization through Logging and monitoring, encompass the folder and project structure of the Google Cloud organization, connectivity, networking structure, firewalls, and identity management. Detective controls are detailed in Detective controls and explain Google Cloud capabilities to detect vulnerabilities, misconfiguration, and malicious behavior in order to enable security posture management. Detective controls also covers how to leverage the native capabilities of Google Cloud in security analytics, as well as how to integrate Google Cloud capabilities with third-party SIEM tools. Billing describes how to interpret billing data from a security perspective. Creating and deploying secured applications describes an example application to show how to design and deploy applications securely, taking advantage of Google Cloud components and controls.

Google Cloud foundation design

Key architectural decisions

The example.com reference architecture that's described in this document is based on certain architectural decisions; if you need to make changes to those assumptions, the example.com architecture also needs to be modified. Table 2.3.1 lists architectural decisions.

Decision area Decision
Organization structure A single organization is used for all environments and services for example.com to manage all resources and policies in one place.
The folder hierarchy has a single layer, consisting of bootstrap, common, production, non-production, and development folders to allow for segregation of policies, privileges, and access.
Google Cloud organization policies are used to augment the organization's security posture by allowing you to define resource configuration constraints that apply consistently across all projects.
Resource deployment Foundation resources are deployed through a deployment pipeline to enable automated and standardized review, approval, and rollback.
Terraform is used as the infrastructure as code (IaC) tool.
An on-premises Git repository is used as a source repository for the Terraform modules.
The deployment pipeline's actions are initiated from the on-premises environment.
An approval stage in the pipeline is needed to deploy resources.
Workloads are deployed through separate pipelines and access patterns.
Authentication and authorization Google Cloud federates with the example.com on-premises Active Directory identity management system.
Single sign-on (SSO) is used to allow users to log into Google Cloud.
A firecall process is used to provide elevated access to the Google Cloud environment.
Groups are used for assigning IAM permissions.
Networking Dedicated Interconnect connections are used to connect the on-premises environment with Google Cloud in a configuration that provides an SLA of 99.99%.
Shared VPC is used as the primary networking topology to enable centralized management of firewall rules and connectivity.
VPC Service Controls provide perimeter protection for services that store highly sensitive data to enable service-level segmentation.
Access to Google Cloud APIs from the cloud and from on-premises is through private IP addresses.
Address allocation for Google Cloud is a contiguous RFC 1918 /16 IP address space.
Cloud DNS is used in Google Cloud, and DNS forwarding is used to communicate with on-premises DNS servers.
Tag-based firewall rules are used to control network traffic flows.
Key and secret management Cloud KMS is used for cryptographic keys
Secret Manager is used to store secrets.
Separate Secret Manager instances are used to store organization-level and folder-level secrets.
Logging Organization-level log sinks are used to aggregate logs.
Logs are sent to Cloud Storage, to BigQuery, and to a SIEM through Pub/Sub.
Detective controls An enterprise SIEM product integrates with Google Cloud Logging.
Security Command Center Premium is enabled to detect infrastructure misconfigurations, vulnerabilities, and active threat behavior, and to share those findings with the enterprise SIEM tools.
Logs in BigQuery are used to augment detection of anomalous behavior by Security Command Center Premium.
Billing Billing alerts are used on a per-project basis with thresholds set at 50%, 75%, 90%, and 95%.

Table 2.3.1 Key architectural decisions

Pre-work

The reference architecture that's described in this document is based on the assumption that you've completed the following pre-work:

For more information, see the Google Cloud onboarding documentation.

Naming conventions

You should have a standardized naming convention for your Google Cloud resources. Table 2.3.2 describes recommended conventions for creating Google Cloud element names for the example.com reference architecture. Table 2.3.3 describes the field values that are used in the names.

Resource Naming convention
Folder

fldr-environment

Example: fldr-prod
Project

prj-business-code-environment-code{-project-label}-unique-number

Example: prj-acde-p-shared-base-43212
VPC

vpc-environment-code-vpc-type{-vpc-label}

Example: vpc-p-shared-base
Subnet

sb-vpc-name-region{-subnet-label}

Example: sb-p-shared-base-us-east1-net1
Firewall

fw-vpc-name-priority-direction-action-src-label-dest-label-protocol-port{-firewall-label}

Example: fw-p-shared-base-1000-i-a-all-all-tcp-80
Cloud router

cr-vpc-name-region{-cloud-router-label}

Example: cr-p-shared-base-us-east1-cr1
Route

rt-vpc-name-priority-instance-tag-next-hop-{route-label}

Example: rt-p-shared-base-1000-all-default-windows-activation
Cloud Interconnect connection

ic-onprem-dc-colo

Example: ic-dal-lga-zone1-1422
Cloud Interconnect VLAN attachment

vl-ic-name-cr-name

Example: vl-dal-da1-zone1-p-shared-base-us-east1-cr1
Group

grp-gcp-group-label

Example: grp-gcp-billing-admin
Role

rl-function{-role-label}

Example: rl-compute-admin
Service account

sa-{-service-account-label}

Example: sa-p-acde-shared-base-data-bkt-reader
Storage bucket

bkt-project-name{-bucket-label}

Example: bkt-p-acde-shared-base-data

Table 2.3.2 Naming conventions for example.com

Field Description Values
environment A description of the folder-level resources within the Google Cloud organization. bootstrap,common,prod, nonprod,dev
environment-code A short form of the environment field. b,c,p, n, d
business-code A 4-character code that's used to associate a project with a business unit or group. A uniquely identifiable 4-character code. For common projects that are not related to a business unit, zzzz is used.
unique-number A globally unique identifier. A 5-digit integer
vpc-type The type of VPC network that's being established. shared, service, float, nic, peer
region The region that the resource is located in. Any valid Google Cloud region. Short forms are used for some names and directions: Australia (au), North American (na), South America (sa), Europe (eu), southeast (se), and northeast (ne).
priority A numerical value that specifies the priority of the Google Cloud route or firewall rule. For details, see the documentation for firewall priorities and routing order.
direction The direction of traffic relative to Google Cloud that the firewall applies. i for ingress, e for egress
action The action to take if a firewall rule matches. a for allow, d for deny
src-label The instance source label to which a firewall is applied. all (indicating 0.0.0.0/0), the source IP address range, or source tags (list)
dest-label The instance destinations to which a firewall is applied. all, the destination tags (list), or the service accounts (list)
protocol The protocols to which a firewall is applied. all, a single protocol, or a combination of protocols (tcp, udp, tcpudp, and so on)
port The port or port range on which a firewall is applied. A port number or port number range
instance-tag The instances to which a route is applied. instance tags
next-hop The next-hop destination where the route will direct traffic, if applicable. default, an instance ID, an IP address, a VPN tunnel name, oran internal load-balancer address
onprem-dc The name of your data center to which an interconnect is connected. Customer dependent
colo The colocation facility name that the interconnect from the on-premises datacenter is peered with. A valid Google Cloud colocation facility code
function The name of the resource type that a custom role is associated with. Resource dependent
*name The name of a resource without its prefix. Resource dependent
*label A descriptive field to enhance the description of the resource. Resource dependent

Table 2.3.3 Naming convention field values

The example.com Google Cloud organization structure

The foundation of creating deployments within Google Cloud is the organization, which you create by setting up Cloud Identity or Google Workspace. The Google Cloud organization provides a resource hierarchy that provides an ownership structure for resources and attachment points for policy and access control.

The resource hierarchy consists of folders, projects, and resources, and it defines the shape and use of Google Cloud services within an organization. Policies are inherited, and these policies can't be altered by resource owners who are lower in the hierarchy. Folders and projects inherently provide you with secure-by-default isolation because all external network and access permissions are denied by default and therefore must either be explicitly enabled or enabled by inheritance from a higher level in the hierarchy.

Folder structure is dependent on the nature of the organization and the type of workload being deployed. Figure 2.4.1 shows the example.com reference architecture.

The architecture consists of five folders as described in Folders and of a number of projects that are detailed in Projects. The organizational structure for example.com is codified in Terraform modules and deployed through the deployment pipeline as detailed in Resource deployment.

Figure 2.4.1 The example.com organization structure

Figure 2.4.1 The example.com organization structure

Folders

You use folders as a method of grouping projects into related groups. You can apply security policies and access permissions at the folder level; these policies and permissions are then inherited by resources within the folder structure.

The key concept of folders is that they separate projects into logical groupings; you can then apply consistent policy to the child projects at the folder level to ease administrative overhead. As shown in Table 2.4.1, the reference architecture used by example.com uses five folders. Three of them (production, non-production, and development) separate the organizations into different environments. These folders are deployed through the deployment pipeline detailed in Resource deployment.

Folder Description
bootstrap Contains the seed and CICD projects that are used to deploy foundation components.
common Contains projects with cloud resources used by the organization.
production Contains projects with cloud resources that have been promoted into production.
non-production Contains a replica of the production environment to let you test workloads before you put them into production.
development Used as a development and sandbox environment.

Table 2.4.1 Folders used in the example.com reference architectures

Projects

Projects in Google Cloud are constructs that contain cloud resources. Each project inherits the policy of the parent folder or folders and of the Google Cloud organization. You can enable or disable APIs within each project to support the services that you need for that project. Each project is associated with a billing account, as explained in Billing account.

Common folder and bootstrap folder projects

In the example.com architecture, a series of projects reside under the common folder and bootstrap folder that contain resources that are used across the example.com organization. These projects, detailed in Table 2.4.2, provide various enterprise functions and are created through the infrastructure deployment pipeline.

Project Description More information
CICD Houses the deployment pipeline that's used to build out the foundation components of the organization. This project is highly restricted. Resource deployment
seed Contains the Terraform state of your infrastructure and Terraform service account to update the state. This project is highly restricted. Resource deployment
interconnect Houses the Cloud Interconnect connections that provide connectivity between an enterprise's on-premises environment and Google Cloud. Enterprise-to-Google Cloud connectivity
DNS hub Provides a central point of communication between an enterprise's on-premises DNS system and Cloud DNS. DNS setup
org-secrets Contains organization-level secrets. Key and secret management
logging Provides a destination for log sink and detective controls. Logging and monitoring
Security Command Center Provides a standalone project for managing Security Command Center alerting. Security Command Center
billing Contains a BigQuery dataset with the organization's billing exports. Billing

Table 2.4.2 Projects in the common folder

Projects present in all environment folder

Under each environment folder (production, non-production, and development) in the example.com reference architecture, a series of common projects are created, as detailed in Table 2.4.3. These projects provide resource connectivity to the on-premises environment and to a deployment environment for workload-specific resources.

Project type Description
Base Shared VPC host project Provides the project for the Shared VPC network that hosts the organization's workloads that don't require VPC Service Controls.
Restricted Shared VPC host project Provides the project for the Shared VPC network that will host the organization's workloads that run within a perimeter controlled by VPC Service Controls.
secrets Contains folder-level secrets.
application projects Involve projects where application resources are deployed. Project deployment patterns are described in Project deployment patterns and consist of Shared VPC service project patterns, floating project patterns, peered project patterns, and dual-NIC project patterns.

Table 2.4.3 Environment folder project types

Organization policy setup

You can use the Organization Policy Service to apply policies to a Google Cloud organization. The organizational policies are inherited by all child resources of the resource node to which the policy is set, unless an explicit override is set on a child resource. Organization policy constraints in the example.com reference architecture are defined in Terraform modules and deployed through the deployment pipeline. Table 2.4.4 provides a list and description of the policies that are set as part of the example.com foundation.

Policy constraint Description Recommended value
compute.disableNestedVirtualization (boolean) When true, disables hardware-accelerated nested virtualization for all Compute Engine VMs. true
compute.disableSerialPortAccess (boolean) When true, disables serial port access to Compute Engine VMs. true
compute.disableGuestAttributesAccess (boolean) When true, disables Compute Engine API access to the guest attributes of Compute Engine VMs. true
compute.vmExternalIpAccess (list) Defines the set of Compute Engine VM instances that are allowed to use external IP addresses. deny all=true
compute.skipDefaultNetworkCreation (boolean) When true, causes Google Cloud to skip the creation of the default network and related resources during Google Cloud project resource creation. true
compute.restrictXpnProjectLienRemoval (boolean) When true, restricts the set of users that can remove a Shared VPC project lien. true
sql.restrictPublicIp (boolean) When true, restricts public IP addresses on Cloud SQL instances. true
iam.allowedPolicyMemberDomains (list) Defines the set of members that can be added to IAM policies. The allowed/denied list must specify one or more Cloud Identity or Google Workspace customer IDs. Note that domain restricted sharing can interfere with some Google Cloud services, and you might need to provide exceptions for some Google Cloud services. your-cloud-identity-id
iam.disableServiceAccountKeyCreation (boolean) When true, disables the creation of service account external keys. true
storage.uniformBucketLevelAccess (boolean) When true, requires buckets to use uniform IAM-based bucket-level access. true
iam.automaticIamGrantsForDefaultServiceAccounts (boolean) When true prevents the default App Engine and Compute Engine service accounts that are created in your projects from being automatically granted any IAM role on the project when the accounts are created true

Table 2.4.4 Organizational policies in example.com

Additional policy controls

In addition to the organizational policy constraints detailed in the previous section, Google Cloud has additional policies that you can apply to an organization to tighten the organization's security posture. Table 2.4.5 lists the additional policy controls that are used in the example.com reference architecture.

Control Description Management
Limit session and gcloud timeouts You can change the session timeout for Google Cloud sessions (including the gcloud tool) to durations as low as 1 hour. Google Workspace or Cloud Identity Admin Console
Disable Cloud Shell Google Cloud provides a native Cloud Shell interface for working with the cloud environment. You can disable this shell. Admin Console
Use phishing- resistant security keys You can enable phishing-resistant security keys as a second-factor authentication (2FA) method that helps provide protection against automated bots, bulk phishing, and targeted attacks. The keys use public key cryptography to verify a user's identity, and they use the URL of the login page to help make sure that an attacker can't access a user account even if the user is tricked into providing a username and password. Google Workspace or Cloud Identity Admin Console. If you're federating identities from an external system, you might need to configure this in your identity provider, or add an additional layer of protection on top of your existing SSO settings directly within Cloud Identity.
Enable access transparency Access transparency provides you with logs that capture the actions that Google personnel take when accessing your content. You must be on Gold, Platinum, Enterprise, or Premium support plans. Contact Google Cloud Support
Enable access approval Access approval enables you to explicitly approve access to your data or configurations on Google Cloud before Google Cloud support or engineering can access the data. Google Cloud console
Restrict data storage locations You can limit the physical location of a new resource in Google Cloud by using resource location constraints. Same mechanism as organization policy controls detailed in Organization policy setup.

Table 2.4.5 Additional example.com policy controls

Restricting resource locations

You can limit the physical location of a new resource in Google Cloud by using resource location constraints. You use the same mechanism that's used to create organizational policy controls as described in Organizational policy setup to restrict resource locations in the example.com organization. When you set up resource restriction policies, you can specify the location type (multi-region, region, or zone) that's associated with the policy. Be aware that if you use resource location constraints, some services might not work correctly, as described in Resource Locations Supported Services.

Assured Workloads

For your regulated workloads that are required to be compliant with specific regulatory compliance regimes (for example, FedRAMP Moderate, FedRAMP High, IL4, CJIS, or ITAR), Google Cloud provides Assured Workloads. You can apply the technical controls and resource location restrictions provided by Assured Workloads to a folder to support your regulatory compliance requirements.

Assured Workloads provides the following security controls:

  • Data residency. Customer data is stored in a multi-region or a region that you select. If your developer attempts to store data at rest in a region outside of the selection, that action will be restricted.
  • Personnel data access controls based on attributes. This feature ensures that only Google personnel who satisfy compliance requirements such as location and background checks are able to perform compliant data access requests, and only while performing support operations for Google's regulated customers.
  • Personnel support case ownership controls based on attributes. Only Google support personnel who satisfy compliance requirements such as location and background checks are able to provide support for Assured Workloads customers.
  • Encryption. Google-managed and customer-managed encryption keys are FIPS-140-2 compliant, and they support regulatory compliance for encryption.

You should use Assured Workloads only if your Google Cloud use case is actively subject to regulatory compliance, or if you want to have additional security controls applied to your projects. You might want these controls so that your projects conform to a compliance regime's security posture, even if the projects are not subject to a direct requirement from a regulatory body. If you use Assured Workloads, we recommend that you select only the compliance regime for your use case. If you are not bound by a compliance regime but you want to apply additional security controls, you can use the US Regions and Support or EU Regions and Support compliance regime selector in the Assured Workloads creation workflow. To see locations where Assured Workloads is available, see Assured Workloads locations.

Assured Workloads are enabled at the folder level, and they require that the folder be registered with Google Cloud before you deploy workloads. To register a folder, you use an onboarding form. Registration can take up to 10 business days. After the folder has been registered, additional folders and projects can be created in that folder (in this blueprint, the Assured folder) for workloads that are subject to various compliance regimes. The example.com code repository provides you with the deployment artifacts that you can use to create compliant workload projects after the folder has been registered.

Google Cloud console

Google Cloud lets you restrict access to the Google Cloud console. You can use context-aware access to provide granular access controls to the console based on a user's identity and the context of their access request. To use context-aware access, you create an access level in Access Context Manager that allows access only from a specified range of IP addresses. For example, you can allow access from the public IP address range of your corporate network. After you create the access context, you create a Google Group in Cloud Identity and add users to that group who have context-aware access restrictions.You then create an access binding between the access level and the Google group.