This tutorial shows you how to set up the infrastructure of the Autoscaler tool for Spanner using a distributed deployment topology. In the deployment option presented in this tutorial, all the components of the Autoscaler are in a single project, except for Cloud Scheduler, the Forwarder topic and its corresponding function.
This tutorial is intended for IT, Operations, and Site Reliability Engineering teams who want to reduce operational overheads and optimize the cost of their Spanner deployments. This document is part of a series:
- Autoscaling Spanner
- Deploy a per-project or centralized Autoscaler tool for Spanner
- Deploy a distributed Autoscaler tool for Spanner (this document)
The distributed deployment topology brings together the following advantageous features of both per-project and the centralized deployments:
- Teams who own the Spanner instances, called application teams, manage Autoscaler configuration parameters for their instances with Cloud Scheduler jobs that they own.
- Outside of the configuration parameters, a central team manages the rest of Autoscaler infrastructure, minimizing management overhead.
Architecture
The following diagram shows Autoscaler architecture in a distributed deployment topology:
For an explanation of the components of Autoscaler and each of the numbered steps in the flow of events, see Autoscaling Spanner.
Forwarder function
Cloud Scheduler can only publish messages to topics in the same project, so for the distributed topology, this tutorial introduces an intermediate component called the Forwarder function.
The Forwarder function takes messages published to Pub/Sub from Cloud Scheduler, checks their JSON syntax, and forwards them to the Poller Pub/Sub topic. The topic can belong to a separate project to the Cloud Scheduler.
The following diagram shows the components used for the forwarding mechanism:
As shown in the preceding diagram, the Spanner instances are in projects named Application 1 and Application 2:
- Cloud Scheduler is the same project as the Spanner instances.
(2a) Cloud Scheduler publishes its messages to the Forwarder topic in the Application 1 and Application 2 projects.
(2b) The Forwarder function reads messages from the Forwarder topic.
(2c) The Forwarder function forwards messages to the Polling topic residing in the Autoscaler project.
The Poller function reads the messages from the polling topic and the process continues, as described in the Autoscaling Spanner.
Objectives
- Deploy Autoscaler using a distributed deployment model.
- Import existing Spanner instances into Terraform state.
- Configure Autoscaler.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you follow the instructions in this tutorial, the costs associated with the operation of Autoscaler components should be zero or close to zero. However, this estimate does not include the costs for the Spanner instances. For an example of how to calculate the costs of Spanner instances see Autoscaling Spanner.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
In a distributed deployment topology, in addition to an Autoscaler project, you set up a second project, which in this tutorial is referred to as the Application project. The Application project holds the application resources, including Spanner. You set up and enable billing and APIs for these two projects separately in this tutorial.
-
In the Google Cloud console, activate Cloud Shell.
In Cloud Shell, clone the following repository:
git clone https://github.com/cloudspannerecosystem/autoscaler
Export variables for the working directories:
export AUTOSCALER_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/autoscaler-project" export APP_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/app-project"
Preparing the Autoscaler project
In this section, you prepare the deployment of the project which contains all of the centralized Autoscaler infrastructure except for Cloud Scheduler.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Make sure that billing is enabled for your Google Cloud project.
Enable the Identity and Access Management (IAM), Resource Manager, App Engine Admin, Firestore, Spanner, Pub/Sub, Cloud Run functions, and Cloud Build APIs.
In Cloud Shell, set environment variables with the ID of your Autoscaler project:
export AUTO_SCALER_PROJECT_ID=INSERT_YOUR_PROJECT_ID gcloud config set project "${AUTO_SCALER_PROJECT_ID}"
Set the region and zone and App Engine location (for Firestore) for Autoscaler infrastructure:
export AUTO_SCALER_REGION=us-central1 export AUTO_SCALER_ZONE=us-central1-c export AUTO_SCALER_APP_ENGINE_LOCATION=us-central
Create a service account:
gcloud iam service-accounts create terraformer --display-name "Terraform service account"
Terraform uses this account to create all the resources in your infrastructure.
Give the project owner role to the service account:
gcloud projects add-iam-policy-binding "${AUTO_SCALER_PROJECT_ID}" \ --member "serviceAccount:terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" \ --role roles/owner
Create a service account key file:
gcloud iam service-accounts keys create \ --iam-account "terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" "${AUTOSCALER_DIR}/key.json"
If your project does not have a Firestore instance yet, create one:
gcloud app create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}" gcloud alpha firestore databases create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}"
Deploying the Autoscaler
In this section, you deploy the components that make up Autoscaler by using Terraform files in the following directories:
Directory | Directory contents |
---|---|
terraform/ |
Top-level configuration, which includes each of the deployment options and the reusable modules. |
|
Configuration for the project where Autoscaler is located. Delegates to the Autoscaler module. |
terraform/modules/autoscaler-functions |
Configuration of the Poller and Scaler Cloud Run functions, and Pub/Sub topics. |
In Cloud Shell, set the project ID, region, zone and App Engine location in the corresponding Terraform environment variables:
export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}" export TF_VAR_region="${AUTO_SCALER_REGION}" export TF_VAR_zone="${AUTO_SCALER_ZONE}" export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"
Change the directory into the Terraform scaler-project directory and initialize it:
cd "${AUTOSCALER_DIR}" terraform init
Create the Autoscaler infrastructure:
terraform apply -parallelism=2
After you verify the resources, type
yes
when prompted.When you run this command in Cloud Shell, you might encounter the following error message:
"Error: cannot assign requested address"
This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
terraform apply -parallelism=1
Preparing the Application project
In this section, in the project which contains the Spanner instances, you prepare the deployment of the Cloud Scheduler and Forwarder topic and function.
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Make sure that billing is enabled for your Google Cloud project.
Enable the Identity and Access Management, Resource Manager, App Engine Admin, Firestore, Spanner, Pub/Sub, Cloud Run functions, and Cloud Build APIs.
In Cloud Shell, set the environment variables with the ID of the Application project:
export APP_PROJECT_ID=INSERT_YOUR_APP_PROJECT_ID gcloud config set project "${APP_PROJECT_ID}"
Set the region and zone and App Engine Location for the Application project:
export APP_REGION=us-central1 export APP_ZONE=us-central1-c export APP_APP_ENGINE_LOCATION=us-central
Create a service account for Terraform to use to create the resources in your infrastructure:
gcloud iam service-accounts create terraformer --display-name "Terraform service account"
Give the project owner role(
roles/owner
) to the service account:gcloud projects add-iam-policy-binding "${APP_PROJECT_ID}" \ --member "serviceAccount:terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com" \ --role roles/owner
Create a service account key file:
gcloud iam service-accounts keys create \ --iam-account "terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com" "${APP_DIR}/key.json"
Create an application to enable Cloud Scheduler:
gcloud app create --region="${APP_APP_ENGINE_LOCATION}"
You don't need to create a Firestore database because the state is stored in the Autoscaler project.
Deploy the Application project infrastructure
The Terraform components that make up the Application project are in the following directories:
terraform/
: Top-level configuration, which includes each of the deployment options and the reusable modules.terraform/distributed/app-project/
: Configuration for the Application project where Spanner is. Uses the following modules:terraform/modules/scheduler/
: Configuration of Cloud Scheduler for triggering polling.terraform/modules/spanner/
: Configuration of Spanner databaseterraform/modules/forwarder/
: Configuration of the Forwarder Cloud Run functions.
In this section, you use the Terraform files to deploy the components that make up the Application project.
In Cloud Shell, set the project ID, region, zone, and App Engine location in the corresponding Terraform environment variables:
export TF_VAR_project_id="${APP_PROJECT_ID}" export TF_VAR_region="${APP_REGION}" export TF_VAR_zone="${APP_ZONE}" export TF_VAR_location="${APP_APP_ENGINE_LOCATION}"
Set the project ID for where the Autoscaler state is stored:
export TF_VAR_state_project_id="${AUTO_SCALER_PROJECT_ID}"
The Autoscaler state includes the timestamps when the scaling events were triggered for each instance.
If you want to create a new Spanner instance for testing Autoscaler, set the following variable:
export TF_VAR_terraform_spanner=true
The Spanner instance that Terraform creates is named
autoscale-test
.If you don't want to create a new Spanner instance because you already have an instance for Autoscaler to monitor, set the name of your instance in the following variable:
export TF_VAR_spanner_name=INSERT_YOUR_SPANNER_INSTANCE_NAME
For more information about how to set Terraform to manage yourSpanner instance, see Importing your Spanner instances
Change the directory into the
Terraform
app-project directory and initialize it:cd "${APP_DIR}" terraform init
Create the infrastructure in the Application project:
terraform import module.scheduler.google_app_engine_application.app "${APP_PROJECT_ID}" terraform apply -parallelism=2
Verify that the list of resources for Terraform to create is correct, then type
yes
when prompted.When you run this command in Cloud Shell, you might encounter the following error message:
"Error: cannot assign requested address"
This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
terraform apply -parallelism=1
Authorize the Forwarder Cloud Run functions to publish to the Poller topic
In Cloud Shell, switch back to Autoscaler terraform project directory and ensure that the Terraform variables are set correctly:
cd "${AUTOSCALER_DIR}" export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}" export TF_VAR_region="${AUTO_SCALER_REGION}" export TF_VAR_zone="${AUTO_SCALER_ZONE}" export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"
Set the Terraform variables for your Forwarder service accounts. Update and add your service accounts as required:
export TF_VAR_forwarder_sa_emails='["serviceAccount:forwarder-sa@'"${APP_PROJECT_ID}"'.iam.gserviceaccount.com"]' terraform apply -parallelism=2
Verify that the list of resources for Terraform to create is correct, then type
yes
when prompted.When you run this command in Cloud Shell, you might encounter the following error message:
"Error: cannot assign requested address"
This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
terraform apply -parallelism=1.
Verifying your deployment
When the infrastructure for the Autoscaler is ready, you can configure its parameters. To learn more, see Configuring Autoscaler.
Because you create a distributed deployment in this tutorial, logs have the following properties:
- Logs from the Poller and Scaler functions appear in the Logs Explorer for the Autoscaler project.
- Logs about syntax errors in the JSON configuration of the Cloud Scheduler payload appear in the Logs Explorer for each Application project. This lets the team responsible for a specific Spanner instance troubleshoot its configuration issues independently.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the project
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
What's next
- Read about the Autoscaler Architecture.
- Read about how to deploy Autoscaler in per-project or centralized mode.
- Read more about Spanner recommended thresholds.
- Read more about Spanner CPU utilization metrics and latency metrics.
- Read more about Spanner schema design best practices.
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.