Deploy a distributed Autoscaler tool for Spanner

This tutorial shows you how to set up the infrastructure of the Autoscaler tool for Spanner using a distributed deployment topology. In the deployment option presented in this tutorial, all the components of the Autoscaler are in a single project, except for Cloud Scheduler, the Forwarder topic and its corresponding function.

This tutorial is intended for IT, Operations, and Site Reliability Engineering teams who want to reduce operational overheads and optimize the cost of their Spanner deployments. This document is part of a series:

Autoscaling Spanner
Deploy a per-project or centralized Autoscaler tool for Spanner
Deploy a distributed Autoscaler tool for Spanner (this document)

The distributed deployment topology brings together the following advantageous features of both per-project and the centralized deployments:

Teams who own the Spanner instances, called application teams, manage Autoscaler configuration parameters for their instances with Cloud Scheduler jobs that they own.
Outside of the configuration parameters, a central team manages the rest of Autoscaler infrastructure, minimizing management overhead.

Architecture

The following diagram shows Autoscaler architecture in a distributed deployment topology:

Distributed deployment topology.

For an explanation of the components of Autoscaler and each of the numbered steps in the flow of events, see Autoscaling Spanner.

Forwarder function

Cloud Scheduler can only publish messages to topics in the same project, so for the distributed topology, this tutorial introduces an intermediate component called the Forwarder function.

The Forwarder function takes messages published to Pub/Sub from Cloud Scheduler, checks their JSON syntax, and forwards them to the Poller Pub/Sub topic. The topic can belong to a separate project to the Cloud Scheduler.

The following diagram shows the components used for the forwarding mechanism:

Forwarding mechanism.

As shown in the preceding diagram, the Spanner instances are in projects named Application 1 and Application 2:

Cloud Scheduler is the same project as the Spanner instances.
(2a) Cloud Scheduler publishes its messages to the Forwarder topic in the Application 1 and Application 2 projects.

(2b) The Forwarder function reads messages from the Forwarder topic.

(2c) The Forwarder function forwards messages to the Polling topic residing in the Autoscaler project.
The Poller function reads the messages from the polling topic and the process continues, as described in the Autoscaling Spanner.

Objectives

Deploy Autoscaler using a distributed deployment model.
Import existing Spanner instances into Terraform state.
Configure Autoscaler.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you follow the instructions in this tutorial, the costs associated with the operation of Autoscaler components should be zero or close to zero. However, this estimate does not include the costs for the Spanner instances. For an example of how to calculate the costs of Spanner instances see Autoscaling Spanner.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

In a distributed deployment topology, in addition to an Autoscaler project, you set up a second project, which in this tutorial is referred to as the Application project. The Application project holds the application resources, including Spanner. You set up and enable billing and APIs for these two projects separately in this tutorial.

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

In Cloud Shell, clone the following repository:

git clone https://github.com/cloudspannerecosystem/autoscaler

Export variables for the working directories:

export AUTOSCALER_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/autoscaler-project"
export APP_DIR="$(pwd)/autoscaler/terraform/cloud-functions/distributed/app-project"

Preparing the Autoscaler project

In this section, you prepare the deployment of the project which contains all of the centralized Autoscaler infrastructure except for Cloud Scheduler.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector
Make sure that billing is enabled for your Google Cloud project.
Enable the Identity and Access Management (IAM), Resource Manager, App Engine Admin, Firestore, Spanner, Pub/Sub, Cloud Run functions, and Cloud Build APIs.
Enable the APIs

In Cloud Shell, set environment variables with the ID of your Autoscaler project:

export AUTO_SCALER_PROJECT_ID=INSERT_YOUR_PROJECT_ID
gcloud config set project "${AUTO_SCALER_PROJECT_ID}"

Set the region and zone and App Engine location (for Firestore) for Autoscaler infrastructure:

 export AUTO_SCALER_REGION=us-central1
 export AUTO_SCALER_ZONE=us-central1-c
 export AUTO_SCALER_APP_ENGINE_LOCATION=us-central

Create a service account:
```
gcloud iam service-accounts create terraformer --display-name "Terraform service account"
```
Terraform uses this account to create all the resources in your infrastructure.

Give the project owner role to the service account:

 gcloud projects add-iam-policy-binding "${AUTO_SCALER_PROJECT_ID}" \
    --member "serviceAccount:terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" \
    --role roles/owner

Create a service account key file:

 gcloud iam service-accounts keys create \
    --iam-account "terraformer@${AUTO_SCALER_PROJECT_ID}.iam.gserviceaccount.com" "${AUTOSCALER_DIR}/key.json"

If your project does not have a Firestore instance yet, create one:

 gcloud app create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}"
 gcloud alpha firestore databases create --region="${AUTO_SCALER_APP_ENGINE_LOCATION}"

Deploying the Autoscaler

In this section, you deploy the components that make up Autoscaler by using Terraform files in the following directories:

Directory	Directory contents
`terraform/`	Top-level configuration, which includes each of the deployment options and the reusable modules.
`terraform/cloud-functions/distributed/autoscaler-project/`	Configuration for the project where Autoscaler is located. Delegates to the Autoscaler module.
`terraform/modules/autoscaler-functions`	Configuration of the Poller and Scaler Cloud Run functions, and Pub/Sub topics.

In Cloud Shell, set the project ID, region, zone and App Engine location in the corresponding Terraform environment variables:

export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}"
export TF_VAR_region="${AUTO_SCALER_REGION}"
export TF_VAR_zone="${AUTO_SCALER_ZONE}"
export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"

Change the directory into the Terraform scaler-project directory and initialize it:
```
cd "${AUTOSCALER_DIR}"
terraform init
```
Create the Autoscaler infrastructure:
```
terraform apply -parallelism=2
```
After you verify the resources, typeyes when prompted.

When you run this command in Cloud Shell, you might encounter the following error message: "Error: cannot assign requested address"

This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
```
terraform apply -parallelism=1
```

Preparing the Application project

In this section, in the project which contains the Spanner instances, you prepare the deployment of the Cloud Scheduler and Forwarder topic and function.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector
Make sure that billing is enabled for your Google Cloud project.
Enable the Identity and Access Management, Resource Manager, App Engine Admin, Firestore, Spanner, Pub/Sub, Cloud Run functions, and Cloud Build APIs.
Enable the APIs

In Cloud Shell, set the environment variables with the ID of the Application project:

export APP_PROJECT_ID=INSERT_YOUR_APP_PROJECT_ID
gcloud config set project "${APP_PROJECT_ID}"

Set the region and zone and App Engine Location for the Application project:

export APP_REGION=us-central1
export APP_ZONE=us-central1-c
export APP_APP_ENGINE_LOCATION=us-central

Create a service account for Terraform to use to create the resources in your infrastructure:

gcloud iam service-accounts create terraformer --display-name "Terraform service account"

Give the project owner role(roles/owner) to the service account:

 gcloud projects add-iam-policy-binding "${APP_PROJECT_ID}" \
    --member "serviceAccount:terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com" \
    --role roles/owner

Create a service account key file:

 gcloud iam service-accounts keys create \
    --iam-account "terraformer@${APP_PROJECT_ID}.iam.gserviceaccount.com"  "${APP_DIR}/key.json"

Create an application to enable Cloud Scheduler:
```
gcloud app create --region="${APP_APP_ENGINE_LOCATION}"
```
You don't need to create a Firestore database because the state is stored in the Autoscaler project.

Deploy the Application project infrastructure

The Terraform components that make up the Application project are in the following directories:

terraform/: Top-level configuration, which includes each of the deployment options and the reusable modules.
terraform/distributed/app-project/: Configuration for the Application project where Spanner is. Uses the following modules:
- terraform/modules/scheduler/: Configuration of Cloud Scheduler for triggering polling.
- terraform/modules/spanner/: Configuration of Spanner database
- terraform/modules/forwarder/: Configuration of the Forwarder Cloud Run functions.

In this section, you use the Terraform files to deploy the components that make up the Application project.

In Cloud Shell, set the project ID, region, zone, and App Engine location in the corresponding Terraform environment variables:

  export TF_VAR_project_id="${APP_PROJECT_ID}"
  export TF_VAR_region="${APP_REGION}"
  export TF_VAR_zone="${APP_ZONE}"
  export TF_VAR_location="${APP_APP_ENGINE_LOCATION}"

Set the project ID for where the Autoscaler state is stored:
```
export TF_VAR_state_project_id="${AUTO_SCALER_PROJECT_ID}"
```
The Autoscaler state includes the timestamps when the scaling events were triggered for each instance.
If you want to create a new Spanner instance for testing Autoscaler, set the following variable:
```
export TF_VAR_terraform_spanner=true
```
The Spanner instance that Terraform creates is named autoscale-test.

If you don't want to create a new Spanner instance because you already have an instance for Autoscaler to monitor, set the name of your instance in the following variable:
```
export TF_VAR_spanner_name=INSERT_YOUR_SPANNER_INSTANCE_NAME
```
For more information about how to set Terraform to manage yourSpanner instance, see Importing your Spanner instances
Change the directory into the Terraform app-project directory and initialize it:
```
cd "${APP_DIR}"
terraform init
```
Create the infrastructure in the Application project:
```
terraform import module.scheduler.google_app_engine_application.app "${APP_PROJECT_ID}"
terraform apply -parallelism=2
```
Verify that the list of resources for Terraform to create is correct, then type yes when prompted.

When you run this command in Cloud Shell, you might encounter the following error message: "Error: cannot assign requested address"

This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
```
terraform apply -parallelism=1
```

Authorize the Forwarder Cloud Run functions to publish to the Poller topic

In Cloud Shell, switch back to Autoscaler terraform project directory and ensure that the Terraform variables are set correctly:

cd "${AUTOSCALER_DIR}"

export TF_VAR_project_id="${AUTO_SCALER_PROJECT_ID}"
export TF_VAR_region="${AUTO_SCALER_REGION}"
export TF_VAR_zone="${AUTO_SCALER_ZONE}"
export TF_VAR_location="${AUTO_SCALER_APP_ENGINE_LOCATION}"

Set the Terraform variables for your Forwarder service accounts. Update and add your service accounts as required:
```
 export TF_VAR_forwarder_sa_emails='["serviceAccount:forwarder-sa@'"${APP_PROJECT_ID}"'.iam.gserviceaccount.com"]'
  terraform apply -parallelism=2
```
Verify that the list of resources for Terraform to create is correct, then type yes when prompted.

When you run this command in Cloud Shell, you might encounter the following error message:"Error: cannot assign requested address"

This error is a known issue in the Terraform Google provider. In this case, retry with the following command:
```
terraform apply -parallelism=1.
```

Verifying your deployment

When the infrastructure for the Autoscaler is ready, you can configure its parameters. To learn more, see Configuring Autoscaler.

Because you create a distributed deployment in this tutorial, logs have the following properties:

Logs from the Poller and Scaler functions appear in the Logs Explorer for the Autoscaler project.
Logs about syntax errors in the JSON configuration of the Cloud Scheduler payload appear in the Logs Explorer for each Application project. This lets the team responsible for a specific Spanner instance troubleshoot its configuration issues independently.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

In the Google Cloud console, go to the Manage resources page.
Go to Manage resources
In the project list, select the project that you want to delete, and then click Delete.
In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

Read about the Autoscaler Architecture.
Read about how to deploy Autoscaler in per-project or centralized mode.
Read more about Spanner recommended thresholds.
Read more about Spanner CPU utilization metrics and latency metrics.
Read more about Spanner schema design best practices.
Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.