Launching new games or features using existing Cloud Spanner databases with the Load Generator tool

Game developers often reuse their existing database platform to launch their new games or launch new features to their existing games. This requires the database to scale in terms of new schema (tables, index, etc.), data and traffic.

For games built with Cloud Spanner databases, most database operations can be done seamlessly, without affecting production game traffic. Cloud Spanner allows you to make schema updates with no downtime. Customers can also scale their database by adding compute capacity that will automatically balance the data and traffic for the new feature/game launch. 

Most features or games expect a steady growth in traffic and Cloud Spanner database will steadily scale to the growing load. However, some games may want to plan for a knife-edge traffic growth pattern, and can use the Cloud Spanner Load Generator tool. This blog details the steps to execute and verify the warmup, including some best practices to follow.

Scaling up before launch

For games or features being launched on new databases or empty tables, customers can directly use the load generator tool to warm up the database or tables and create splits. Customers can subsequently delete all rows before their launch day and the Cloud Spanner database will scale to the expected production traffic. Customers can follow this post to warm up their database and verify the database performance for a flawless launch. 

For games or features using an existing database or tables (i.e. database or tables that have existing production data), the load generator tool can also be used, though we recommend to track the synthetic data added and delete it before game launch.

Warmup for various schema entities 

Tables and Interleaved Tables

Using `gcsb load` command to seed data into tables will automatically create 5 children rows for each parent row across all interleaved (child) tables. To automatically delete the child table rows created by the load generator, the schema can either set a TTL policy on child tables or set the `ON DELETE CASCADE` clause.  

Index

All  Secondary and Interleaved Index will automatically populate synthetic data based on the data generated for the base table. This exercise creates splits to the Index tables as well, and there is no special handling required for warming-up the Index. Index will also drop the data when the rows from the base table are deleted, either manually using PDML or automatically based on TTL policy.

Foreign Keys

This is currently unsupported in the load generator tool. 

Track and Delete Synthetic Data

1. Add a nullable commit timestamp column to the relevant tables that will be used for the warmup exercise, using the following DDL. It is not recommended to reuse an existing `allow_commit_timestamp` column. The use of a separate column (that is not used by the game application) will be auto-populated by the load generator tool, and provides a strong correlation to synthetic data. All rows generated by production traffic should have the column value as NULL.

2. Add a Cloud Spanner TTL policy using the following DDL. Row deletion policies run at a low priority, ideal for incremental clean-up. However, if the warmup had to create large amounts of data to create adequate splits, this will likely lengthen the time it takes to delete the synthetic data in a busy database because Cloud Spanner's internal work scheduler will prioritize production game traffic.

3. Execute the load generator tool with the load expected at game launch for at least an hour.

4. Verify the performance of the database by executing `gcsb run`, specifically to ensure the latency and throughput of the database.

5. Based on the configured TTL, the synthetic data will be deleted automatically after X days (in our example in step #2, all synthetic rows will be deleted after 1 day).

6. Optionally, you can drop the column (LoadGeneratorDataTimestamp) to avoid confusion or mis-use in your game application.

Closing Remarks

Cloud Spanner recommends to warm up your database before major launch of your application, and suggests the following steps for warmup:

• Make sure that the primary keys you generate for your test data loads are in the same key space. This can be configured in the gscb.yaml config within the load generator tool

• Run a load test no more than two days before your launch. Run the load test for at least one hour at the expected peak load. The load test causes Cloud Spanner to create more splits due to load-based splitting.

• After the load test is complete, you can delete the rows generated by your load test from your tables, but don't delete the tables themselves. This keeps the splits available for your launch window.