Rolling out a canary config

This page describes two different options for rolling out a new Game Servers config with a canary phase. In the canary phase, you deploy the new config to small set of customers. This helps you to identify potential problems without impacting all your users.

The first way is to use a canary region. The setup for this option is simpler, but a bad rollout affects the entire region. The other option is to create a canary deployment. This option is more flexible, but it's also a more complex process for you to manage.

Before you begin

Before you start, we recommend you familiarize yourself with key concepts in the Game Servers Overview. Make sure you have also performed the following tasks:

Ensure that you have enabled the Game Services API.

Enable Game Services API

Either choose a shell with gcloud CLI installed, or use an API client:

Cloud Shell

To launch Cloud Shell, perform the following steps:

Go to Google Cloud console.

Google Cloud console
From the upper-right corner of the console, click the Activate Cloud Shell button:

A Cloud Shell session opens inside a frame lower on the console. You use this shell to run gcloud commands.

Local shell

Install the gcloud CLI.

Verify that you have set desired default project for Google Cloud CLI (otherwise you need to specify flag --project explicitly for each command later):

gcloud config list project

If not you can run the following command to set a default project, replacing PROJECT_ID with your desired project ID :

gcloud config set project PROJECT_ID

Run the following command to verify your version of the Google Cloud CLI. Game Servers requires version 306.0.0 or higher of the gcloud CLI.

gcloud version

To update your installation, run the following command:

gcloud components update

curl / PowerShell

To use the REST API with curl or Windows PowerShell, do the following:

Create a service account.
Download a private key as a JSON file.
Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your service account key. This variable only applies to your current shell session, so if you open a new session, set the variable again.
Example: Linux or macOS
```
export GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"
```
Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:
```
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"
```
Example: Windows

For PowerShell:
```
$env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"
```
Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:
```
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"
```
For command prompt:
```
set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH
```
Replace KEY_PATH with the path of the JSON file that contains your service account key.

Client Library

Google Cloud Game Servers can be controlled programmatically using a client library. See Client Libraries Overview for instructions on using the library and authenticating.

Verifying the health of a rollout

Make sure you check the health of your servers at various points during the rollout. You can use the data you gather to help you decide whether it is safe to continue with the rollout. If the health checks reveal a problem, it's best to pause or cancel the rollout.

You can use the following recommendations to verify the health of a rollout:

Make sure that the state of individual game server instances is Ready.
Verify that Agones can allocate and manage the full lifecycle of a GameServer. Ensure that both the main and canary deployment GameServers have the same labels that match to the expected GameServerAllocation required selector, so that your matchmaker doesn't require a change to allocate from the canary deployment. Optionally, you could preferentially allocate from the canary fleet using the "preferred" attribute in a GameServerAllocation specification.
Monitor and make sure that any relevant game metrics and system metrics in your deployment don't indicate any degradation in performance.

Rolling out to a canary region

Game Servers lets you override a config in one or more realms. Make sure that the clusters in your canary region are in a separate realm.

In this example, the "australia" realm is the canary realm.

Architecture that uses a separate realm for canary

To set v2 as the active config in the australia realm:

Copy the following code into a file:

- realmsSelector:
    realms:
        - projects/games/locations/australia-southeast1/realms/australia
  configVersion: v2

To apply the changes, run the following command:
```
gcloud game servers deployments update-rollout stk --config-overrides-file configOverrideFile  --no-dry-run
```
where configOverrideFile is the path to the file with override config.

Architecture after canary rollout

When you have confirmed that the canary region is healthy, run the following command to deploy the new version everywhere:

gcloud game servers deployments update-rollout stk --default-config "v2" --no-dry-run

When the operation completes, the v2 config is rolled out everywhere. Architecture after full rollout

Now clean up the override by running the following command:

gcloud game servers deployments update-rollout stk --no-dry-run --clear-config-overrides

Rolling out to a canary deployment

You can use a separate deployment to test the config. Initially, there isn't a dedicated canary deployment.

Separate canary deployment initial state

Start by creating the canary deployment:
```
gcloud game servers deployments create canary
```
Make sure that the selector used in the GameServerAllocation is able to select GameServers from both your main fleet and the canary fleet, so that no changes are required to your current allocation strategies.
Next, run the following command to create a v1 config that is identical to the active config in the non-canary deployment:
```
gcloud game servers configs create v1 --deployment canary --fleet-configs-file fleetSpecFile --scaling-configs-file scalingConfigFile
```
where fleetSpecFile is the path of the file that contains the fleet specification. scalingConfigFile is the path of the file that contains all of the scaling configs.

We recommended setting the scaling configs so that the canary fleets are much smaller than the regular fleets.
Run the following command to update the canary rollout so that the v1 config is rolled out everywhere:
```
gcloud game servers deployments update-rollout canary --default-config v1 --no-dry-run
```
Verify that the canary fleets are healthy.
Run the following command to create the v2 config in the canary deployment:
```
gcloud game servers configs create v2 --deployment canary --fleet-configs-file fleetSpecFile --scaling-configs-file scalingConfigFile
```
where fleetSpecFile is the path of the file that contains the fleet specification. scalingConfigFile is the path of the file that contains all of the scaling configs. We recommend you configure the canary fleets to a small size.
Run the following command to roll out the v2 config to the canary deployment:
```
gcloud game servers deployments update-rollout canary --default-config v2 --no-dry-run
```
Verify that the canary fleets are healthy.
Run the following command to roll out the v2 config to the main deployment:
```
gcloud game servers deployments update-rollout stk --default-config v2 --no-dry-run
```
Verify that the fleets are healthy. At this stage, the v2 is rolled out everywhere.

Clean up the canary deployment.

Clear the rollout for the canary deployment by running:

gcloud game servers deployments update-rollout canary --clear-default-config --no-dry-run

Run the following command to delete the 'v1' config:

gcloud game servers configs delete v1 --deployment=canary

Delete the 'v2' config by running:

gcloud game servers configs delete v2 --deployment=canary

Remove the canary deployment by running:

gcloud game servers deployments delete canary

Architecture after cleanup

For more fine-grained control over a deployment, you can roll out a config to selected realms using overrides, before setting the config as the default. Follow the same procedure to roll out the canary more gradually.

You can skip the first three steps and the last step by maintaining a standing canary deployment. For frequent and regular rollouts, we recommend this approach.