Configuring multicluster allocation

This page explains how to configure multicluster allocation for Game Servers. Multicluster allocation allows you to request a GameServer in a Ready state from one cluster within a realm, and get back a GameServer in a Ready state from any cluster within that realm. Multicluster allocation requires manual configuration and setup to enable the clusters to connect to one another and also to the multicluster allocation endpoint.

To learn more about GameServer allocations, see the Agones Create a Game Server Fleet quickstart for a comprehensive overview.

Before you begin

Before you start, we recommend you familiarize yourself with key concepts in the Game Servers Overview. Make sure you have also performed the following tasks:

  • Ensure that you have enabled the Game Services API.
  • Enable Game Services API
  • Either choose a shell with Cloud SDK installed, or use an API client:
  • Cloud Shell

    To launch Cloud Shell, perform the following steps:

    1. Go to Google Cloud Console.

      Google Cloud Console

    2. From the upper-right corner of the console, click the Activate Cloud Shell button:

    A Cloud Shell session opens inside a frame lower on the console. You use this shell to run gcloud commands.

    Local shell

    To install gcloud, install the Cloud SDK, which includes the gcloud command-line tool.

    Verify that you have set desired default project for gcloud command-line tool (otherwise you need to specify flag --project explicitly for each command later):

    gcloud config list project
    

    If not you can run the following command to set a default project, replacing PROJECT_ID with your desired project ID :

    gcloud config set project PROJECT_ID
    

    Run the following command to verify your version of the Google Cloud SDK. Game Servers requires version 306.0.0 or higher of the SDK.

    gcloud version
    

    To update your installation, run the following command:

    gcloud components update
    

    curl / PowerShell

    To use the REST API with curl or Windows PowerShell, do the following:

    1. Create a service account.
    2. Download a private key as a JSON file.
    3. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains your service account key. This variable only applies to your current shell session, so if you open a new session, set the variable again.

    Client Library

    Google Cloud Game Servers can be controlled programmatically using a client library. See Client Libraries Overview for instructions on using the library and authenticating.

Configuring the Agones allocator service

Make sure that the Agones allocator service is set up for your Agones installation and that the single cluster allocation works using the same guide. Note that, by default, the agones-allocator service does not have a valid service certificate installed and the certificate needs to be replaced. You can reserve a regional Google Cloud Platform IP address and install the agones-allocator using the reserved IP, which also issues a valid certificate for the service. To reserve an IP, run the following command:

gcloud compute addresses create allocator-service --region REGION

To find the reserved IP, run the following command:

gcloud compute addresses describe allocator-service --region REGION --format="value(address)"

Then install/upgrade Agones by passing the reserved IP as RESERVED_IP from the previous command:

helm upgrade RELEASE --install --set agones.allocator.http.loadBalancerIP=RESERVED_IP --namespace agones-system --create-namespace agones/agones

RESERVED_IP is the IP address to use for the agones-allocator service.

RELEASE is the name of your Agones release for helm chart installation.

Installing Citadel

Game Servers uses the Citadel project as a certificate manager for the multicluster allocation client-side certificate, used by Agones to facilitate secure connections between clusters. You must install it on each Google Kubernetes Engine cluster registered with Game Servers.

For every namespace on the cluster, Citadel will issue an istio.default Kubernetes secret that is used by Game Servers as the client-side certificate.

You can skip this step if you have already installed Istio on your Google Kubernetes Engine cluster. Note that there are currently known issues with Agones compatibility with a full Istio installation.

If you're a Terraform user, you can install Citadel using Terraform. You can review the example configuration files for installing Citadel using Terraform on GitHub.

This guide assumes you have command line tools installed for helm v3 and git.

To install Citadel:

  1. Clone the Istio GitHub repo:

    git clone -b release-1.5 https://github.com/istio/istio.git
    
  2. Create a new namespace for Istio:

    kubectl create ns istio-system
    

  3. Copy and run each of the following commands to generate Citadel YAML installation files from Istio Helm templates:

    helm template istio/install/kubernetes/helm/istio --name-template istio --namespace istio-system -s charts/security/templates/serviceaccount.yaml -s charts/security/templates/clusterrole.yaml -s charts/security/templates/clusterrolebinding.yaml -s charts/security/templates/deployment.yaml > citadel.yaml
    

  4. Next, run the following command to apply the YAML file you generated in the previous step to the Google Kubernetes Engine cluster:

    kubectl apply -f citadel.yaml
    
  5. Validate that Citadel is working correctly:

    kubectl get pods --namespace=istio-system
    

    Verify that the Citadel deployment is running in a single pod.

Networking requirements for multicluster allocation

For multicluster allocation to work in your realm, you need to ensure that each cluster is running an allocator service where the service has a public external address.

To verify that this is configured, check that an external load balancer is allocated for the service:

kubectl get service agones-allocator -n agones-system
NAME               TYPE                      CLUSTER-IP   EXTERNAL-IP    PORT(S)
agones-allocator   LoadBalancer              10.86.3.77   RESERVED_IP  443:30219/TCP

RESERVED_IP must be non-empty.

Validating multicluster allocation

To test the allocation process, complete the following steps:

  1. Register 2 Game Servers clusters in your realm and configure them for multicluster allocation as shown above.

  2. Create a Game Servers deployment.

  3. Create a Game Servers config with a fleet spec that has replicas set to 5 and the label gameName: udp-server applied.

  4. Update the rollout for the Game Servers deployment to make the config live.

  5. Copy the following YAML into a LOCAL_FILE :

    apiVersion: "allocation.agones.dev/v1"
    kind: GameServerAllocation
    spec:
      multiClusterSetting:
        enabled: true
      required:
        matchLabels:
          gameName: udp-server
    

Use the following command to apply the YAML file to one of the Game Servers clusters in your realm:

kubectl create -f LOCAL_FILE -o yaml

This will allocate 1 of the 10 ready replicas across the two clusters. Repeat the command 9 more times and ensure that all 10 replicas in both clusters are allocated.

Any endpoint on any cluster within the realm can be used to allocate a GameServer from any cluster in the realm. The GameServer allocations are load balanced across all clusters within the realm using a round-robin scheme.

Troubleshooting

If you run into issues with the procedures in this guide, we recommend that you review the following troubleshooting documents:

No multicluster allocation policy is specified

You attempt to allocate a game server and see the following error:

no multi-cluster allocation policy is specified

Verify that you have correctly installed citadel on the cluster by making sure the istio.default secret is created by citadel in the game server namespace:

kubectl get secret istio.default -n NAMESPACE

Certificate signed by unknown authority

You attempt to allocate a game server and see the following error:

transport: authentication handshake failed: x509: certificate signed by unknown authority

Verify that you have correctly set the allocator-tls-ca Kubernetes secret for all clusters. Helm installation also updates the allocator-tls-ca secret, but if you manually change the TLS secret, you need to update the allocator-tls-ca secret as well. To verify the certificate authority, run the following commands:

kubectl get secret allocator-tls-ca -n agones-system -ojsonpath="{.data.tls-ca\.crt}" | base64 -d > ca.crt
kubectl get secret allocator-tls -n agones-system -ojsonpath="{.data.tls\.crt}" | base64 -d > tls.crt
openssl verify -verbose -CAfile ca.crt tls.crt

Verification should print out tls.crt: OK.

Root cause fixed, issue remains

Reconciling changes made to a single cluster across all clusters can take up to an hour. You can trigger reconciliation immediately by making a change to your Game Servers resources, for example, by updating labels on a realm resource:

gcloud game servers realms update REALM_NAME --update-labels=usage=testing --location=REALM_LOCATION

What's next