This tutorial provides you with recommended practices for creating a stateful application and upgrading the Google Kubernetes Engine (GKE) cluster that's running the application. This tutorial provides examples for deploying a Redis application, but the same concepts are applicable to other types of stateful applications deployed on GKE.
Glossary
Terms used in this tutorial:
- Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.
- A Redis Cluster is a distributed implementation of Redis that provides high availability. Redis Clusters consist of leader nodes and follower nodes.
- A stateful application can remember information about its state each time that it runs. Redis is a popular in-memory database for stateful applications.
Objectives
This tutorial covers the following steps:
- In GKE, create a Redis Cluster with three leaders and three followers on top of three GKE nodes.
- Deploy a Redis client application. The application counts the number of requests to a website.
- Upgrade your cluster with surge upgrade.
- Test the application's workload disruption and status disruption.
The following diagram shows you an overview of the cluster architecture you create by completing these objectives:
Costs
This tutorial uses billable components of Google Cloud, including:
- GKE
Use the pricing calculator to generate a cost estimate based on your projected usage. New Google Cloud users might be eligible for a free trial.
Before you begin
Take the following steps to enable the Kubernetes Engine API:- Visit the Kubernetes Engine page in the Google Cloud Console.
- Create or select a project.
- Wait for the API and related services to be enabled. This can take several minutes.
-
Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.
Install the following command-line tools used in this tutorial:
-
gcloud
is used to create and delete Kubernetes Engine clusters.gcloud
is included in the Google Cloud SDK. -
kubectl
is used to manage Kubernetes, the cluster orchestration system used by Kubernetes Engine. You can installkubectl
usinggcloud
:gcloud components install kubectl
Creating a GKE cluster
In this section, you create a cluster in GKE with three nodes and verify that the cluster is functioning.
Setting defaults for the gcloud command-line tool
To save time typing your project
ID and
Compute Engine zone options in the
gcloud
command-line tool, set the following defaults:
gcloud config set project PROJECT-ID
gcloud config set compute/zone COMPUTE-ZONE
Creating a GKE cluster
To create your GKE cluster, complete the following steps:
Create a cluster named
redis-test
that has three nodes:gcloud container clusters create redis-test \ --num-nodes=3
Once the cluster is created, you should see output similar to the following example:
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS redis-test COMPUTE-ZONE 1.15.12-gke.20 35.232.77.38 n1-standard-1 1.15.12-gke.20 3 RUNNING
Configure
kubectl
to communicate with the cluster:gcloud container clusters get-credentials redis-test
Verify that your cluster is running:
kubectl get nodes
You should see output similar to the following example:
NAME STATUS ROLES AGE VERSION gke-redis-test-default-pool-c4e4225c-mw3w Ready <none> 2m1s v1.15.12-gke.20 gke-redis-test-default-pool-c4e4225c-pv51 Ready <none> 2m1s v1.15.12-gke.20 gke-redis-test-default-pool-c4e4225c-whl5 Ready <none> 2m1s v1.15.12-gke.20
Creating a Redis Cluster on GKE
In this section you create a Redis Cluster on top of the GKE cluster you created in the previous section by creating a ConfigMap, StatefulSet, and a headless Service.
Clone the sample manifests:
git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples cd kubernetes-engine-samples/hello-app-redis/manifests
A ConfigMap named
redis-configmap.yaml
stores the Redis configuration.To learn more about the Redis parameters in this ConfigMap, see the Redis Cluster configuration parameters section in the Redis Cluster tutorial.
Deploy the ConfigMap by running the following command:
kubectl apply -f redis-configmap.yaml
The Statefulset named
redis-cluster.yaml
. has the following key fields:- The
replicas
field is set to6
. This is so that every GKE node has two Pods, for the 3 Redis leaders and 3 Redis followers. - The
volumeClaimTemplates
field provides stable storage using PersistentVolumes. - The
affinity
field creates a Pod anti-affinity rule to spread Pods across the Kubernetes nodes.
- The
Deploy the StatefulSet by running the following command:
kubectl apply -f redis-cluster.yaml
The headless Service named
redis-service.yaml
is for the Redis nodes' connection. To create a headless Service, set theclusterIP
field toNone
.Deploy the Service by running the following command:
kubectl apply -f redis-service.yaml
Wait approximately two minutes and verify all the Pods are running by using the following command:
kubectl get pods
You should see output similar to the following example:
NAME READY STATUS RESTARTS AGE redis-0 1/1 Running 0 2m29s redis-1 1/1 Running 0 2m8s redis-2 1/1 Running 0 107s redis-3 1/1 Running 0 85s redis-4 1/1 Running 0 54s redis-5 1/1 Running 0 23s
Verify the persistent volumes were created by running the following command:
kubectl get pv
You should see output similar to the following example:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-HASH 1Gi RWO Delete Bound default/data-redis-5 standard 75s pvc-HASH 1Gi RWO Delete Bound default/data-redis-1 standard 2m59s pvc-HASH 1Gi RWO Delete Bound default/data-redis-3 standard 2m16s pvc-HASH 1Gi RWO Delete Bound default/data-redis-2 standard 2m38s pvc-HASH 1Gi RWO Delete Bound default/data-redis-0 standard 3m20s pvc-HASH 1Gi RWO Delete Bound default/data-redis-4 standard 104s
In this output, HASH represents a hash which is attached to each persistent volume name.
Assigning roles to your Redis Cluster
Once the configuration is complete, you assign roles to your Redis Cluster.
To assign the roles, complete the following steps:
Set the first three Redis nodes as leaders and last three Redis nodes as followers:
Retrieve and make a copy of your Pod IP addresses:
kubectl get pods -l app=redis -o jsonpath='{range.items[*]}{.status.podIP} '
Assign the leader and follower roles by pasting each of the Pod IP addresses into the following command and type
yes
when prompted:kubectl exec -it redis-0 -- redis-cli --cluster create --cluster-replicas 1 \ POD-IP-1:6379 POD-IP-2:6379 POD-IP-3:6379 \ POD-IP-4:6379 POD-IP-5:6379 POD-IP-6:6379
Verify the Redis Cluster is running:
kubectl exec -it redis-0 -- redis-cli cluster info
You should see output similar to the following example:
cluster_state:ok # ...other output...
Log in to a Redis node to check its role. For example, to verify that that
redis-0
has a leader role, run the following command:kubectl exec -it redis-0 -- redis-cli role
You should see output similar to the following example:
1) "master" 2) (integer) 574 3) 1) 1) "10.28.2.3" 2) "6379" 3) "574"
Creating a Redis client application
In this section, you create an application named hello-app-redis
that uses
Redis as a cache database, counts the number of requests it receives, and prints
out the number on a website. If the Redis Service works, the number keeps
increasing.
You can download the image directly from gcr.io/google-samples/hello-app-redis:1.0 in the Google Cloud Console.
To pull the image, run the following command:
docker pull gcr.io/google-samples/hello-app-redis:1.0
To learn more about building images, see Build the container image.
Deploying the Redis client application to GKE
To deploy your application to the GKE cluster you created, you need a Deployment to define your application.
To create the Deployment, complete the following steps:
The file named
app-deployment.yaml
contains the details for the application.To learn more about the probes and Pod affinity rules used in this Deployment, see GKE best practices: Designing and building highly available clusters.
Apply the Deployment by running the following command:
kubectl apply -f app-deployment.yaml
Make sure you run this command in the same directory
app-deployment.yaml
is in.Expose the application through a load balancer:
kubectl expose deployment hello-web \ --type=LoadBalancer \ --port 80 \ --target-port 8080
Wait approximately one minute and retrieve the application's external IP address by running the following command:
kubectl get service
From the output, copy the value listed in
hello-web's
EXTERNAL-IP
column:NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-web LoadBalancer 10.13.10.55 EXTERNAL_IP 80:30703/TCP 166m
Verify the application is working by pasting the EXTERNAL_IP into your web browser. You should see output similar to the following example:
I have been hit [1] times since deployment!
Take note of the visit number. You need to use it in the Testing the application's disruption section.
Set a variable for the EXTERNAL_IP you just copied. You use this value when you create scripts to test your application in the next section:
export IP=EXTERNAL_IP
Upgrading the GKE cluster and testing workload disruption
In the following sections, you upgrade your GKE cluster and observe what happens by using scripts you create.
Testing your application
In this section you use two scripts, one that sends requests to your application, and one that measures the success rate of the requests. You use these scripts to measure what happens when you upgrade your cluster.
To create the scripts:
Change to the directory containing the scripts:
cd cd kubernetes-engine-samples/hello-app-redis/scripts
The script named
generate_load.sh
sends a queries per second (QPS) request to your application. This script saves the HTTP response code into the current directory to a file namedoutput
. The value ofoutput
is used in the script you create in the next step.A second script named
print_error_rate.sh
calculates the success rate based on the output generated bygenerate_load.sh
.Give yourself permission to run the scripts:
chmod u+x generate_load.sh print_error_rate.sh
Set a variable for the number of QPS. This value is used in the
generate_load.sh
script as is the variable you set for the EXTERNAL_IP. We recommend you set a value of 40.export QPS=40
Run the
generate_load.sh
script to start sending QPS:./generate_load.sh $IP $QPS 2>&1
Leave the
generate_load.sh
script running and open a new terminal. In the new terminal, run theprint_error_rate.sh
script to check the error rate:watch ./print_error_rate.sh
You should see a 100% success rate and 0% error rates as the QPS are made.
Leave both scripts running and open a third terminal in preparation for the next section.
Upgrading your cluster
In this section, you upgrade your workloads:
In the terminal you just opened, define the settings for the upgrade using surge upgrade:
gcloud container node-pools update default-pool \ --max-surge-upgrade=1 \ --max-unavailable-upgrade=0 \ --cluster=redis-test
With this configuration (
maxSurge=1
andmaxUnavailable=0
), only one surge node can be added to the node pool during an upgrade, so only one node can be upgraded at a time. This setting speeds up Pod restarts during upgrades while progressing conservatively.Determine which GKE version the
redis-test
cluster is using:V=$(gcloud container clusters describe redis-test | grep "version:" | sed "s/version: //") echo $V
You should see output similar to the following example:
1.15.12-gke.20
.Retrieve a list of available Kubernetes versions:
gcloud container get-server-config
In the list of versions, locate the
validMasterVersions:
section and look for theredis-cluster
version you retrieved in the previous step. To avoid version skew, copy the version from the list that is immediately above theredis-cluster
version.Upgrade the cluster's control plane to the version you selected and type
y
when prompted:gcloud container clusters upgrade redis-test \ --master \ --cluster-version VERSION
Replace VERSION with the version you selected from the list in the previous step.
The control plane upgrade takes several minutes.
Upgrade the cluster's nodes to the version you selected and type
y
when prompted:gcloud container clusters upgrade redis-test \ --cluster-version=VERSION \ --node-pool=default-pool
Replace VERSION with the version you selected from the list.
Testing workload disruption
In this section, you test how your application's status and workload disruption.
Return to the terminal window running
./print_error_rate.sh
and observe how the success rate changed during the upgrade. You should notice a slight decrease in the success rate and a slight increase in the app network error rate as the nodes are taken down to be upgraded.In the
Success rate
field, you'll see how many visits were successfully made to the website. Take a note of this value.Stop both scripts from running by entering
CTRL+C
in the relevant terminals.Return to the website for your application by entering its IP address (this is the EXTERNAL_IP you copied during the Deploying to GKE section) into your browser.
Observe the visit number for your application. The number you see should equal:
ORIGINAL_VISIT_NUMBER + SUCCESSFUL_VISIT_NUMBER
where ORIGINAL_VISIT_NUMBER is the number you recorded in the final step of Deploying to GKE and SUCCESSFUL_VISIT_NUMBER is the value you recorded in the first step of this section.
Cleaning up
After you've finished the Upgrading a stateful workload tutorial, you can clean up the resources that you created on Google Cloud so they won't take up quota and you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.
Deleting the project
The easiest way to eliminate billing is to delete the project that you created for the tutorial.
To delete the project:
- In the Cloud Console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Deleting clusters
To delete the cluster you created for this tutorial, run the following command:
gcloud container clusters delete redis-test
What's next
- Try out other Google Cloud features for yourself. Have a look at our tutorials.