Getting new software releases to your users without interrupting their experience is difficult. How do you upgrade an application that's in active use? If you take an application offline for an update, what happens if something goes wrong? Application downtime is expensive, so most enterprises strive to minimize or eliminate it.
This tutorial uses Google Kubernetes Engine (GKE) to walk through several software deployment strategies (recreate, rolling update, and blue/green) and testing strategies (canary, shadow, and A/B). For more information about these strategies, see Application deployment and testing strategies.
Testing patterns such as canary, shadow, and A/B testing require that you split traffic between multiple services that are deployed on the GKE cluster. To split traffic, you use Istio. Istio creates a network of deployed services such as load balancing, traffic routing, service-to-service authentication, and monitoring. No changes are required in the application code in order to run Istio.
This tutorial is intended for system administrators and DevOps engineers who define and implement release and deployment strategies for various applications, systems, and frameworks. This tutorial assumes the reader understands core Kubernetes concepts.
Example workload
This tutorial uses a
Spring Boot application
that is deployed on GKE. This application is packaged as
container images that represent the current (app:current
) and new (app:new
)
versions of the application. In the following sections, the Kubernetes
deployments use these container images.
The application version information is exposed through the /<version>
endpoint. In the following sections, you use these endpoints to monitor the
version of the deployed application:
curl http://ip-address-current/version {"id":1,"content":"current"} curl http://ip-address-new/version {"id":2,"content":"new"}
Replace the following:
ip-address-current
: The IP address for the current application version.ip-address-new
: The IP address for the new application version.
Objectives
Each of the deployment and testing patterns discussed in this tutorial has the following objectives:
- Create a GKE cluster.
- Deploy a sample application.
- Verify that the application is serving traffic.
- Deploy or test a new version of the application by using a specific method.
- Verify that the new version is deployed.
Costs
This tutorial uses the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.
Before you begin
-
In the Google Cloud console, go to the project selector page.
-
Select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
-
In the console, activate Cloud Shell.
At the bottom of the console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
- Export the following environment variables:
export PROJECT=$(gcloud config get-value project) export ZONE="us-central1-a" export CLUSTER="test-cluster"
These environment variables are used to create a cluster calledtest-cluster
in theus-central1
zone. For more information, see Geography and regions. - Enable the APIs for Compute Engine, Google Kubernetes Engine,
Container Analysis, and Container Registry:
gcloud services enable \ compute.googleapis.com \ container.googleapis.com \ containeranalysis.googleapis.com \ containerregistry.googleapis.com
Create a GKE cluster
Create a GKE cluster with Istio enabled:
gcloud beta container clusters create $CLUSTER --zone $ZONE \ --addons Istio \ --istio-config auth=MTLS_PERMISSIVE
Clone the GitHub repository
Clone the code repository:
git clone \ https://github.com/GoogleCloudPlatform/gke-deployment-testing-strategies/
Change to the working directory:
cd gke-deployment-testing-strategies
Build your container images
In this section, you build the container images for the current and new versions of the application.
The GitHub repository that accompanies this tutorial includes the source used to build the container images.
The container images that you build in this section are exposed publicly under the
Container Registry gcr.io/$PROJECT
.
To deploy the
app:current
image, run the following commands, which change to the Dockerfile folder, build the docker image, and upload it:cd ~/gke-deployment-testing-strategies/app/current gcloud builds submit --tag gcr.io/$PROJECT/app:current
To deploy the
app:new
image, run the following commands, which change to the Dockerfile folder, build the docker image, and upload it:cd ~/gke-deployment-testing-strategies/app/new gcloud builds submit --tag gcr.io/$PROJECT/app:new
Update the reference in all deployment YAML files to the newly created docker image:
cd ~/gke-deployment-testing-strategies find . -type f -name "*.yaml" -exec sed -i -e "s/{{YOUR_PROJECT_ID}}/$PROJECT_ID/g" {} +
Test your container images
To verify that the container images work as intended, follow these steps:
Pull the container image
app:current
from the Container Registry and run the container:docker run --name curr -d \ -p 9001:8080 gcr.io/$PROJECT/app:current && \ while ! curl -s http://localhost:9001/version; \ do sleep 5; done
The output is similar to the following:
[...] {"id":1,"content":"current"}
Pull the container image
app:new
from the Container Registry and run the container:docker run --name new -d \ -p 9002:8080 gcr.io/$PROJECT/app:new && \ while ! curl -s http://localhost:9002/version; \ do sleep 5; done
The output is similar to the following:
[...] {"id":2,"content":"new"}
Stop the running containers:
docker rm -f curr && docker rm -f new
Deploy the new application version
In this section, you use different deployment patterns (recreate, rolling
update, and blue/green) to deploy a new application version (app:new
) that
replaces the current version (app:current
). After each deployment method, you
test the deployment before deploying the new version. You then delete the
resources that you used in that deployment and try a new deployment method.
Perform a recreate deployment
With a recreate deployment, you terminate the current version of your application and then roll out the new version, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Test the deployed application by generating traffic against it.
- Deploy a new version of the application (
app:new
). - Verify that the current version is terminated and that the traffic
switched from
app:current
toapp:new
.
Deploy the current version
Deploy the current application version:
kubectl apply -f recreate/deployment-old.yaml
A Kubernetes deployment named
app
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app
The output is similar to the following:
[...] deployment "app" successfully rolled out
Create a Kubernetes service that serves external traffic:
kubectl apply -f recreate/service.yaml
Verify that the service is created successfully:
kubectl get svc/app -w
Before proceeding, check that an external IP address is allocated, as indicated by output similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE app LoadBalancer 10.11.254.22 <pending> 8080:30343/TCP 36s app LoadBalancer 10.11.254.22 [EXTERNAL_IP] 8080:30343/TCP 37s
To end the watch loop, press
CTRL+C
.
Test the deployment
- In Cloud Shell, click Open a new tab add to start a new Cloud Shell session.
Get the load balancer IP address:
SERVICE_IP=$(kubectl get svc app \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Generate traffic against the deployed application:
while(true); do \ curl "http://${SERVICE_IP}:8080/version"; echo; sleep 2; done
After a few seconds, the output is similar to the following:
{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Keep the
curl
command running. You revisit this session after you deploy the new version.
Deploy the new version
In your original Cloud Shell session, deploy the new application version:
kubectl apply -f recreate/deployment-new.yaml
In the terminal session where you ran the
curl
command, monitor the output to verify that the traffic switched.Old replicas are terminated, and then new replicas with the updated version are created. Downtime occurs during the update. After the update finishes, the new version serves requests.
The output indicates the
Initial
,Intermediate
, andFinal
states of the update:{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} curl: (7) Failed to connect to [GKE_Service_IP] port 8080: Connection refused {"id":2,"content":"new"} {"id":2,"content":"new"} {"id":2,"content":"new"}
To stop the
curl
command, pressCTRL+C
.
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f recreate/ --ignore-not-found
Close the other Cloud Shell session.
Perform a rolling update deployment
With a rolling update deployment, you replace the current version of your application by gradually rolling out the new version, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Test the deployed application by generating traffic against it.
- Deploy a new version of the application (
app:new
). - Verify that the traffic switched from
app:current
toapp:new
.
Deploy the current version
Deploy the current application version:
kubectl apply -f rollingupdate/deployment-old.yaml
A Kubernetes deployment named
app
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app
The output is similar to the following:
[...] deployment "app" successfully rolled out
Create a Kubernetes service to serve external traffic:
kubectl apply -f rollingupdate/service.yaml
Verify that the service is created successfully:
kubectl get svc/app -w
Before proceeding, wait for an external IP address to be allocated, as indicated by output similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE app LoadBalancer 10.11.254.22 <pending> 8080:30343/TCP 36s app LoadBalancer 10.11.254.22 [EXTERNAL_IP] 8080:30343/TCP 37s
To end the watch loop, press
CTRL+C
.
Test the deployment
- In Cloud Shell, click Open a new tab add to start a new Cloud Shell session.
Get the load balancer IP address:
SERVICE_IP=$(kubectl get svc app \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Generate traffic against the deployed application:
while(true); do \ curl "http://${SERVICE_IP}:8080/version"; echo; sleep 2; done
After a few seconds, the output is similar to the following:
{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Keep the
curl
command running. You revisit this session after you deploy the new version.
Deploy the new version
In your original Cloud Shell session, deploy the new application version:
kubectl apply -f rollingupdate/deployment-new.yaml
In the terminal session where you ran the
curl
command, monitor the responses to verify that the traffic switched.The rolling update configuration ensures that at least two Pods serve the client traffic at any time (calculated as
number of replicas - maxUnavailable
), and at most one Pod is replaced at a time (defined bymaxSurge
).Liveness and readiness probes are used to verify that the new Pods are ready to serve traffic. During the update, the active Pods serve traffic requests. After all replicas are updated, the new application version serves all requests.
The output indicates the
Initial
,Intermediate
, andFinal
states of the update:{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":2,"content":"new"} {"id":2,"content":"new"} {"id":2,"content":"new"} {"id":2,"content":"new"}
To stop the
curl
command, pressCTRL+C
.
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f rollingupdate/ --ignore-not-found
Close the other Cloud Shell session.
Perform a blue/green deployment
With a blue/green deployment, you release the new version of your application alongside the current version. After you conduct appropriate tests, you switch the traffic to the new version, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Test the deployed application by generating traffic against it.
- Deploy a new version of the application (
app:new
) alongside the current version. - Switch the traffic from
app:current
toapp:new
at the load balancer layer by updating the service selector.
Deploy the current version (blue deployment)
Deploy the current application version:
kubectl apply -f bluegreen/deployment-old.yaml
A Kubernetes deployment named
app-01
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-01
The output is similar to the following:
[...] deployment "app-01" successfully rolled out
Create a Kubernetes service to serve external traffic:
kubectl apply -f bluegreen/service-old.yaml
Verify that the service is created successfully:
kubectl get svc/app -w
Before proceeding, wait for an external IP address to be allocated, as indicated by output similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE app LoadBalancer 10.11.254.22 <pending> 8080:30343/TCP 36s app LoadBalancer 10.11.254.22 [EXTERNAL_IP] 8080:30343/TCP 37s
To end the watch loop, press
CTRL+C
.
Test the deployment
- In Cloud Shell, click Open a new tab add to start a new Cloud Shell session.
Get the load balancer IP address:
SERVICE_IP=$(kubectl get svc app \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Generate traffic against the deployed application:
while(true); \ do curl "http://${SERVICE_IP}:8080/version"; echo; sleep 2; done
After a few seconds, the output is similar to the following:
{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Keep the
curl
command running. You revisit this session after you deploy the new version.
Deploy the new version (green deployment)
In your original Cloud Shell session, deploy the new application version:
kubectl apply -f bluegreen/deployment-new.yaml
A Kubernetes deployment named
app-02
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-02
The output is similar to the following:
[...] deployment "app-02" successfully rolled out
Switch traffic from the blue deployment to the green deployment
Update the service selector to point to the new version:
kubectl apply -f bluegreen/service-new.yaml
In the terminal session where you ran the
curl
command, monitor the responses to verify that the traffic switched. After the service is updated, the requests are routed to the new version.The output indicates the
Initial
andFinal
states of the update:{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":2,"content":"new"} {"id":2,"content":"new"} {"id":2,"content":"new"}
To stop the
curl
command, pressCTRL+C
.
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f bluegreen/ --ignore-not-found
Close the other Cloud Shell session.
Testing the new application version
In this section, you deploy a new application version (app:new
) that
replaces the current version (app:current
). You then use different testing
patterns (canary, A/B, and shadow) to test the new version.
Perform a canary test
With a canary test, you partially roll out the new version of your application to a subset of users and evaluate its performance against a baseline deployment, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Use Istio resources to serve external traffic.
- Deploy a new version of the application (
app:new
) alongside the current version. - Use Istio to split and route traffic between the two versions based on predefined weights.
Deploy the current version
Deploy the current version of the application:
kubectl apply -f canary/deployment-old.yaml
A Kubernetes deployment named
app-01
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-01
The output is similar to the following:
[...] deployment "app-01" successfully rolled out
Following Istio's recommendations for Pods and services, this deployment uses a version label
-01
to specify the version of the application.Deploy the Istio resources:
kubectl apply \ -f canary/gateway.yaml \ -f canary/virtualservice.yaml
An Istio ingress gateway and a virtual service are created. The ingress gateway describes a load balancer operating at the edge of the service mesh. All incoming HTTP/TCP connections are handled by the gateway and forwarded to a virtual service, which defines a set of routing rules to direct traffic inside the service mesh. For more information, see about how Istio routes ingress traffic.
Test the deployment
- In Cloud Shell, click Open a new tab add to start a new Cloud Shell session.
Get the Istio ingress gateway IP address:
SERVICE_IP=$(kubectl get service istio-ingressgateway \ -n istio-system \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Generate traffic against the deployed application:
while(true); \ do curl "http://${SERVICE_IP}/version"; echo; sleep 2; done
The output is similar to the following:
{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Keep the
curl
command running. You revisit this session after the new version is deployed.
Deploy the new version (canary)
In your original Cloud Shell session, deploy the new application version:
kubectl apply -f canary/deployment-new.yaml
A Kubernetes deployment named
app-02
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-02
The output is similar to the following:
[...] deployment "app-02" successfully rolled out
Split the traffic
A canary deployment consists of gradually shifting production traffic from one
version to another. In an 80-20 split, 80% of the requests go to app:current
,
and 20% go to app:new
.
To split the traffic, perform the following steps:
Update the virtual service that you created earlier:
kubectl apply \ -f canary/destinationrule.yaml \ -f canary/virtualservice-split.yaml
The
destinationrule.yaml
and thevirtualservice-split.yaml
files are configured to enforce an 80-20 traffic split between the application versions. For more information, see how Istio splits traffic between versions.In the terminal session where you ran the
curl
command, monitor the responses.The output is similar to the following:
[...] {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":2,"content":"new"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} {"id":2,"content":"new"} [...]
To stop the
curl
command, pressCTRL+C
.
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f canary/ --ignore-not-found
Close the other Cloud Shell session.
Perform an A/B test
With an A/B test, you release the new version of your application to a subset of users defined by specific conditions (for example, location, browser version, or user agent) and then test a theory or hypothesis, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Deploy a new version of the application (
app:new
) alongside the current version. - Use Istio to route incoming requests that have the username
test
in the request's cookie toapp:new
. All other requests are routed toapp:current
.
Deploy the current version
Deploy the current application version:
kubectl apply -f ab/deployment-old.yaml
A Kubernetes deployment named
app-01
with the label01
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-01
The output is similar to the following:
[...] deployment "app-01" successfully rolled out
Deploy the Istio resources:
kubectl apply -f ab/gateway.yaml -f ab/virtualservice.yaml
Test the deployment
Get the Istio ingress gateway IP address:
SERVICE_IP=$(kubectl get service istio-ingressgateway \ -n istio-system \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Send a request to the application:
curl "http://${SERVICE_IP}/version"
The output is similar to the following:
{"id":1,"content":"current"}
Deploy the new version
Deploy the new version of the application:
kubectl apply -f ab/deployment-new.yaml
A Kubernetes deployment named
app-02
with the version label02
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-02
The output is similar to the following:
[...] deployment "app-02" successfully rolled out
Split the traffic
Split the traffic based on the username received in the request's cookie:
kubectl apply \ -f ab/destinationrule.yaml \ -f ab/virtualservice-split.yaml
All requests where
user
is identified astest
go to the new application version.Send a request to the application in which
user
is identified astest
:curl --cookie "user=test" "http://${SERVICE_IP}/version"
The output is similar to the following:
{"id":2,"content":"new"}
Send a request without the cookie:
curl "http://${SERVICE_IP}/version"
The output is similar to the following:
{"id":1,"content":"current"}
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f ab/ --ignore-not-found
Close the other Cloud Shell session.
Perform a shadow test
With a shadow test, you test the new version of your application by mirroring user traffic from the current application version without impacting the user requests, as the following diagram shows.
To try this pattern, you perform the following steps:
- Deploy the current version of the application (
app:current
) on the GKE cluster. - Deploy a new version of the application (
app:new
) alongside the current version. - Use Istio to
mirror
all incoming requests to
app:new
. - Verify that the traffic is mirrored to
app:new
and does not impact any end-user requests toapp:current
.
Deploy the current version
Deploy the current version of the application:
kubectl apply -f shadow/deployment-old.yaml
A Kubernetes deployment named
app-01
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-01
The output is similar to the following:
[...] deployment "app-01" successfully rolled out
Deploy the Istio resources:
kubectl apply \ -f shadow/gateway.yaml \ -f shadow/virtualservice.yaml
Test the deployment
- In Cloud Shell, click Open a new tab add to start a new Cloud Shell session.
Get the Istio ingress gateway IP address:
SERVICE_IP=$(kubectl get service istio-ingressgateway \ -n istio-system \ -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
Generate traffic against the deployed application:
while(true); \ do curl "http://${SERVICE_IP}/version"; echo; sleep 2; done
The output is similar to the following:
{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Keep the
curl
command running. You revisit this session after you deploy the new version.
Deploy the new version
In your original Cloud Shell session, deploy the new application version:
kubectl apply -f shadow/deployment-new.yaml
A Kubernetes deployment named
app-02
is created.Verify that the deployment is created successfully:
kubectl rollout status deploy app-02
The output is similar to the following:
[...] deployment "app-02" successfully rolled out
Set up traffic mirroring
Set up traffic mirroring by updating the Istio resources:
kubectl apply -f shadow/virtualservice-mirror.yaml
In the terminal where you ran the
curl
command, you see thatapp:current
still serves requests, as the following output indicates:{"id":1,"content":"current"} {"id":1,"content":"current"} {"id":1,"content":"current"} [...]
Check the new deployment logs to ensure that the traffic is mirrored:
kubectl logs -f --tail=3 deployment/app-02
The output is similar to the following:
2019-07-15 11:03:12.927 INFO 1 --- [nio-8080-exec-5] com.google.springboot.SpringBootDemo : Serving request from version 2 2019-07-15 11:03:13.345 INFO 1 --- [nio-8080-exec-7] com.google.springboot.SpringBootDemo : Serving request from version 2 2019-07-15 11:03:13.752 INFO 1 --- [nio-8080-exec-8] com.google.springboot.SpringBootDemo : Serving request from version 2 [...]
The traffic is mirrored asynchronously and out of band from the production traffic. This means that Istio prioritizes the production path. That is, Istio returns responses from the production service without waiting for any responses from the shadow service. Also, only requests are shadowed; responses from the shadowed service are dropped.
To stop the
curl
command, pressCTRL+C
.
Clean up the resources
In your original Cloud Shell session, clean up the deployment and service resources used in this example:
kubectl delete -f shadow/ --ignore-not-found
Close the other Cloud Shell session.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the Google Cloud project
The easiest way to eliminate billing is to delete the project you created for the tutorial.
- In the console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete the resources
If you want to keep the Google Cloud project you used in this tutorial, delete the individual resources:
Delete the GKE cluster:
gcloud container clusters delete $CLUSTER --zone $ZONE --async
Delete downloaded code, artifacts, and other dependencies:
cd .. && rm -rf kubernetes-deployment-patterns
Delete the image in Container Registry:
gcloud container images list-tags gcr.io/$PROJECT/app \ --format 'value(digest)' | \ xargs -I {} gcloud container images delete \ --force-delete-tags --quiet \ gcr.io/${PROJECT}/app@sha256:{}
What's next
- Read Application deployment and testing strategies.
- Read about the concepts of this document in Kubernetes and the challenges of continuous software delivery.
- Learn more about other DevOps solutions on Google Cloud.
- Learn more about best practices for running containers.
- Learn more about continuous integration and delivery on GKE.
- Read more about DevOps tools that can help in cloud migration.
- Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.