Continuous delivery pipelines with Spinnaker and GKE

This tutorial shows you how to create a continuous delivery pipeline using Google Kubernetes Engine (GKE), Cloud Source Repositories, Cloud Build, and Spinnaker for Google Cloud. After you create a sample app, you configure these services to automatically build, test, and deploy it. When you modify the app code, the changes trigger the continuous delivery pipeline to automatically rebuild, retest, and redeploy the new version.

The State of DevOps reports identified capabilities that drive software delivery performance. This tutorial demonstrates the following capabilities:

Pipeline architecture

The following diagram illustrates the architecture of the continuous delivery pipeline.

Architecture of pipeline for developers and users.

To continuously deliver app updates to your users, you need an automated process that reliably builds, tests, and updates your software. Code changes automatically flow through a pipeline that includes artifact creation, unit testing, functional testing, and production rollout. In some cases, you want a code update to apply to only a subset of your users, so that it is tested realistically before you push it to your entire user base. If one of these canary releases proves unsatisfactory, your automated procedure must be able to quickly roll back the software changes.

With GKE and Spinnaker, you can create a robust continuous delivery flow that helps to ensure your software is shipped as quickly as it is developed and validated. Although rapid iteration is your end goal, you must first ensure that each app revision passes through a series of automated validations before becoming a candidate for production rollout. When a given change has been vetted through automation, you can also validate the app manually and conduct further prerelease testing.

After your team decides the app is ready for production, one of your team members can approve it for production deployment.

App delivery pipeline

In this tutorial, you build the continuous delivery pipeline shown in the following diagram.

Architecture of the app delivery pipeline.

The high-level steps of this pipeline are as follows:

  1. A developer changes code and pushes it to a repository.

  2. Cloud Build detects the changes, builds the Docker image, tests the image, and pushes the image to Spinnaker.

  3. Spinnaker detects the image, deploys image to Canary, and tests the Canary deployment. After a manual approval, Spinnaker deploys the image to production.

Objectives

  • Set up your environment by launching Cloud Shell and deploying Spinnaker for Google Cloud.
  • Create a GKE cluster to deploy the sample application to.
  • Download a sample app, create a Git repository, and upload it to a Cloud Source Repositories.
  • Build your Docker image.
  • Create triggers to create Docker images when your app changes.
  • Configure a Spinnaker pipeline to reliably and continuously deploy your app to GKE.
  • Deploy a code change, triggering the pipeline, and watch the change deploy to production.

Costs

This tutorial uses billable components of Google Cloud, including:

  • GKE
  • Cloud Load Balancing
  • Cloud Build
  • Cloud Source Repositories
  • Container Registry

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. Enable the GKE, Cloud Build, and Cloud Source Repositories APIs.

    Enable the APIs

Set up your environment

In this section, you configure the infrastructure required to complete the tutorial.

Deploy Spinnaker for Google Cloud using Cloud Shell

Run all the terminal commands in this tutorial from Cloud Shell.

With Spinnaker for Google Cloud, you can set up and manage Spinnaker in a production-ready configuration, optimized for Google Cloud. Spinnaker for Google Cloud sets up resources (GKE, Memorystore, Cloud Storage buckets and service accounts), integrates Spinnaker with related services such as Cloud Build, and provides a Cloud Shell-based management environment for your Spinnaker installations, with helpers and common tools such as spin and hal.

  1. Open Spinnaker for Google Cloud in Cloud Shell. This clones the Spinnaker for Google Cloud repository into your Cloud Shell environment and launches the detailed installation instructions.

    Go to Cloud Shell

  2. Configure Git. If you've configured Git before, you can skip this step.

    git config --global user.email "EMAIL_ADDRESS"
git config --global user.name "USERNAME"

    Replace the following:

    • EMAIL_ADDRESS: your Git email address
    • USERNAME: your Git username

  3. Set an environment variable to define which Cloud project to install Spinnaker to:

    DEVSHELL_PROJECT_ID=YOUR_PROJECT_ID

    Replace YOUR_PROJECT_ID with the project that you selected or created for this tutorial.

  4. Install Spinnaker for Google Cloud.

    PROJECT_ID=${DEVSHELL_PROJECT_ID} ~/cloudshell_open/spinnaker-for-gcp/scripts/install/setup_properties.sh
~/cloudshell_open/spinnaker-for-gcp/scripts/install/setup.sh

  5. Restart Cloud Shell to load new environment settings.

  6. Connect to Spinnaker.

    ~/cloudshell_open/spinnaker-for-gcp/scripts/manage/connect_unsecured.sh

  7. In Cloud Shell, click the Web Preview icon and select Preview on port 8080.

  8. The welcome screen appears, followed by the Spinnaker UI.

Create a GKE cluster for application deployments

A common pattern is to have a GKE cluster used for builds, deployments, and so on, and then other GKE clusters for running applications. In this section you create another GKE cluster, app-cluster, to deploy the sample application to.

  1. In Cloud Shell, create a new GKE cluster:

    ZONE=us-east1-c
gcloud config set compute/zone $ZONE
gcloud container clusters create app-cluster \
--machine-type=n1-standard-2

  2. Add the new GKE cluster to Spinnaker. The default values should be correct.

    ~/cloudshell_open/spinnaker-for-gcp/scripts/manage/add_gke_account.sh

    Example output and values: 

    Please enter the context you wish to use to manage your GKE resources: gke_spinnaker-246920_us-east1-c_app-cluster
Please enter the id of the project within which the referenced cluster lives: spinnaker-246920
Please enter a name for the new Spinnaker account: app-cluster-acct

  3. Change the kubernetes context back to your Spinnaker cluster:

    kubectl config use-context gke_${DEVSHELL_PROJECT_ID}_${ZONE}_spinnaker-1

  4. Push and apply the configuration changes to Spinnaker:

    ~/cloudshell_open/spinnaker-for-gcp/scripts/manage/push_and_apply.sh

Building the Docker image

In this section, you configure Cloud Build to detect changes to your app source code, build a Docker image, and then push it to Container Registry.

Create your source code repository

  1. In Cloud Shell, download the sample source code:

    cd ~
wget https://gke-spinnaker.storage.googleapis.com/sample-app-v4.tgz

  2. Unpack the source code:

    tar xzfv sample-app-v4.tgz

  3. Change directories to the source code:

    cd sample-app

  4. Make the initial commit to your source code repository:

    git init
git add .
git commit -m "Initial commit"

  5. Create a repository to host your code:

    gcloud source repos create sample-app
git config credential.helper gcloud.sh

  6. Add your newly created repository as remote:

    export PROJECT=$(gcloud info --format='value(config.project)')
git remote add origin https://source.developers.google.com/p/$PROJECT/r/sample-app

  7. Push your code to the new repository's master branch:

    git push origin master

  8. Check that you can see your source code in the console:

    Go to the Source Code

Configure your build triggers

The following diagram illustrates the trigger that you build in this section.

Spinnaker workflow

You configure Cloud Build to build and push your Docker images every time you push Git tags to your source repository. Cloud Build automatically checks out your source code, builds the Docker image from the Dockerfile in your repository, and pushes that image to Container Registry.

  1. In the Cloud Console, in the Cloud Build section, click Triggers, and then click Create Trigger.

    Go to Build Triggers

  2. Set the following trigger settings:

    • Name: sample-app-tags
    • Event: Select Push new tag
    • Repository: sample-app
    • Tag (regex): v.*
    • Configuration: Cloud Build configuration file (yaml or json)
    • Cloud Build configuration file location: /cloudbuild.yaml

  3. Click Create.

From now on, whenever you push a Git tag prefixed with the letter "v" to your source code repository, Cloud Build automatically builds and pushes your app as a Docker image to Container Registry.

Prepare your Kubernetes manifests for use in Spinnaker

Spinnaker needs access to your Kubernetes manifests in order to deploy them to your clusters. This section creates a Cloud Storage bucket that is populated with your manifests during the CI process in Cloud Build. After your manifests are in Cloud Storage, Spinnaker can download and apply them during your pipeline's execution.

  1. Create the bucket.

    export PROJECT=$(gcloud info --format='value(config.project)')
gsutil mb gs://$PROJECT-kubernetes-manifests

  2. Enable versioning on the bucket so that you have a history of your manifests.

    gsutil versioning set on gs://$PROJECT-kubernetes-manifests

  3. Set the correct Google Cloud project ID in your Kubernetes deployment manifests:

    sed -i s/PROJECT/$PROJECT/g k8s/deployments/*

  4. Commit the changes to the repository:

    git commit -a -m "Set project ID"

Configuring your deployment pipelines

Now that your images are building automatically, you need to deploy them to the Kubernetes cluster.

The following diagram illustrates the deployment pipeline steps.

Configure deployment pipeline.

You deploy to a scaled-down environment for integration testing. After the integration tests pass, you must manually approve the changes to deploy the code to production services.

Create the deployment pipeline

  1. Use spin to create an app in Spinnaker.

    spin application save --application-name sample \
                      --owner-email example@example.com \
                      --cloud-providers kubernetes \
                      --gate-endpoint http://localhost:8080/gate

    Next, you create the continuous delivery pipeline. In this tutorial, the pipeline is configured to detect when a Docker image with a tag prefixed with v has arrived in your Container Registry.

  2. In a new tab of Cloud Shell, run the following command in the source code directory to upload an example pipeline to your Spinnaker instance:

    export PROJECT=$(gcloud info --format='value(config.project)')
sed s/PROJECT/$PROJECT/g spinnaker/pipeline-deploy.json > pipeline.json
spin pipeline save --gate-endpoint http://localhost:8080/gate -f pipeline.json

Build your image

Push your first image using the following steps:

  1. Go to your source code folder in Cloud Shell.

  2. Create a Git tag:

    git tag v1.0.0

  3. Push the tag:

    git push --tags

  4. In Cloud Build, click Build History to check that the build has been triggered. If not, verify the trigger was configured properly in the previous section.

    Go to Build History

    build history

View your pipeline execution

The configuration you created uses notifications of newly tagged images being pushed to trigger a Spinnaker pipeline. In a previous step, you pushed a tag to the Cloud Source Repositories which triggered Cloud Build to build and push your image to Container Registry. You can now check on the pipeline that was triggered.

  1. Return to the Pipelines page by clicking Pipelines.

  2. Click Details to see more information about the pipeline's progress. This section shows the status of the deployment pipeline and its steps. Steps in blue are currently running, green ones have completed successfully, and red ones have failed. Click a stage to see details about it.

    After 3 to 5 minutes the integration test phase completes and the pipeline requires manual approval to continue the deployment.

  3. Hold the pointer over Push to production, and then click Continue.

    Your rollout continues to the production frontend and backend deployments. It completes after a few minutes.

  4. To view the app, select Infrastructure > Load Balancers in the top of the Spinnaker UI.

    Infrastructure drop-down list with Load Balancers selected.

  5. Scroll down the list of load balancers and click Default, under sample-frontend-production.

    Default load balancer.

  6. Scroll down the details pane on the right and copy your app's IP address by clicking the clipboard button on the Ingress IP. The ingress IP link from the Spinnaker UI uses HTTPS by default, but the application is configured to use HTTP.

    Details pane with your app's IP address.

  7. Paste the address into your browser to view the production version of the app.

    Production version of the app.

    You have now manually triggered the pipeline to build, test, and deploy your app.

Triggering your pipeline from code changes

In this section, you test the pipeline end to end by making a code change, pushing a Git tag, and watching the pipeline run in response. By pushing a Git tag that starts with "v", you trigger Cloud Build to build a new Docker image and push it to Container Registry. Spinnaker detects that the new image tag begins with "v" and triggers a pipeline to deploy the image to canaries, run tests, and roll out the same image to all pods in the deployment.

  1. Change the color of the app from orange to blue:

    cd ~/sample-app
sed -i 's/orange/blue/g' cmd/gke-info/common-service.go

  2. Tag your change and push it to the source code repository:

    git commit -a -m "Change color to blue"
git tag v1.0.1
git push --tags

  3. See the new build appear in the Cloud Build Build History.

  4. Click Pipelines to watch the pipeline start to deploy the image.

  5. Observe the canary deployments. When the deployment is paused, waiting to roll out to production, start refreshing the tab that contains your app. Four of your backends are running the previous version of your app, while only one backend is running the canary. The new, blue version of your app appears about every tenth time you refresh.

  6. After testing completes, return to the Spinnaker tab and approve the deployment.

  7. When the pipeline completes, your app looks like the following screenshot. The Version field now reads v1.0.1.

    Production version of the updated app.

    You have now successfully rolled out your app to your entire production environment!

  8. Optionally, you can roll back this change by reverting your previous commit. Rolling back adds a new tag (v1.0.2), and pushes the tag back through the same pipeline you used to deploy v1.0.1:

    git revert v1.0.1
git tag v1.0.2
git push --tags

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

  1. Delete Spinnaker for Google Cloud:

    cd ~
~/cloudshell_open/spinnaker-for-gcp/scripts/manage/generate_deletion_script.sh
~/cloudshell_open/spinnaker-for-gcp/scripts/manage/delete-all_${DEVSHELL_PROJECT_ID}_spinnaker-1_spinnaker-1.sh

  2. Delete the GKE cluster:

    gcloud container clusters delete app-cluster --zone=us-east1-c

  3. Delete the repository:

    gcloud source repos delete sample-app

  4. Delete the bucket:

    export PROJECT=$(gcloud info --format='value(config.project)')
export BUCKET=$PROJECT-kubernetes-manifests
gsutil -m rm -r gs://$BUCKET

  5. Delete your container images:

    export PROJECT=$(gcloud info --format='value(config.project)')
gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.0
gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.1

  6. If you created v1.0.2 in the optional rollback step above, delete that container image:

    gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.2

What's next