Continuous Delivery Pipelines with Spinnaker and Container Engine

This tutorial shows you how to create a continuous delivery pipeline using Google Container Engine, Google Cloud Source Repositories, Google Cloud Container Builder, and Spinnaker. After you create a sample application, you configure these services to automatically build, test, and deploy it. When you modify the application code, the changes trigger the continuous delivery pipeline to automatically rebuild, retest, and redeploy the new version.

Pipeline architecture

pipeline architecture

To continuously deliver application updates to your users, you need an automated process that reliably builds, tests, and updates your software. Code changes should automatically flow through a pipeline that includes artifact creation, unit testing, functional testing, and production rollout. In some cases, you want a code update to apply to only a subset of your users, so that it is exercised realistically before you push it to your entire user base. If one of these canary releases proves unsatisfactory, your automated procedure must be able to quickly roll back the software changes.

With Container Engine and Spinnaker you can create a robust continuous delivery flow that helps to ensure your software is shipped as quickly as it is developed and validated. Although rapid iteration is your end goal, you must first ensure that each application revision passes through a gamut of automated validations before becoming a candidate for production rollout. When a given change has been vetted through automation, you can also validate the application manually and conduct further pre-release testing.

After your team decides the application is ready for production, one of your team members can approve it for production deployment.

Application delivery pipeline

In this tutorial, you build the continuous delivery pipeline shown in the following diagram.

application delivery pipeline

Objectives

  • Set up your environment by launching Google Cloud Shell, creating a Container Engine cluster, and configuring your identity and user management scheme.
  • Download a sample application, create a Git repository then upload it to a Google Cloud Source Repository.
  • Deploy Spinnaker to Container Engine using Helm.
  • Build your Docker image.
  • Create triggers to create Docker images when your application changes.
  • Configure a Spinnaker pipeline to reliably and continuously deploy your application to Container Engine.
  • Deploy a code change, triggering the pipeline, and watch it roll out to production.

Costs

This tutorial uses billable components of Cloud Platform, including:

  • Container Engine
  • Google Cloud Load Balancing
  • Google Cloud Container Builder

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Cloud Platform users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Manage resources page

  3. Enable billing for your project.

    Enable billing

  4. Enable the Container Engine and Resource Manager APIs.

    Enable the APIs

Set up your environment

In this section you configure the infrastructure and identities required to complete the tutorial.

Start a Cloud Shell instance and create a Container Engine cluster

  1. You'll run all the terminal commands in this tutorial from Cloud Shell.

    OPEN CLOUD SHELL

  2. Create a Container Engine cluster to deploy Spinnaker and the sample application with the following commands:

    gcloud config set compute/zone us-central1-f
    
    gcloud container clusters create spinnaker-tutorial \
        --machine-type=n1-standard-2
    

Configure identity and access management

You create a Cloud Identity Access Management (Cloud IAM) service account to delegate permissions to Spinnaker, allowing it to store data in Cloud Storage. Spinnaker stores its pipeline data in Cloud Storage to ensure reliability and resiliency. If your Spinnaker deployment unexpectedly fails, you can create an identical deployment in minutes with access to the same pipeline data as the original.

  1. Create the service account:

    gcloud iam service-accounts create  spinnaker-storage-account \
        --display-name spinnaker-storage-account
    

  2. Store the service account email address and your current project ID in environment variables for use in later commands:

    export SA_EMAIL=$(gcloud iam service-accounts list \
        --filter="displayName:spinnaker-storage-account" \
        --format='value(email)')
    export PROJECT=$(gcloud info --format='value(config.project)')
    

  3. Bind the storage.admin role to your service account:

    gcloud projects add-iam-policy-binding \
        $PROJECT --role roles/storage.admin --member serviceAccount:$SA_EMAIL
    

  4. Download the service account key. In a later step, you will install Spinnaker and upload this key to Container Engine.

    gcloud iam service-accounts keys create spinnaker-sa.json --iam-account $SA_EMAIL
    

Deploying Spinnaker using Helm

In this section you use Helm to deploy Spinnaker from the Charts repository. Helm is a package manager you can use to configure and deploy Kubernetes applications.

Install Helm

  1. Download and install the helm binary:

    wget https://storage.googleapis.com/kubernetes-helm/helm-v2.5.0-linux-amd64.tar.gz
    

  2. Unzip the file to your local system:

    tar zxfv helm-v2.5.0-linux-amd64.tar.gz
    
    cp linux-amd64/helm .
    

  3. Initialize Helm to install Tiller, the server side of Helm, in your cluster:

    ./helm init
    ./helm update

  4. Ensure that Helm is properly installed by running the following command. If Helm is correctly installed, v2.5.0 appears for both client and server.

    ./helm version

    Client: &version.Version{SemVer:"v2.5.0",
    GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6",
    GitTreeState:"clean"}Server: &version.Version{SemVer:"v2.5.0",
    GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6",
    GitTreeState:"clean"}
    

Configure Spinnaker

  1. Create a bucket for Spinnaker to store its pipeline configuration:

    export PROJECT=$(gcloud info \
        --format='value(config.project)')
    export BUCKET=$PROJECT-spinnaker-config
    gsutil mb -c regional -l us-central1 gs://$BUCKET

  2. Create your configuration file by executing the following commands in Cloud Shell:

    export SA_JSON=$(cat spinnaker-sa.json)
    export PROJECT=$(gcloud info --format='value(config.project)')
    export BUCKET=$PROJECT-spinnaker-config
    cat > spinnaker-config.yaml <<EOF
    storageBucket: $BUCKET
    gcs:
      enabled: true
      project: $PROJECT
      jsonKey: '$SA_JSON'

    # Disable minio the default minio: enabled: false

    # Configure your Docker registries here accounts: - name: gcr address: https://gcr.io username: _json_key password: '$SA_JSON' email: 1234@5678.com EOF

Deploy the Spinnaker chart

  1. Use the Helm command-line interface to deploy the chart with your configuration set. This command typically takes five to ten minutes to complete.

    ./helm install -n cd stable/spinnaker -f spinnaker-config.yaml --timeout 600 \
        --version 0.3.1

  2. After the command completes, run the following command to set up port forwarding to the Spinnaker UI from Cloud Shell:

    export DECK_POD=$(kubectl get pods --namespace default -l "component=deck" \
        -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward --namespace default $DECK_POD 8080:9000 >> /dev/null &
    

  3. To open the Spinnaker user interface, click Web Preview in Cloud Shell and click Preview on port 8080.

    port8080

  4. You should see the welcome screen, followed by the Spinnaker UI:

    hello

    spinui

Building the Docker image

In this section you configure Container Builder to detect changes to your application source code, build a Docker image, and then push it to Container Registry.

Create your source code repository

  1. Download the source code:

    wget https://gke-spinnaker.storage.googleapis.com/sample-app.tgz
    

  2. Unpack the source code:

    tar xzfv sample-app.tgz
    

  3. Change directories to source code:

    cd sample-app
    

  4. Set the username and email address for your Git commits in this repository. Replace [EMAIL_ADDRESS] with your Git email address, and replace [USERNAME] with your Git username.

    git config --global user.email "[EMAIL_ADDRESS]"
    git config --global user.name "[USERNAME]"
    

  5. Make the initial commit to your source code repository:

    git init
    git add .
    git commit -m "Initial commit"
    

  6. Create a repository to host your code:

    gcloud source repos create sample-app
    git config credential.helper gcloud.sh
    

  7. Add your newly created repository as remote:

    export PROJECT=$(gcloud info --format='value(config.project)')
    git remote add origin https://source.developers.google.com/p/$PROJECT/r/sample-app
    

  8. Push your code to the new repository's master branch:

    git push origin master

  9. Check that you can see your source code in the console:

    https://console.cloud.google.com/code/develop/browse/sample-app/master

Configure your build triggers

In this section, you'll configure Container Builder to build and push your Docker images every time you push Git tags to your source repository. Container Builder automatically checks out your source code, builds the Docker image from the Dockerfile in your repository, and pushes that image to Google Cloud Container Registry.

image

  1. In the Cloud Platform Console, click Build Triggers in the Container Registry section.

    GO TO THE BUILD TRIGGERS PAGE

  2. Select Cloud Source Repository and click Continue.

  3. Select your newly created sample-app repository from the list, and click Continue.
  4. Set the following trigger settings:

    • Name:sample-app-tags
    • Trigger type: Tag
    • Tag (regex): v.*
    • Build configuration: cloudbuild.yaml
    • cloudbuild.yaml location: /cloudbuild.yaml

  5. Click Create trigger.

    image

From now on, whenever you push a Git tag prefixed with the letter "v" to your source code repository, Container Builder automatically builds and pushes your application as a Docker image to Container Registry.

Build your image

Push your first image using the following steps:

  1. Go to your source code folder in Cloud Shell.
  2. Create a Git tag:

    git tag v1.0.0

  3. Push the tag:

    git push --tags

  4. In Container Registry, click Build History to check that the build has been triggered. If not, verify the trigger was configured properly in the previous section.

    GO TO BUILD HISTORY

    image

Configuring your deployment pipelines

Now that your images are building automatically, you need to deploy them to the Kubernetes cluster.

You deploy to a scaled-down environment for integration testing. After the integration tests pass, you must manually approve the changes to deploy the code to production services.

configure deployment pipeline

Create the application

  1. In the Spinnaker UI, click Actions, then click Create Application.

    image

  2. In the New Application dialog, enter the following fields:

    • Name: sample
    • Owner Email: [your email address]

  3. Click Create.

    image

Create service load balancers

To avoid having to enter the information manually in the UI, use the Kubernetes command-line interface to create load balancers for your services. Alternatively, you can perform this operation in the Spinnaker UI.

In Cloud Shell, run the following command from the sample-app root directory:

kubectl apply -f k8s/services

Create the deployment pipeline

Next, you create the continuous delivery pipeline. In this tutorial, the pipeline is configured to detect when a Docker image with a tag prefixed with "v" has arrived in your Container Registry.

  1. In a new tab of Cloud Shell, run the following command in the source code directory to upload an example pipeline to your Spinnaker instance:

    export PROJECT=$(gcloud info --format='value(config.project)')
    sed s/PROJECT/$PROJECT/g spinnaker/pipeline-deploy.json | curl -d@- -X \
        POST --header "Content-Type: application/json" --header \
        "Accept: /" http://localhost:8080/gate/pipelines
    

  2. In the Spinnaker UI, click Pipelines on the top navigation bar.

    image

  3. Click Configure in the Deploy pipeline.

    image

    The continuous delivery pipeline configuration appears in the UI:

    image

Run your pipeline manually

The configuration you just created contains a trigger to start the pipeline when you push a new Git tag containing the prefix "v". In this section of the tutorial, you test the pipeline by running it manually. In the next section, you test it by pushing a Git tag and watching the pipeline run automatically.

  1. Return to the Pipelines page by clicking Pipelines.
  2. Click Start Manual Execution.

    manual start

  3. Select the v1.0.0 tag from the Tag drop-down list, then click Run.

    execution

  4. After the pipeline starts, click Details to see more information about the build's progress. This section shows the status of the deployment pipeline and its steps. Steps in blue are currently running, green ones have completed successfully, and red ones have failed. Click a stage to see details about it.

    After 3 to 5 minutes the integration test phase completes and the pipeline requires manual approval to continue the deployment.

  5. Hover over the yellow "person" icon and click Continue.

    push to production

    Your rollout continues to the production frontend and backend deployments. It completes after a few minutes.

  6. To view the app, click Load Balancers in the top right of the Spinnaker UI.

    load balancers

  7. Scroll down the list of load balancers and click Default, under sample-frontend-prod.

    image

  8. Scroll down the details pane on the right and copy your application's IP address by clicking the clipboard button on the Ingress IP.

    image

  9. Paste the address into your browser to view the production version of the application.

    backend

    You have now manually triggered the pipeline to build, test, and deploy your application.

Triggering your pipeline from code changes

In this section, you test the pipeline end to end by making a code change, pushing a Git tag, and watching the pipeline run in response. By pushing a Git tag that starts with "v", you trigger Container Builder to build a new Docker image and push it to Container Registry. Spinnaker detects that the new image tag begins with "v" and triggers a pipeline to deploy the image to canaries, run tests, and roll out the same image to all pods in the deployment.

  1. Change the color of the app from orange to blue.

    sed -i 's/orange/blue/g' cmd/gke-info/common-service.go
    

  2. Tag your change and push it to the source code repository.

    git commit -a -m "Change color to blue"
    git tag v1.0.1
    git push --tags
    

  3. See the new build appear in the Container Builder Build History.

  4. Click Pipelines to watch the pipeline start to deploy the image.

  5. Observe the canary deployments. When the deployment is paused, waiting to roll out to production, start refreshing the tab that contains your application. Nine of your backends are running the previous version of your application, while only one backend is running the canary. You should see the new, blue version of your application appear about every tenth time you refresh.

  6. After testing completes, return to the Spinnaker tab and approve the deployment.

  7. When the pipeline completes, your application looks like the following screenshot. Note that the color has changed to blue because of your code change, and that the Version field now reads v1.0.1.

    image

    You have now successfully rolled out your application to your entire production environment!

  8. Optionally, you can roll back this change by reverting your previous commit. Rolling back adds a new tag (v1.0.2), and pushes the tag back through the same pipeline you used to deploy v1.0.1:

    git revert v1.0.1
    git tag v1.0.2
    git push --tags

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  1. Delete the Spinnaker installation:

    ../helm delete --purge cd
    

  2. Delete the sample app services:

    kubectl delete -f k8s/services
    

  3. Delete the service account:

    export SA_EMAIL=$(gcloud iam service-accounts list \
    --filter="displayName:spinnaker-storage-account" --format='value(email)')
    gcloud iam service-accounts delete $SA_EMAIL
    

  4. Delete the Container Engine cluster:

    gcloud container clusters delete spinnaker-tutorial
    

  5. Delete the repository:

    gcloud source repos delete sample-app
    

  6. Delete the bucket:

    export PROJECT=$(gcloud info --format='value(config.project)')
    export BUCKET=$PROJECT-spinnaker-config
    gsutil -m rm -r gs://$BUCKET
    

  7. Delete your container images:

    export PROJECT=$(gcloud info --format='value(config.project)')
    gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.0
    gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.1
    

  8. If you created v1.0.2 in the optional rollback step above, delete that container image:

    gcloud container images delete gcr.io/$PROJECT/sample-app:v1.0.2
    

What's next

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...