Continuous Deployment on Compute Engine Using Ansible with Spinnaker

This tutorial shows you how to set up a continuous delivery pipeline using Ansible, Spinnaker and Google Compute Engine as described in the following diagram. This architecture uses baking and deploying immutable images as an example.

architecture for baking and deploying immutable images

Spinnaker is an open source tool for orchestrating continuous delivery pipelines that deploy software to cloud resources. It is typically used for deployments based on Debian packages, but with Packer templates, you can configure Spinnaker to support your provisioner of choice.

If you already use the Ansible open source configuration management tool, you are familiar with infrastructure as code; with Spinnaker, you pair infrastructure as code with immutable infrastructure. With immutable infrastructure, changes are baked into artifacts like virtual machine (VM) images or container images, rather than being applied incrementally to long- running servers.

Objectives

  • Deploy Spinnaker to Compute Engine using Google Cloud Deployment Manager.
  • Prepare a GitHub repository for use with the Spinnaker pipeline.
  • Create a 3-stage Spinnaker pipeline: test, bake, and deploy.
  • Execute the pipeline 2 ways: manually, and using a GitHub pull request.
  • Roll back changes using a red/black deployment strategy.

Costs

This tutorial uses billable components of Cloud Platform, including:

  • Compute Engine
  • Google Cloud Load Balancing
  • Google Cloud Storage
  • Google Cloud IAM

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Cloud Platform users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Manage resources page

  3. Enable billing for your project.

    Enable billing

  4. Enable the Compute Engine API.

    Enable the API

You download and deploy the remaining components, such as your Spinnaker environment and a sample app, from GitHub.

Before you deploy the stack, you must ensure that the default service account includes the necessary IAM roles to complete the tutorial.

  1. In the Google Cloud Platform Console, go to the Cloud IAM page.

    Go to Cloud IAM

  2. Make sure your Google API service account includes the following Cloud IAM roles:

    • Compute Instance Admin (v1)
    • Compute Network Admin
    • Compute Storage Admin
    • Storage Admin

    For details about how to create a service account and set up Cloud IAM roles, see creating a new service account.

  3. Finally, you'll run all the terminal commands in this tutorial from Cloud Shell.

    OPEN CLOUD SHELL

Deploying Spinnaker

First, you'll deploy Spinnaker to Compute Engine using Deployment Manager.

  1. Launch your Spinnaker environment with Deployment Manager from the prepared GitHub repository:

    git clone https://github.com/GoogleCloudPlatform/spinnaker-deploymentmanager
    cd spinnaker-deploymentmanager

    export GOOGLE_PROJECT=$(gcloud config get-value project)
    export DEPLOYMENT_NAME="${USER}-test1"
    export JENKINS_PASSWORD=$(openssl rand -base64 15)

    gcloud deployment-manager deployments create --config config.jinja ${DEPLOYMENT_NAME} --properties jenkinsPassword:${JENKINS_PASSWORD}

    The deployment might take a few minutes to complete, and then the Spinnaker instance will be ready.

  2. Save the instance URI to a variable so that it can be easily referenced:

    export SPINNAKER_VM=$(gcloud compute instances list --regexp "${DEPLOYMENT_NAME}-spinnaker.+" --uri)

  3. Use SSH to connect to the Spinnaker instance and forward the Spinnaker UI and Jenkins UI ports to your Cloud Shell:

    gcloud compute ssh ${SPINNAKER_VM} -- -L 8081:localhost:8081 -L 8082:localhost:8082

    Spinnaker is configured using a startup-script, which takes up to 10 minutes to complete. When the script is finished and Spinnaker is ready, you will see a message like following in /var/log/syslog:

    $ tail /var/log/syslog
    ...
    INFO startup-script: Return code 0.
    INFO Finished running startup scripts.
    ...
    

  4. Click the Cloud Shell Web Preview button.

  5. Click Change port > Port 8081 to open the Spinnaker UI in your browser.

    Click preview button

    Spinnaker opens in a new tab:

    Spinnaker opens in a new tab

Configuring your application to bake with Ansible

In this tutorial, you deploy a sample application (sample-app).

  1. Fork the repository on GitHub into your own account.
  2. Clone sample-app to your computer:

    git clone git@github.com:YOUR_GH_USER/sample-app.git
    

This repository contains a file named bake.yml, which is an Ansible playbook that is run locally by Packer during the bake step. This playbook is responsible for building the application and installing the necessary services. When a Compute Engine instance is started with the image, the services are automatically started and ready to serve traffic. A file named build.sh is also in the repository. This script is invoked by Jenkins during the test stage of the Spinnaker pipeline, and it should run the tests for your application and return a zero error code if the tests pass.

The Packer template invoked by Spinnaker looks like the following:

    "provisioners": [
        {
        "type": "shell",
        "inline": ["sudo apt-add-repository -y ppa:ansible/ansible",
                "sudo apt-get update",
                "sudo apt-get install -y ansible=2.3.0.0-1ppa~xenial
    git=1:2.7.4-0ubuntu1",
                "sudo git clone [[user 'repository']] /opt/go/src/deploy",
                "cd /opt/go/src/deploy && sudo git checkout [[user
    'repository_hash']]",
                "sudo ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook -i
    'localhost,' bake.yml"]
        }
    ]

This template is added to the Spinnaker VM during deployment and written to /opt/rosco/config/packer/gce-ansible.json. The template is referenced later when you configure the pipeline.

The provisioner in the Packer template does the following:

  • Installs Ansible from the Personal Packages Archive (PPA).
  • Clones the repository and checks out the commit hash passed down from the pipeline.
  • Runs Ansible with the bake.yml playbook file.

In this example, the Ansible playbook does the following:

  • Installs Nginx and configures it to upstream to the application port.
  • Downloads and installs golang.
  • Compiles and installs the application.
  • Installs systemd services to run the application on startup.

You should tailor the Ansible playbook for your application so that when it is complete, all of the runtime dependencies, configuration, logging, and service startup are installed and run on first boot of an instance created with the image. This is a core concept of baking images, because they do not change after they are created.

Creating the Spinnaker application

In this section, you create a new Spinnaker application and prepare it for the sample application pipeline by adding a load balancer and security group. GCP load balancers use forwarding rules, so you must apply the firewall rules to the instances to allow traffic. The process of creating and applying forwarding rules and security groups is abstracted and provisioned for you in Spinnaker.

On the Applications tab in Spinnaker:

  1. In the top right of the UI, click Actions > Create Application.
  2. Set Name to sample.
  3. Set Owner Email to your email address.
  4. Set Account (s) to my-google-account.
  5. Click Create.

    image

Create a load balancer

Next, create a load balancer with HTTP health checks:

  1. In the top right of the UI, click Load Balancers.
  2. Click the Create Load Balancer button.
  3. For load balancer type, choose Network.
  4. Click Create network load balancer.
  5. Set the TCP Port Range to 80.
  6. Check the Enable health check? box.
  7. Click Create.

    image

Spinnaker executes the necessary tasks in your GCP account to create the load balancer and forwarding rules.

Create a security group

Next, create a security group for the application:

  1. At the top right, click Security Groups.
  2. Click Create Security Group, and then make the following selections:

    • Under Location, set Detail to http, and Description to allow-http.
    • Under Targets, click the Targets drop-down, click Specify target tags, click Add New Target Tag, and then type "allow-http".
    • Under Source Filters, under Source CIDRs, click Add New Source CIDR, 0.0.0.0/0.
    • Under Ingress, click Add New Protocol and Port Range, set Protocol to TCP, and set both Start Port and End Port to 80.
  3. Click Create.

    image

Spinnaker executes the necessary tasks in your GCP account to create the firewall rules for the security group.

Creating the Spinnaker pipeline

In this section, you define the pipeline of stages to test, bake, and deploy your application. The stages are triggered using GitHub Webhooks.

image

In the sample application, create the pipeline:

  1. Click Pipelines and then click Create Pipeline or Strategy.

    image

  2. Set Type to Pipeline.

  3. Set Pipeline Name to sample-app.
  4. Click Create Pipeline.

Configure triggers and add parameters

Create a trigger that fires so that when a commit is made to the sample-app repository, the pipeline then builds and deploys the new version. Parameters flow through the pipeline stages and are referenced in the gce-ansible.json template. Parameters also make your pipeline more generic, so that you can use it as a template for other applications.

  1. Configure GitHub Webhooks with a signing secret to receive commit notifications for the repository.
  2. From your local terminal, generate a secret to share with the Spinnaker pipeline configuration:

    export GH_WEBHOOK_SECRET=$(openssl rand -base64 15)
    echo $GH_WEBHOOK_SECRET

  3. In the Spinnaker UI, under Automated Triggers, click Add Trigger, and then fill in the fields as follows:

    • Select Type: Git.
    • Select Repo Type: github.
    • Set Organization or User to your repository owner.
    • Set Project to your repository name, sample-app.
    • Set Branch to master.
    • Set Secret to the value of the GH_WEBHOOK_SECRET you generated earlier.
  4. Click Save Changes.

    image

Add a pipeline parameter

Add a pipeline parameter to specify the repository URL. You use this parameter later.

  1. Under Parameters, click Add Parameter.
  2. Set Name to repository.
  3. Set Default Value to the HTTPS clone URL of your GitHub repository. Replace [YOUR_GH_USER] with your GitHub username:

    https://github.com/[YOUR_GH_USER]/sample-app.git
    
  4. Click Save Changes.

    image

Create the test stage

The test stage starts a Jenkins job to test your application prior to deployment. The job calls the build.sh script in the root of your repository.

  1. At the top of the Pipeline configuration page, click Add stage.
  2. For Type, select Jenkins.
  3. For Stage Name, type "Test", and then make the following selections:

    • Under Jenkins Configuration, for Master, select Jenkins.
    • For Job, select runSpinnakerScript.
  4. Under Job Parameters, make the following selections:

    • Set COMMAND to bash build.sh.
    • Set REPO_URL to ${parameters.repository}.
  5. Click Save Changes.

    image

Create the bake stage

The bake-with-Ansible stage runs after the test stage. You use your custom gce-ansible.json template to run the Ansible provisioner to generate a new immutable image for deployment.

  1. Click Add Stage.
  2. For Type, select Bake.
  3. For Stage Name, type "Bake with Ansible", and then confirm that Depends On is set to Test.
  4. Under Bake Configuration, make the following selections:

    • Set Package to none.
    • For Base OS, click xenial (v16.04).
    • For Base Label, click Release.
    • For Rebake, check the Rebake image without regard to the status of any existing bake box.
  5. Click Show Advanced Options.

  6. Set Template File Name to gce-ansible.json.
  7. Click Add Extended Attribute.

    • Set Key to repository and Value to ${parameters.repository}, and then click Add.
    • Click Add Extended Attribute again, set Key to repository_hash, Value to ${trigger.hash ?: 'master'}, and then click Add.
    • Set the Base Image to ubuntu-1604-xenial-v20170328.
  8. Click Save Changes.

    image

Create the deploy stage

The deploy stage runs after the bake-with-Ansible stage and performs all of the automation for creating new server groups with VMs launched from the baked image, adding them to the load balancer and swapping out the old server group using a red/black (blue/green) strategy so that changes can be rolled back.

  1. Click the Add stage button.
  2. For Type, select Deploy.
  3. Set Stage Name to Deploy, and then confirm that Depends On is set to Bake with Ansible.
  4. Under Deploy Configuration, click the Add server group button.
  5. Select Copy configuration from: None.
  6. Click Continue without a template, and then make the following selections:

    • Set Stack to nginx.
    • Select Strategy: Red/Black.
    • Set Maximum number of server groups to leave to: 2.
    • Under Load Balancers, select the sample load balancer that you created earlier.
    • Under Security Groups, select the sample-http security group that you created earlier.

    image

Continue the deployment cluster configuration

  1. Under Instance Type, select General Purpose, size Small.
  2. Under AutoHealing Policy, check Enable AutoHealing.
  3. Set HTTP Health Check to the sample-hc- health check that was created automatically when you created the load balancer earlier.
  4. Select Metric: percent and set Max Unavailable to 10%.
  5. Confirm that under Advanced Settings, the sample-http- tag is added under Tags.
  6. Click Add, and then click Save Changes.

Executing the pipeline manually

Now that you have created all 3 stages of the pipeline, start a manual execution to verify that the stages execute and deploy properly before you enable the commit trigger:

  1. Click the Pipelines list at the top of the Sample application view.
  2. Click the Start Manual Execution button on the sample-app pipeline, and then click Run to run it for the first time.

The test, bake-with-Ansible, and deploy stages run to deploy the master branch of the sample-app repository.

You can monitor each stage of the pipeline by clicking it to expand the details. During the bake stage, click the View Bakery Details link to open a new tab that shows output from the Packer execution.

image

All stages should pass, and in a few minutes the app should be available at the IP address of the load balancer.

To find the IP of the load balancer in the Spinnaker UI:

  1. Click the Load Balancers button.
  2. Under the sample load balancer, click the region US-WEST1.
  3. Find the IP address under the DNS Name field.

Triggering a build from GitHub webhook

Next you configure your GitHub repository to send notifications to the Spinnaker API that trigger a new build. Webhooks are generally the preferred method for detecting changes in a source repository, as opposed to polling. Using webhooks can reduce the overall load on the repository host and can make it easier to extract detailed commit changes.

In this solution, because you used the deployment manager template, the Spinnaker webhook API endpoint /gate/webhooks/git/github was exposed automatically. GitHub will send notifications to this endpoint. For security purposes, this is the only Spinnaker API endpoint exposed by the load balancer.

Configure GitHub webhooks

  1. Find the IP address of the global forwarding rule that points to the Spinnaker API, which was created with the Cloud Deployment Manager template.

    gcloud compute forwarding-rules list --regexp="${DEPLOYMENT_NAME}-spinnaker-api-lb"
    
  2. Set the webhook to the following URL:

    http://HTTP_LB_IP/gate/webhooks/git/github
    

In the preceding URL, HTTP_LB_IP is the IP address of the forwarding rule.

  1. Next, open your GitHub repository settings:

    https://github.com/[YOUR_GH_USER]/sample-app/settings/hooks
    

    In the preceding URL, [YOUR_GH_USER] is the user or org you forked the sample-app repository into.

  2. Click the Add Webhook button.

  3. Set Payload URL to your webhook URL.
  4. Set Content type to application/json.
  5. Set Secret to the value of GH_WEBHOOK_SECRET created earlier.
  6. Click Add webhook.

    image

After you click Add webhook, a test payload is sent to Spinnaker. If it is successful, a green checkmark appears by the webhook in the GitHub page.

image

You can also see the receipt of the webhook in the logs of the Echo service on the Spinnaker VM:

grep Webhook /var/log/spinnaker/echo/echo.log

Trigger the Spinnaker pipeline by merging a pull request

Merging a pull request on GitHub is the same as committing a new change from a branch in the Git repository. Pull requests are considered best practice for the collaborative review and safer merging of open source software on GitHub.

  1. Edit the html.go file in the GitHub code browser.
  2. Change Color from Green to Blue, and then select Create a new branch for this commit and start a pull request.
  3. Click the Commit changes button.

After you merge the pull request, the Spinnaker pipeline is triggered and the pipeline stages start executing.

image

Notice that this time the source of the execution is GIT.

image

After the deployment is complete, under Clusters, there are two server groups: V000 and V001. The V000 server group is disabled.

Rolling back a cluster to a previous version

Earlier you configured the deploy stage of the pipeline with the Red/Black strategy, which leaves the older versions of the server groups in place but disables them so they are not part of the load balancer. If you need to roll back a change, the old server group is still available; you don't need to recreate it.

Roll back to the previous version of the sample-app:

  1. Click Clusters at the top of the sample application view.
  2. Click the V001 server group.
  3. On the upper right-hand side of the UI, click the Server Group Actions button, and then select Rollback.
  4. Select Restore to: sample-nginx-v000.
  5. Type the name of your account in the field at the bottom of the dialog.
  6. Click Submit.

image

This executes several tasks to re-enable the V000 server group and add it to the load balancer.

After a few minutes, traffic to the load balancer should reflect the state of the sample-app before the pull request was committed.

Now, commit another change to the html.go file using a pull request and trigger the pipeline to roll out V002.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Spinnaker stores its pipeline configurations to your Cloud Storage bucket so that they persist across restarts of the Spinnaker server. You must delete this Cloud Storage bucket before you delete the deployment that created your Spinnaker stack.

Follow the steps below to clean up and remove all resources created by this tutorial, so that you don't incur additional charges.

Delete resources using Spinnaker

In the Spinnaker UI:

  1. Under Pipelines, Configure the sample-app pipeline, and then click Pipeline Actions and Delete to delete the pipeline.
  2. Under Clusters, destroy all server groups.
  3. Under Load Balancers, delete the load balancer.
  4. Under Security Groups, delete the security group.
  5. On the main application page, click Config, and then click Delete application.

Delete resources by using your local terminal

From your local terminal, run the following code to stop front50, delete the object and the bucket, and then delete the deployment.

gcloud compute ssh ${SPINNAKER_VM} -- sudo service front50 stop
gsutil rm -r gs://spinnaker-${GOOGLE_PROJECT}-${DEPLOYMENT_NAME}/front50
gsutil rb gs://spinnaker-${GOOGLE_PROJECT}-${DEPLOYMENT_NAME}

From your local terminal, delete each of the images created by Packer:

gcloud compute images list --regexp "none-all-[0-9]{13}.+-xenial"

From your local terminal, delete the deployment:

gcloud deployment-manager deployments delete ${DEPLOYMENT_NAME}

What's next

Send feedback about...