Continuous Delivery with Travis CI

Bill Prin
September 2015

Testing is an important aspect of creating high-quality software. To ensure that they run unit tests against every build of a project, many engineering organizations have adopted the practice of continuous integration (CI), which involves using tools like Jenkins or Travis CI. Continuous delivery (CD) goes one step farther by ensuring every build can also be deployed into a production- like environment and then pass integration tests in that environment. It's easy to run integration tests by using a CI service on Google Cloud Platform to deploy your app as part of the build process.

Although this article discusses Travis CI, most of the ideas apply to other CI tools, such as Jenkins. While Travis lacks some of the flexibility and configurability of Jenkins, it's easy to get started, integrates very well with Github, runs tests in isolation, and best of all, you don't need to manage your own infrastructure.

About the tutorial

This tutorial demonstrates a Python app that runs on Google App Engine with Python Flask and interacts with the Google Books API. The app demonstrates how Travis can deploy and run end-to-end tests on a staging environment, as part of the testing process, triggered by a git push command. The following diagram shows the process from a high level:

Travis CI runs local tests between GitHub and Cloud Platform deployment

After walking through the App Engine example, this article discusses considerations for integration with the Google App Engine flexible environment and Compute Engine. The GitHub repo that contains the sample also has a managed VMs branch that demonstrates deploying the same app by using the flexible environment.

Travis CI has built-in support for deploying to App Engine standard or flexible environments by using deployment providers. When you use these providers, some of the basic setup steps stay the same, but the configuration file is simpler. This tutorial demonstrates what these providers do under the hood. It downloads the Google Cloud SDK, which can also be used for purposes other than deployment, such as running local emulators for your unit tests. For more information, see Using Travis CI deployment providers, later in this article.

Prerequisites

  1. Run Linux or Mac OS X.
  2. Install the Google Cloud SDK.
  3. Have a GitHub account and install git on your local computer. This tutorial assumes that you have a basic knowledge of GitHub and git.
  4. Sign up for Travis. Travis integrates with GitHub by using your GitHub account.
  5. Install the Travis command-line tool. You'll need this tool to encrypt credentials. You might need to install Ruby first.

Set up the sample code

  1. Fork the sample repo. In GitHub, click Fork. It's important to fork because you will replace the sample project’s credentials with your own project’s credentials.

  2. Clone the fork that you created. In a terminal window, enter the following command. Replace <your-GitHub-username> with your username:

    $ git clone https://github.com/<your-GitHub-username>/continuous-deployment-demo.git
    
  3. Change directory to the repo:

    $ cd continuous-deployment-demo
    

How the sample works

The sample app has a simple Python Flask endpoint that looks up the author of a book by it’s title:

@app.route('/get_author/<title>')
def get_author(title):
    host = 'https://www.googleapis.com/books/v1/volumes?q={}&key={}&country=US'.format(title, key)
    request = urllib2.Request(host)
    try:
        response = urllib2.urlopen(request)
    except urllib2.HTTPError, error:
        contents = error.read()
        print ('Received error from Books API {}'.format(contents))
        return str(contents)
    html = response.read()
    author = json.loads(html)['items'][0]['volumeInfo']['authors'][0]
    return author

An integration test accesses this endpoint, and then checks that, given the book title “Ulysses”, the app returns the author named “James Joyce”:

response = urllib2.urlopen("{}/get_author/ulysses".format(HOST))
html = response.read()
assert(html == "James Joyce")

Creating a project

Create a Google Cloud Platform Console project to use as a staging environment for deployment.

  1. Select or create a Cloud Platform project.

    Go to the Projects page

  2. Enable the Books API.

    Enable the API

Specifying the host

  1. You specify the App Engine app in e2e_test.py. Edit the file and modify the following line to use the URL for your app in place of the sample URL:

    HOST='http://continuous-deployment-python.appspot.com'
    
  2. You must specify your project ID in the configuration file, sometimes called the Travis file. Edit the file named .travis.yml. In the following line, replace continuous-deployment-python with the name of your Cloud Platform Console project:

    - gcloud config set project `continuous-deployment-python`
    
  3. Commit the files:

    $ git add e2e_test.py .travis.yml
    $ git commit -m "Changed host and project ID"
    

Creating the credentials

To deploy the app to this project, you must create two types of credentials:

  • Service account credentials that enable the Cloud SDK to authenticate successfully with the Cloud Platform project.
  • A public API key that the App Engine app uses to communicate with the Books API.

Create the service account credentials

  1. In your Cloud Platform Console project, open the Credentials page.
  2. Click Create credentials > Service account key.
  3. Under Service account select New service account.
  4. Enter a Service account name such as continuous-integration-test.
  5. Under Role, select Project > Editor.
  6. Under Key type, select JSON.
  7. Click Create. The Cloud Platform Console downloads a new JSON file to your computer. The name of this file starts with your project ID.
  8. Copy this file to the root of your GitHub repo and rename the file client-secret.json.

Create the public API key

  1. From the same Credentials page, click Create credentials > API key.

    A key will automatically be created.

  2. Copy the API key to the clipboard.

  3. Edit api_key.py.sample and then replace YOUR-API-KEY with the API key you copied from the console.
  4. Save the file. When you save it, change the name by removing .sample from the file name extension:

    api_key.py
    
  5. Exit the editor.

Encryption and decryption

Because these credentials are added to a public GitHub repo, they need to be encrypted and decrypted on the Travis side. You can read more about encrypting files for Travis CI.

  1. Create a tar archive file containing both of these credentials. This is important because Travis CI can decrypt only one file. In a terminal window, run the following command:

    $ tar -czf credentials.tar.gz client-secret.json api_key.py
    
  2. In the .travis.yml file, remove the encryption information. This information will be automatically added when you encrypt your keys in the next step. Edit the file and after before_install, remove the follow line:

    - openssl aes-256-cbc -K $encrypted_4cb330591e04_key -iv $encrypted_4cb330591e04_iv -in credentials.tar.gz.enc -out credentials.tar.gz
    
  3. Log in to Travis. You'll be prompted for your GitHub login credentials:

    $ sudo travis login
    
  4. Encrypt the file locally. If prompted to overwrite the existing file, respond yes:

    $ sudo travis encrypt-file credentials.tar.gz --add
    

    It's the --add option that causes the Travis client to automatically add the decryption step to the before_install step in the .travis.yml file.

  5. Add the encrypted archive to the repo:

    $ git add credentials.tar.gz.enc .travis.yml
    $ git commit -m "Adds encrypted credentials for Travis"
    

Understanding the .travis.yml file

The .travis.yml file in the sample contains line-by-line comments that explain what's in the file. Here are some things worth noting:

  • The cache command keeps the Cloud SDK distribution instantiated between runs, making repeated runs of tests somewhat faster.

  • In the before_install section, the credentials tarball file is decrypted with the keys you uploaded by using the Travis tool. Then, gcloud auth activate-service-account authenticates by using those credentials, allowing deployment to the project.

    # Decrypt the credentials we added to the repo using the key we added with the Travis command line tool
    - openssl aes-256-cbc -K $encrypted_45d1b36fa803_key -iv $encrypted_45d1b36fa803_iv -in credentials.tar.gz.enc -out credentials.tar.gz -d
    # If the SDK is not already cached, download it and unpack it
    - if [ ! -d ${HOME}/google-cloud-sdk ]; then
         curl https://sdk.cloud.google.com | bash;
      fi
    - tar -xzf credentials.tar.gz
    - mkdir -p lib
    # Here we use the decrypted service account credentials to authenticate the command line tool
    - gcloud auth activate-service-account --key-file client-secret.json

  • Finally, during the script stage, the project is deployed and the end-to-end test is run.

    # Deploy the app
    - gcloud -q preview app deploy app.yaml --promote
    # Run and end to end test
    - python e2e_test.py

Enabling your fork in Travis CI

Now that everything is ready, enable your fork:

  1. On the Travis website, click your user name in the upper-right corner to open your Travis account settings.
  2. If you don't see the sample repo in the list, click Sync.
  3. When the sample repo is in the list, flick the repository switch on, as shown in the following image:

Travis CI account settings page shows the list of projects

Note that your list will show your username in place of GoogleCloudPlatform.

Every time you push a commit to the codebase, the code is deployed and the integration test runs against in an actual environment. Do a push now to start a build:

    $ git push

Seeing Travis at work

When the tests finish, you can examine the results to see if your tests pass or fail.

When Travis tests pass, the page is green

See what happens if you add a small typo to the code by removing the 's' from the word volumes in the request to the Books API.

  1. Edit main.py.
  2. Change the following line. Remove the letter 's' from the word volumes:

    host = 'https://www.googleapis.com/books/v1/volumes?q={}&key={}&country=US'.format(title, api_key)

  3. Save the file and exit the editor.

  4. Commit the changes:

    $ git add main.py
    $ git commit -m "Intentional error."
    $ git push
    

After committing this code to the repo, the integration test fails:

When Travis tests fail, the page is red

Continuous deployment on App Engine flexible environment instances

If you use flexible environment, you only need to make one small change to keep the deployment working on Travis. You must add the following line to the .travis.yml file in the before_install section:

- ssh-keygen -q -N "" -f ~/.ssh/google_compute_engine

This step creates the necessary SSH keys, with an empty passphrase, which the Cloud SDK uses to copy the project files to a VM and build the Docker image for the application:

- gcloud -q app deploy

Continuous deployment on Compute Engine

Because the steps to deploy to a Compute Engine instance can vary greatly, the Travis steps to deploy will vary accordingly. The one step that will likely remain the same is downloading the Cloud SDK and authenticating by using service account credentials.

One possible way to deploy is to have an existing VM setup beforehand, then use gcloud compute copy-files to copy the build artifact to the instance, followed by a gcloud compute ssh command to kick off the deployment scripts.

You also need the ssh-keygen step, previously described for the App Engine flexible environment, to run the gcloud compute copy-files command from a Travis build.

Using Travis CI deployment providers

Travis CI has added experimental support to incorporate many of the steps built into their deployment tool as a simplification of the Travis file. You still need to download service account credentials, but Travis automatically downloads the Cloud SDK and runs the deployment command. You can configure the branch to deploy from, the version to deploy to, the name of the service account credentials, and other settings.

You can view the appengine_travis_deploy and managed_vms_travis_deploy repos to see what the Travis configuration looks like for this option. Here’s what the deployment configuration looks like for the App Engine example:

deploy:
    provider: gae
    # Skip cleanup so api_key.py and vendored dependencies are still there
    skip_cleanup: true
    keyfile: client-secret.json
    project: continuous-deployment-python
    default: true
    on:
        all_branches: true

Note the skip_cleanup setting. This option prevents the working directory from being cleaned before the deploy step. The demo requires this option to prevent deletion of api_key.py and the libraries installed by pip.

Other considerations

In the gcloud app deploy command, you can specify a specific version by using the--version option. You could use Travis environment variables to deploy a specific version for that build and then test against the version URL.

If you run end-to-end tests in the same job that you use to block merges from your development branch to your master branch, you might slow down your development process. Instead, you can use only the fast tests to approve the merge, while having the slow tests run periodically against the latest commit. Alternatively, you could kick off a Cloud Pub/Sub message to trigger slow tests in the background.

Cleaning up

After you've finished the Travis CI tutorial, you can clean up the resources you created on Google Cloud Platform so you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.

Deleting the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

  1. In the Cloud Platform Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete project. After selecting the checkbox next to the project name, click
      Delete project
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Stopping the App Engine app

The only way you can delete the default version of your App Engine app is by deleting your project. However, you can stop the default version in the Cloud Platform Console. This action shuts down all instances associated with the version. You can restart these instances later if needed.

In the App Engine standard environment, you can stop the default version only if your app has manual or basic scaling.

Deleting App Engine flexible environment instances

To delete an app version:

  1. In the Cloud Platform Console, go to the App Engine Versions page.

    Go to the Versions page

  2. Click the checkbox next to the non-default app version you want to delete.
  3. Click the Delete button at the top of the page to delete the app version.

Next steps

  • Learn how to use Google Cloud Platform products to build end-to-end solutions.
  • Try out other Google Cloud Platform features for yourself. Have a look at our tutorials.

Send feedback about...