This tutorial shows you how to set up a continuous delivery pipeline using Jenkins and Google Kubernetes Engine (GKE), as described in the following diagram.
Objectives
- Understand a sample application.
- Deploy an application to GKE.
- Upload code to Cloud Source Repositories.
- Create deployment pipelines in Jenkins.
- Deploy development environments.
- Deploy a canary release.
- Deploy production environments.
Costs
This tutorial uses billable components of Google Cloud, including:
- Compute Engine
- Google Kubernetes Engine
- Cloud Source Repositories
- Cloud Build
Use the pricing calculator to generate a cost
estimate based on your projected usage for this tutorial.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
-
Enable the Compute Engine, GKE, and Cloud Build APIs.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
-
Enable the Compute Engine, GKE, and Cloud Build APIs.
Preparing your environment
Complete the Setting up Jenkins on GKE tutorial. Ensure that you have a working Jenkins install running in GKE.
In Cloud Shell, clone the sample code:
cd ~/ git clone https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes
Go to the sample app directory:
cd ~/continuous-deployment-on-kubernetes/sample-app
Apply the
cluster-admin
role to the Jenkins service account:kubectl create clusterrolebinding jenkins-deploy \ --clusterrole=cluster-admin --serviceaccount=default:cd-jenkins
In this tutorial, the Jenkins service account needs
cluster-admin
permissions so that it can create Kubernetes namespaces and any other resources that the app requires. For production use, you should catalog the individual permissions necessary and apply them to the service account individually.
Understanding the application
You'll deploy the sample application, gceme
, in your
continuous deployment pipeline. The application is written in the Go language,
and is located in the repository's sample-app
directory. When you run the
gceme
binary on a Compute Engine instance, the app displays the
instance's metadata in an info card.
The application mimics a microservice by supporting two operation modes:
In backend mode,
gceme
listens on port 8080 and returns Compute Engine instance metadata in JSON format.In frontend mode,
gceme
queries the backendgceme
service, and renders the resulting JSON in the user interface.
The frontend and backend modes support two additional URLs:
/version
prints the running version./healthz
reports the application's health. In frontend mode, the health displays asOK
if the backend is reachable.
Building the sample app
Build the sample app by running the following commands:
export PROJECT_ID=$(gcloud info --format='value(config.project)') find . -type f -name "*" -exec sed -i 's|gcr.io/cloud-solutions-images/gceme:1.0.0|gcr.io/'"${PROJECT_ID}"'/gceme:1.0.0|g' {} + gcloud builds submit --tag gcr.io/${PROJECT_ID}/gceme:1.0.0
Deploying the sample app to Kubernetes
Deploy the gceme
frontend and backend to Kubernetes using
manifest files that describe the deployment environment. The files use a default
image that is updated later in this tutorial.
Deploy the applications into two environments.
Production. The live site that your users access.
Canary. A smaller-capacity site that receives a percentage of your user traffic. Use this environment to sanity check your software with live traffic before it's released to the production environment.
First, deploy your application into the production environment to seed the pipeline with working code.
Create the Kubernetes namespace to logically isolate the production deployment:
kubectl create ns production
Create the canary and production deployments and services:
kubectl --namespace=production apply -f k8s/production kubectl --namespace=production apply -f k8s/canary kubectl --namespace=production apply -f k8s/services
Scale up the production environment frontends:
kubectl --namespace=production scale deployment gceme-frontend-production --replicas=4
Retrieve the external IP address for the production services. It can take several minutes before you see the load balancer IP address.
kubectl --namespace=production get service gceme-frontend
When the process completes, the external IP address is displayed in the
EXTERNAL-IP
column.NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gceme-frontend LoadBalancer 10.35.254.91 35.196.48.78 80:31088/TCP 1m
If the external IP address does not appear, wait a few minutes and repeat the previous step until the external IP address is shown.
After the external IP address appears, store the frontend service load balancer IP address in an environment variable:
export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)
Confirm that both services are working by going to the frontend external IP address in your browser.
Poll the production endpoint's
/version
URL so you can observe rolling updates in the next section:while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1; done
Press
Ctrl+C
to exit the loop.
Creating a repository to host the sample app source code
Next, create a copy of the gceme
sample app and push it to
Cloud Source Repositories.
Create the repository in Cloud Source Repositories:
gcloud services enable sourcerepo.googleapis.com gcloud source repos create gceme
Initialize the local Git repository:
rm -rf ../.git git config --global init.defaultBranch main git init git config credential.helper gcloud.sh export PROJECT_ID=$(gcloud config get-value project) git remote add origin https://source.developers.google.com/p/$PROJECT_ID/r/gceme
Set the username and email address for your Git commits in this repository to the values from your logged-in account:
git config --global user.email $(gcloud config list account --format "value(core.account)") git config --global user.name $(gcloud config list account --format "value(core.account)")
Add, commit, and push the files:
git add . git commit -m "Initial commit" git push origin main
Create a service account
In this section, you create a service account that Jenkins will use to access your Git repository and execute deployments to GKE.
Create the service account and grant source and GKE roles to it.
export SA=jenkins-sa export SA_EMAIL=${SA}@${PROJECT_ID}.iam.gserviceaccount.com gcloud iam service-accounts create $SA gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SA_EMAIL \ --role roles/source.writer gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SA_EMAIL \ --role roles/container.developer
Create and download a JSON service account key for your newly created service account:
gcloud iam service-accounts keys create ~/jenkins-gke-key.json --iam-account $SA_EMAIL
Take note of where the file was created, because you upload it to Jenkins in a later step.
For more information about downloading a service account key, see Create a service account key.
Open the Jenkins web user interface
If you haven't done so already, set up port forwarding to enable access to the Jenkins web UI:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/component=jenkins-main" -l "app.kubernetes.io/instance=cd-jenkins" -o jsonpath="{.items[0].metadata.name}") kubectl port-forward $POD_NAME 8080:8080 >> /dev/null 2>&1 &
Open the Jenkins user interface, click Web Preview in Cloud Shell, and click Preview on port 8080.
Configure Jenkins Cloud for Kubernetes
- In the Jenkins user interface, select Manage Jenkins > Manage nodes and clouds.
- Click Configure Clouds in the left navigation pane.
- Click Add a new cloud and select Kubernetes.
- Click Kubernetes Cloud Details.
- In the Jenkins URL field, enter the following value:
http://cd-jenkins:8080
- In the Jenkins tunnel field, enter the following value:
cd-jenkins-agent:50000
- Click Save.
Creating a pipeline
Use Jenkins to define and run a pipeline for testing, building, and
deploying your copy of gceme
to your Kubernetes cluster.
Add your service account credentials
Configure your credentials to allow Jenkins to access the code repository.
- In the Jenkins user interface, select Manage Jenkins > Manage Credentials in the left navigation pane.
Click the (global) link
Click Add Credentials in the left navigation.
In the Kind menu, select Google Service Account from private key.
Enter your project name, and then select your JSON key that was created in a previous section.
Click Create.
Make a note of the credential's name for use later in this tutorial.
Create a Jenkins job
Next, use the Jenkins Pipeline feature to configure the build pipeline. Jenkins Pipeline files are written using a Groovy-like syntax.
Navigate to your Jenkins user interface and follow these steps to configure a Pipeline job.
- Click the Jenkins link in the top left of the interface.
- Click the New Item link in the left navigation.
- Name the project sample-app, choose the Multibranch Pipeline option, and then click OK.
- In the Branch Sources section, click Add Source and select git.
Paste the HTTPS clone URL of your
sample-app
repository in Cloud Source Repositories into the Project Repository field. Replace[PROJECT_ID]
with your project ID.https://source.developers.google.com/p/[PROJECT_ID]/r/gceme
From the Credentials drop-down list, select the name of the credentials that you created when adding your service account.
In the Scan Multibranch Pipeline section, select the Periodically if not otherwise run box. Set the Interval value to '1 minute'.
Click Save.
After you complete these steps, a job named "Branch indexing" runs.
This meta-job identifies the branches in your repository and ensures
changes haven't occurred in existing branches. If you refresh Jenkins,
the main
branch displays this job.
The first run of the job fails until you make a few code changes in the next step.
Modify the pipeline definition
Create a branch for the canary environment called
canary
.git checkout -b canary
Update the
Jenkinsfile
to replaceREPLACE_WITH_YOUR_PROJECT_ID
on line 2 with your project ID.sed -i 's|REPLACE_WITH_YOUR_PROJECT_ID|'"$PROJECT_ID"'|' Jenkinsfile
The Jenkinsfile container that defines that pipeline is written using the Jenkins Pipeline Groovy syntax. Using a Jenkinsfile allows an entire build pipeline to be expressed in a single file that lives alongside your source code. Pipelines support powerful features like parallelization and requiring manual user approval.
Deploying a canary release
Now that your pipeline is configured properly, you can make a change to the
gceme
application and let your pipeline test, package, and deploy it.
The canary environment is set up as a canary release.
As such, your change is released to a small percentage of the pods behind
the production load balancer. You accomplish this in Kubernetes by maintaining
multiple deployments that share the same labels. For this application, the
gceme-frontend
services load balance across all pods that have the labels
app: gceme
and role: frontend
. The k8s/frontend-canary.yaml
canary
manifest file sets the replicas to 1
and includes labels required for the
gceme-frontend
service.
Currently, you have 1 out of 5 of the frontend pods running the canary code while the other 4 are running the production code. This helps ensure that the canary code doesn't negatively affect users before rolling out to your full fleet of pods.
- Open
html.go
and replace the two instances ofblue
withorange
. Open
main.go
and change the version number from1.0.0
to2.0.0
:const version string = "2.0.0"
Next, add and commit those files to your local repository:
git add Jenkinsfile html.go main.go git commit -m "Version 2"
Finally, push your changes to the remote Git server:
git push origin canary
After the change is pushed to the Git repository, navigate to the Jenkins user interface where you can see that your build started.
After the build is running, click the down arrow next to the build in the left navigation and select Console Output.
During execution, you may notice the build waiting for a while with the following message:
Still waiting to schedule task 'Jenkins' doesn't have label 'sample-app'
This message means that Jenkins is waiting for GKE to create the pods needed for the build steps. You can watch the progress on the GKE Workloads page.
Track the output of the build. When the build finishes, poll the endpoint at
/version
:while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1; done
The version begins to change in some of the requests. You have now rolled out that change to a subset of users.
Press
Ctrl+C
to exit the loop.After the change is deployed to the canary environment, you can continue to roll it out to the rest of your users by merging the code with the
main
branch and pushing that to the Git server:git checkout main git merge canary git push origin main
In approximately 1 minute, the
main
job in thesample-app
folder kicks off.Click the
main
link to show the pipeline steps, as well as pass/fail information and timing characteristics.Poll the production URL to verify that the new version 2.0.0 has been rolled out and is serving requests from all users.
export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend) while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1; done
You can look at the Jenkinsfile in the project to see the workflow. The
Jenkinsfile is located at https://source.cloud.google.com/[PROJECT_ID]/gceme/+/main:Jenkinsfile
Deploying a development branch
Sometimes you need to work with nontrivial changes that can't be pushed directly to the canary environment. A development branch is a set of environments your developers use to test their code changes before submitting them for integration into the production environment. These environments are a scaled-down version of your application, but are deployed using the same mechanisms as the production environment.
To create a development environment from a feature branch, you can push the branch to the Git server and let Jenkins deploy your environment. In a development scenario, you wouldn't use a public-facing load balancer. To help secure your application you can use kubectl proxy. The proxy authenticates itself with the Kubernetes API, and proxies requests from your local machine to the service in the cluster without exposing your service to the Internet.
In your first terminal window, create another branch and push it to the Git server:
git checkout -b new-feature git push origin new-feature
Open the Jenkins web UI and review the console output for the
new-feature
job. A new job is created and your development environment is in the process of being created. At the bottom of the console output of the job are instructions for accessing your environment.After the build has completed, start the proxy in the background:
kubectl proxy &
Verify that your application is accessible by using localhost:
curl http://localhost:8001/api/v1/namespaces/new-feature/services/gceme-frontend:80/proxy/
You can now push code to this branch to update your development environment. When you are done, merge your branch back into
canary
to deploy that code to the canary environment.git checkout canary git merge new-feature git push origin canary
When you are confident that your code won't cause problems in the production environment, merge from the
canary
branch to themain
branch to kick off the deployment:git checkout main git merge canary git push origin main
When you are done with the development branch, delete it from the server and delete the environment from your Kubernetes cluster:
git push origin :new-feature kubectl delete ns new-feature
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Deleting the project
The easiest way to eliminate billing is to delete the project that you created for the tutorial.
To delete the project:
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
What's next
- Learn more about Jenkins on GKE best practices.
See the setting up Jenkins on GKE tutorial.
Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.