This topic teaches you best practices for using Jenkins with Google Kubernetes Engine. To implement this solution, see setting up Jenkins on Kubernetes Engine.
Jenkins is an open-source automation server that lets you flexibly orchestrate your build, test, and deployment pipelines. Kubernetes Engine is a hosted version of Kubernetes, a powerful cluster manager and orchestration system for containers.
When you need to set up a continuous delivery (CD) pipeline, deploying Jenkins on Kubernetes Engine provides important benefits over a standard VM-based deployment:
When your build process uses containers, one virtual host can run jobs against different operating systems.
Kubernetes Engine provides ephemeral build executors, allowing each build to run in a clean environment that's identical to the builds before it.
As part of the ephemerality of the build executors, the Kubernetes Engine cluster is only utilized when builds are actively running, leaving resources available for other cluster tasks such as batch processing jobs.
Build executors launch in seconds.
Kubernetes Engine leverages the Google global load balancer to route web traffic to your instance. The load balancer handles SSL termination, and provides a global IP address that routes users to your web front end on one of the fastest paths from the point of presence closest to your users through the Google backbone network.
For a deep dive into Jenkins on Kubernetes Engine, watch the Next 2018 talk on YouTube:
The State of DevOps reports identified capabilities that drive software delivery performance. This topic will help you with the following capabilities:
Deploying the Jenkins controller with Helm
The following image describes the architecture for deploying Jenkins in a multi-node Kubernetes cluster.
Deploy the Jenkins controller into a separate namespace in the Kubernetes cluster. Namespaces allow for creating quotas for the Jenkins deployment as well as logically separating Jenkins from other deployments within the cluster.
Creating Jenkins services
Jenkins provides two services that the cluster needs access to. Deploy these services separately so they can be individually managed and named.
An externally-exposed NodePort service on port 8080 that allows pods and external users to access the Jenkins user interface. This type of service can be load balanced by an HTTP load balancer.
An internal, private ClusterIP service on port 50000 that the Jenkins executors use to communicate with the Jenkins controller from inside the cluster.
The following sections show sample service definitions. (The controller is
master in the definitions.)
--- kind: Service apiVersion: v1 metadata: name: jenkins-ui namespace: jenkins spec: type: NodePort selector: app: master ports: - protocol: TCP port: 8080 targetPort: 8080 name: ui
--- kind: Service apiVersion: v1 metadata: name: jenkins-discovery namespace: jenkins spec: selector: app: master ports: - protocol: TCP port: 50000 targetPort: 50000 name: slaves
Creating the Jenkins deployment
Deploy the Jenkins controller as a deployment with a replica count of 1. This ensures that there is a single Jenkins controller running in the cluster at all times. If the Jenkins controller pod dies or the node that it is running on shuts down, Kubernetes restarts the pod elsewhere in the cluster.
It's important to set requests and limits as part of the Helm deployment, so that the container is guaranteed a certain amount of CPU and memory resources inside the cluster before being scheduled. Otherwise, your controller could go down due to CPU or memory starvation.
The Jenkins home volume stores XML configuration files and plugin JAR files
that make up your configuration. This data is stored on a Persistent Disk
managed by the GKE cluster and will persist data across restarts of Jenkins. To
change the size of the persistent disk edit the
Persistence.Size value when installing
Jenkins with Helm.
Connecting to Jenkins
Once the Jenkins pod has been created you can create a load balancer endpoint to connect to it from outside of Cloud Platform. Consider the following best practices.
Use a Kubernetes ingress resource for an easy-to-configure L7 load balancer with SSL termination.
Provide SSL certs to the load balancer using Kubernetes secrets. Use
tls.keyvalues, and reference the values in your ingress resource configuration.
After you connect to Jenkins for the first time, it's important to immediately secure Jenkins. You can follow the Jenkins standard security setup tutorial for a simple procedure that leverages an internal user database. This setup doesn't require additional infrastructure and provides the ability to lock out anonymous users.
You can install the following plugins to enhance the interactions between Jenkins and Kubernetes Engine.
The Kubernetes plugin enables using Kubernetes service accounts for authentication, and creating labeled executor configurations with different base images. The plugin creates a pod when an executor is required and destroys the pod when a job ends.
The Google Authenticated Source plugin enables using your service account credentials when accessing Cloud Platform services such as Cloud Source Repositories.
To add additional plugins using the Helm chart, edit the list of plugins in the values file that you pass to the Helm install or upgrade commands.
Customizing the Jenkins Agent Docker image
When creating a pod template, you can either provide an existing Docker image, or you can create a custom image that has most of your build-time dependencies installed. Using a custom image can decrease overall build time and create more consistent build environments.
Your custom Docker image must install and configure the Jenkins JNLP inbound agent. The JNLP agent is software that communicates with the Jenkins controller to coordinate running your Jenkins jobs and reporting job status.
One option is to add
FROM jenkins/inbound-agent to your image configuration.
For example, if your application build process depends on the Go runtime, you
can create the following Dockerfile to extend the existing image with your own
dependencies and build artifacts.
FROM jenkins/inbound-agent RUN apt-get update && apt-get install -y golang
Then, build and upload the image to your project's Container Registry repository by running the following commands.
docker build -t gcr.io/[PROJECT]/my-jenkins-image .
gcloud auth configure-docker
docker push gcr.io/[PROJECT]/my-jenkins-image
When creating a pod template, you can now set the Docker image field to the
following string, where
[PROJECT] is replaced with your project name and
[IMAGE_NAME] is replaced with the image name.
The above example ensures that the Go language runtime is pre-installed when your Jenkins job starts.
Building Docker Images in Jenkins
Cloud Build can be used from within your Jenkins jobs to build Docker images without
needing to host your own Docker daemon. Your Jenkins job must have service account
credentials available that have been granted the
For an example Jenkins Pipeline file, see this GitHub repository.
Kaniko is another option for users looking to build containers inside their clusters. Kaniko does not require a Docker daemon to be present in order to build and push images to a remote registry.
For an example of using Kaniko in Jenkins, see this GitHub repository.
Learn about how to set up continuous deployment to Kubernetes Engine using Jenkins.
Read our resources about DevOps.
Learn more about the DevOps capabilities related to this solution:
Take the DevOps quick check to understand where you stand in comparison with the rest of the industry.
Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.