Creating custom base images for Compute Engine with Jenkins and Packer

This tutorial shows how to use Jenkins, Packer, and QEMU to convert your existing ISO CD-ROM and Kickstart script installations into custom base images for Compute Engine. Because GCP includes a rich set of base images, you don't always need to create custom base images.

architecture diagram

The tutorial uses the Jenkins plugin for Compute Engine to provision compute instances as Jenkins agents that ultimately build the custom base image.

Objectives

  • Create a Google Kubernetes Engine cluster.
  • Install Helm, a Kubernetes package manager.
  • Install Jenkins by using Helm.
  • Configure Jenkins to use Compute Engine instances as Jenkins agents.
  • Configure a Jenkins job to run Packer jobs.
  • Create a Jenkins job that customizes an existing ISO image and imports the result as a Compute Engine image.

Costs

This tutorial uses the following billable components of Google Cloud Platform:

  • Compute Engine
  • GKE
  • Cloud Storage

You can use the pricing calculator to generate a cost estimate based on your projected usage. New GCP users might be eligible for a free trial.

For example: assuming you use a single Compute Engine virtual machine once, for 77 minutes, your total cost will be $176. For details, see this estimate.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Google Cloud Platform project.

    Go to the Manage resources page

  3. Make sure that billing is enabled for your Google Cloud Platform project.

    Learn how to enable billing

  4. Enable the Compute Engine API.

    Enable the API

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See Cleaning up for more detail.

Preparing your environment

Cloud Shell gives you access to the command line in the GCP Console and includes Cloud SDK and other tools that you use to develop in GCP. Cloud Shell appears as a window at the bottom of the GCP Console. It can take several minutes to initialize, but the window appears immediately.

  1. Activate Cloud Shell:

    ACTIVATE Cloud Shell

  2. Set up your environment in Cloud Shell:

    export PROJECT=$(gcloud info --format='value(config.project)')
  3. Set the default Compute Engine zone:

    export ZONE=us-central1-b
    gcloud config set compute/zone $ZONE
    
  4. Enable the Kubernetes Engine API:

    gcloud services enable container.googleapis.com
    
  5. Create a storage bucket to upload images:

    gsutil mb "gs://$PROJECT-custom-images"
    

Deploying the Jenkins server

In this section, you create a Kubernetes cluster to host Jenkins, install Helm to deploy Jenkins on the cluster, install Jenkins, and connect to Jenkins to set up a service account.

Create and configure the Kubernetes cluster

  1. Create the Kubernetes cluster that will host Jenkins:

    VERSION=$(gcloud container get-server-config --zone $ZONE \
         --format='value(validMasterVersions[0])')
    gcloud container clusters create builder --zone=$ZONE \
        --cluster-version=${VERSION} \
        --machine-type n1-standard-2 \
        --num-nodes 2 \
        --scopes='https://www.googleapis.com/auth/projecthosting,storage-rw,cloud-platform'
    
  2. Confirm that the cluster's status is RUNNING:

    gcloud container clusters list
    

    Expected output:

    NAME     LOCATION       MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUS
    builder  us-central1-b  1.10.7-gke.11   35.184.60.60  n1-standard-2  1.10.7-gke.6 *  2          RUNNING
    

Install Helm

You use Helm to deploy applications to your Kubernetes cluster. After creating your cluster, you configure Helm to work with the cluster.

  1. In Cloud Shell, get the credentials for the Kubernetes cluster:

    gcloud container clusters get-credentials builder
    
  2. Download and extract Helm:

    wget https://storage.googleapis.com/kubernetes-helm/\
    helm-v2.9.1-linux-amd64.tar.gz
    tar zxfv helm-v2.9.1-linux-amd64.tar.gz
    cp linux-amd64/helm .
    
  3. Assign the appropriate access controls so that you can grant Jenkins permissions in the cluster:

    kubectl create clusterrolebinding \
        cluster-admin-binding --clusterrole=cluster-admin \
        --user=$(gcloud config get-value account)
    
  4. Grant the cluster-admin role in your cluster to Tiller, the server side of Helm:

    kubectl create serviceaccount tiller --namespace kube-system
    kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
    
  5. Install Helm:

    ./helm init --service-account=tiller
    ./helm update
    
  6. Verify the Helm installation:

    ./helm version
    

    Expected output:

    Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
    

Install Jenkins in the Kubernetes cluster

  1. In Cloud Shell, create the initial Jenkins configuration by using the values for the Helm chart:

    cat <<CONFIG >values.yaml
    Master:
      InstallPlugins:
        - kubernetes:1.12.6
        - workflow-aggregator:2.5
        - workflow-job:2.21
        - credentials-binding:1.16
        - git:3.9.1
        - google-oauth-plugin:0.6
        - google-source-plugin:0.3
        - google-compute-engine:1.0.5
        - google-storage-plugin:1.2
        - packer:1.5
      Cpu: "1"
      Memory: "3500Mi"
      JavaOpts: "-Xms3500m -Xmx3500m"
      ServiceType: ClusterIP
    Agent:
      Enabled: false
    Persistence:
      Size: 100Gi
    NetworkPolicy:
      ApiVersion: networking.k8s.io/v1
    rbac:
      install: true
      serviceAccountName: cd-jenkins
    CONFIG
    
  2. In Cloud Shell, use the Helm command-line interface to deploy the chart with your configuration set:

    ./helm install -n cd \
    stable/jenkins -f values.yaml --wait
    
  3. Verify the Jenkins installation, and ensure that the Jenkins pod is in a Running state:

    kubectl get pods
    

    Expected output:

    NAME                            READY     STATUS    RESTARTS   AGE
    cd-jenkins-7c786475dd-qdxpr     1/1       Running   0          3h
    

Connect to Jenkins

  1. In Cloud Shell, get the Jenkins password, and make a note of it for later in this tutorial:

    printf $(kubectl get secret cd-jenkins -o \
        jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
    
  2. Forward port 8080 to the Jenkins deployment in your Kubernetes cluster:

    export POD_NAME=$(kubectl get pods -l "component=cd-jenkins-master"\
        -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward $POD_NAME 8080:8080 >> /dev/null &
    
  3. To open the Jenkins user interface, click Web preview in Cloud Shell, and then click Preview on port 8080.

  4. Log in as admin with the password from step 1.

Create the Jenkins service account

The Jenkins plugin in Compute Engine uses the Jenkins service account to provision agents that perform the nested image build.

  1. Return to the original Cloud Shell window.
  2. Create a service account that is named jenkins and store its email address:

    gcloud iam service-accounts create jenkins --display-name jenkins
    export SA_EMAIL=$(gcloud iam service-accounts list \
        --filter="displayName:jenkins" --format='value(email)')
    
  3. Bind the following roles to the service account:

    gcloud projects add-iam-policy-binding $PROJECT \
        --role roles/storage.admin --member serviceAccount:$SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT --role roles/compute.instanceAdmin.v1 \
        --member serviceAccount:$SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT --role roles/compute.networkAdmin \
        --member serviceAccount:$SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT --role roles/compute.securityAdmin \
        --member serviceAccount:$SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT --role roles/iam.serviceAccountActor \
        --member serviceAccount:$SA_EMAIL
    
  4. Create the service account key:

    gcloud iam service-accounts keys \
        create jenkins-sa.json --iam-account $SA_EMAIL
    
  5. Copy the result of the following command to the clipboard:

    echo "$(pwd)/jenkins-sa.json"
    
  6. In Cloud Shell, click the More more_vert button.

  7. Click Download file, which opens a dialog box for entering the path to the file.

  8. Paste the copied contents into the text box.

  9. Click Download to save the file locally.

Building the Jenkins agent image

In this section, you build an image for the Jenkins agent that includes the following:

  • Nested virtualization
  • Packer
  • QEMU-KVM
  • Java

Create a base image with nested virtualization enabled

  1. Create a disk with a standard centos-7 image:

    gcloud compute disks create disk1 \
        --image-project centos-cloud \
        --image-family centos-7 \
        --zone ${ZONE}
    
  2. Create an image that has nested virtualization turned on:

    gcloud compute images create nested-vm-image \
        --source-disk disk1 \
        --source-disk-zone ${ZONE} \
        --licenses \
        "https://www.googleapis.com/compute/v1/\
    projects/vm-options/global/licenses/enable-vmx"
    

Create the Jenkins agent image

  1. Create an instance that specifies Intel Haswell as the minimum CPU platform (see nested virtualization restrictions).

    gcloud compute instances create packer-build-host \
        --zone ${ZONE} \
        --machine-type=n1-standard-4 \
        --boot-disk-size=20GB \
        --boot-disk-type=pd-ssd \
        --image nested-vm-image \
        --min-cpu-platform='Intel Haswell' \
        --metadata startup-script='#! /bin/bash
    # Install agent requirements
    sudo su -
    yum update -y
    yum install git unzip wget qemu-kvm -y
    echo "export PATH=$PATH:/usr/libexec" > /etc/profile.d/libexec-path.sh
    source /etc/profile.d/libexec-path.sh
    curl -LO \
        https://releases.hashicorp.com/packer/1.3.0/packer_1.3.0_linux_amd64.zip
    unzip packer_1.3.0_linux_amd64.zip
    cp packer /usr/bin/packerio
    yum install java-1.8.0-openjdk-devel -y'
    

    The instance is used to create an image and has the nested virtualization environment and tools for creating the builds. The image uses Git to download the image configuration from the source repository, QEMU-KVM to provide virtualization support, Java for interaction with the Jenkins agent, and Packer to automate the creation of machine images.

  2. Create the jenkins-packer-agent image, using the instance for the Packer build host as the source:

    gcloud compute images create jenkins-packer-agent-v1 \
        --source-disk packer-build-host \
        --source-disk-zone $ZONE \
        --family jenkins-packer-agent \
        --force
    

Create a Packer agent service key

  1. In Cloud Shell, create the Packer agent service account that will be assigned to agent instances:

    gcloud iam service-accounts create packer-agent-sa \
        --display-name packer-agent-sa
    export PACKER_SA_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:packer-agent-sa" --format='value(email)')
    
  2. Bind the roles to the service account:

    gcloud projects add-iam-policy-binding $PROJECT \
        --role roles/storage.admin --member serviceAccount:$PACKER_SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT \
        --role roles/compute.instanceAdmin.v1 \
        --member serviceAccount:$PACKER_SA_EMAIL
    gcloud projects add-iam-policy-binding $PROJECT --role roles/iam.serviceAccountActor \
        --member serviceAccount:$PACKER_SA_EMAIL
    

Configuring Jenkins

In this section, you add your credentials to Jenkins, set up a Compute Engine backend, configure Packer to build a Compute Engine image, run the build, and create and connect to an instance based on the image.

Configure credentials

  1. From the web preview that you opened previously, log in to Jenkins using the admin account and password.
  2. In Jenkins, in the left menu, click Credentials.
  3. In the left menu, click System.
  4. In the main pane of the UI, click Global credentials (unrestricted).
  5. In the left menu, click Add Credentials.
  6. Set Kind to Google Service Account from private key.
  7. In the Project Name field, enter your project ID.
  8. Click Choose File.
  9. Select the jenkins-sa.json file that you previously downloaded from Cloud Shell.
  10. Click OK.

Add a Compute Engine backend

  1. In Jenkins, in the left menu, click Manage Jenkins.
  2. In the main pane of the UI, click Configure System.
  3. Scroll down to the bottom Cloud section.
  4. Click Add a new cloud and select Google Compute Engine.
  5. Under Google Compute Engine, enter or select the following values:
    • Name: jenkins-packer-agent
    • Project ID: your GCP project name
    • Instance Cap: 1
    • Service Account Credentials: the credentials created in the previous section
    • Name Prefix: packerqemuagent
    • Description: packer agent qemu configuration
    • Labels: packer-qemu
    • Run as user: jenkins
    • Region: us-central-1 (This should match the default compute region set above.)
    • Zone: us-central1-b (This should match the default compute region set above.)
    • Machine Type: n1-standard-4
    • Minimum Cpu Platform: Intel Haswell
    • Network: default
    • Subnetwork: default
    • External IP?: selected
    • Image project: your project
    • Image name: jenkins-packer-agent-v1
    • Size: 30
    • Delete on termination: selected
    • Service Account E-mail: the Packer agent service account
  6. Click Save.

Configure a Jenkins Packer job

You create a Jenkins job that converts an ISO image to a Compute Engine image based on the provided sample configuration.

  1. On the left side of the Jenkins UI, click New Item.
  2. In the Enter an item name field, enter ISO Image Factory.
  3. Select Freestyle project.
  4. Click OK.
  5. Select Restrict where this project can be run.
  6. In the Label Expression field, enter packer-qemu.
  7. Under Source Code Management, select Git.
  8. In the Repository URL field, enter the sample repository:

    https://github.com/GoogleCloudPlatform/compute-custom-boot-images
    

Add the image build steps

There are two steps for building the image. First, in case the build agent is being reused, the Jenkins job cleans up any existing Packer builds. Second, the Jenkins job uses Packer to build a base image that uses the Packer script that's included in the sample repository. The Packer script includes provisioning scripts that install the necessary packages for use with Compute Engine, including the Linux Guest Environment.

  1. In the Build section, click Add build step.
  2. Click Execute shell.
  3. In the Command field, enter the following command, which deletes any generated images on the Packer agent:

    if [[ -d output ]]; then
      rm -rf output
    fi
    
  4. Click Add build step again.

  5. Click Execute shell.

  6. In the Command field, enter the following command, which builds the partially provisioned image and imports it into Compute Engine. Replace [PROJECT] with your project name and [ZONE] with the zone that you want to use to build your image.

    packerio build \
    -var "project=[PROJECT]" \
    -var "image_family=acme-centos" \
    -var "gcs_bucket=gs://[PROJECT]-custom-images" \
    -var "build_number=$BUILD_NUMBER" \
    -var "zone=[ZONE]" \
    *-from-iso.json
    

Run the build

  1. In the upper left corner, click Jenkins.
  2. Next to the build that you just created, click the Build Now button.

Launch an instance by using the built image

When the build finishes, you create an instance from the image and connect to it with SSH.

  1. In Cloud Shell, create the instance (this may take a minute or two):

    gcloud compute instances create test-image-build \
      --zone=us-central-1b \
      --image-family=acme-centos
    
  2. Test that you can log in to the image with SSH:

    gcloud compute ssh test-image-build
    

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Clean up individual resources

  1. Delete the test instance:

    gcloud compute instances delete test-image-build
    
  2. Delete the GKE cluster:

    gcloud container clusters delete builder
    
  3. Delete the image:

    gcloud compute images delete \
        $(gcloud compute images list --filter="family=acme-centos" --format="value(name)")
    
  4. Delete the storage bucket:

    gsutil rm -rf gs://$PROJECT-custom-images
    

What's next

Was this page helpful? Let us know how we did:

Send feedback about...