Managing cloud infrastructure using kpt

This tutorial introduces kpt, an open source tool by Google that lets you work with Kubernetes configurations (also known as manifests): package them, pull them, update them, and modify them. Kpt is an alternative to template-based tools when you want to keep a clean separation between configurations and operations on those configurations. Kpt lets you reuse and share code that is acting on the configurations (either to modify or inspect them).

This tutorial also showcases how you can combine kpt with other Google solutions like Config Sync and Anthos blueprints. Whether you are a developer working with Kubernetes or a platform engineer managing a Kubernetes-based platform, this tutorial lets you discover how you can use kpt in your own Kubernetes-related workflows. This tutorial assumes that you are familiar with Kubernetes and Google Cloud.

Declarative configuration for cloud infrastructure is a well-established practice in the IT industry. It brings a powerful abstraction of the underlying systems. This abstraction frees you from having to manage low-level configuration details and dependencies. Therefore, declarative configuration has an advantage compared to imperative approaches, such as operations performed in graphical and command line interfaces.

The Kubernetes resource model has been influential in making declarative configuration approaches mainstream. Because the Kubernetes API is fully declarative by nature, you only tell Kubernetes what you want, not how to achieve what you want. The Kubernetes API lets you cleanly separate the configuration (whether desired or real) from operations on the configuration (adding, removing, and modifying objects). In other words, in the Kubernetes resource model, the configuration is data, and not code.

This separation of configuration from operations has many advantages: people and automated systems can understand and work on the configuration, and the software modifying the configuration is easily reusable. This separation also lets you easily implement a GitOps methodology (as defined in the GitOps-style continuous delivery with Cloud Build tutorial).

In this tutorial, you explore this separation of configuration declaration from configuration operations using kpt. This tutorial highlights the following features of kpt:

  • Package management: download and update Kubernetes configuration packages.
  • Setters: apply customization possibilities that a package maintainer includes within a package.
  • Functions: run arbitrary pieces of code to either modify or validate your configurations.
  • Resource management: apply, update, and delete the resources that correspond to your configurations in a Kubernetes cluster.


  • Create a Google Kubernetes Engine (GKE) cluster.
  • Use kpt to download an existing set of Kubernetes configurations.
  • Use kpt setters and configuration functions to customize the configurations.
  • Apply your configuration to the GKE cluster.
  • Use kpt to pull in some upstream changes for your configuration.
  • Use kpt in a real-world scenario to harden your GKE cluster.


This tutorial uses the following billable components of Google Cloud:

  • Google Kubernetes Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  5. Configure Cloud Shell to use your project:

    gcloud config set project PROJECT_ID
  6. In Cloud Shell, enable the Google Kubernetes Engine and Cloud Source Repositories APIs:

    gcloud services enable \

When you finish this tutorial, you can avoid continued billing by deleting the resources that you created. For more details, see Cleaning up.

Creating a GKE cluster

In this section, you create the GKE cluster where you deploy configurations later in the tutorial.

  1. In Cloud Shell, create a GKE cluster:

    gcloud container clusters create kpt-tutorial \
       --num-nodes=1 --machine-type=n1-standard-4 \
       --zone=us-central1-a --enable-network-policy
  2. Verify that you have access to the cluster. The following command returns information about the node or nodes that are in the cluster.

    kubectl get nodes

Applying a kpt package

In this section, you use kpt to download a set of configurations, customize them, and apply them to the cluster that you created in the previous section.

  1. In Cloud Shell, install kpt:

    sudo apt-get install google-cloud-sdk-kpt
  2. Download an example set of configurations. For more information, see kpt pkg get.

    kpt pkg get ./

    The preceding command downloads the helloworld-set sample package that is available in the kpt GitHub repository, at the version tagged package-examples/helloworld-set/v0.3.0.

  3. Examine the setters that are available in the configurations. For more information, see kpt cfg list-setters.

    kpt cfg list-setters helloworld-set/

    The output looks like the following:

    NAME      VALUE       SET BY             DESCRIPTION        COUNT   REQUIRED
    http-port   80      package-default   helloworld port         3       No
    image-tag   0.1.0   package-default   hello-world image tag   1       No
    replicas    5       package-default   helloworld replicas     1       No

    As the output shows, this set of configurations supports the following customizations by default:

    • http-port is the port on which the app listens.
    • image-tag is the tag of the container image to use (which is different from the version of the package).
    • replicas is the number of replicas that you want for the app.
  4. Update the image tag that is used in the configurations:

    kpt cfg set helloworld-set/ image-tag --values v0.1.0
  5. Change the number of replicas in the configurations:

    kpt cfg set helloworld-set/ replicas --values=1

    In the preceding steps, you updated some parameters that the package creator built in the package. This action is similar to what you can achieve with a configuration-as-code approach.

  6. Verify that the new values are correctly set:

    kpt cfg list-setters helloworld-set/

    The output looks like the following:

    NAME      VALUE        SET BY             DESCRIPTION        COUNT    REQUIRED   IS SET
    http-port   80       package-default   helloworld port         3       No         No
    image-tag   v0.1.0                     hello-world image tag   1       No         Yes
    replicas    1                          helloworld replicas     1       No         Yes
  7. Annotate the configurations. In this step, you use the sample-annotation: sample-value annotation.

    kpt fn run helloworld-set/ \ \
      -- annotation_name=sample-annotation \

    To see the new annotation, examine the helloworld-set/service.yaml and helloworld-set/deploy.yaml files.

    In this step, you customized the configurations in a way that the package creator didn't plan. You can make this customization because the configurations declaration is separated from the actions on those configurations. To achieve the same result with a configuration-as-code tool, you would have to edit the code.

  8. Validate the Kubernetes configuration files. In this step, you run kubeval through kpt.

    kpt fn run --network helloworld-set/ \

    This step is an example of the benefits of the Kubernetes resource model: because your configurations are represented in this well-known model, you can use existing Kubernetes tools like kubeval.

  9. Examine the differences between your local version of the configurations and the upstream configuration:

    kpt pkg diff helloworld-set/
  10. Initialize the package for deployment:

    kpt live init helloworld-set/

    The preceding command builds an inventory of the configurations that are in the package. Among other things, kpt uses the inventory to prune configurations when you remove them from the package. For more information, see kpt live.

  11. Apply the configurations to your GKE cluster:

    kpt live apply helloworld-set/

    The output looks like the following:

    service/helloworld-gke created
    deployment.apps/helloworld-gke created
    2 resource(s) applied. 2 created, 0 unchanged, 0 configured
    0 resource(s) pruned, 0 skipped
  12. Wait a few minutes, and then verify that everything is running as expected:

    kpt live status helloworld-set/

    The output looks like the following:

    service/helloworld-gke is Current: Service is ready
    deployment.apps/helloworld-gke is Current: Deployment is available. Replicas: 1

Updating your local package

In this section, you update your local version of the package with some changes from the upstream package.

  1. Create a Git repository for your configurations and configure your email and name. You need a local Git repository to be able to update the package. Replace YOUR_EMAIL with your email address and replace YOUR_NAME with your name.

    cd helloworld-set/
    git init
    git config "YOUR_EMAIL"
    git config "YOUR_NAME"
  2. Commit your configurations:

    git add .
    git commit -m "First version of helloworld"
  3. Update your local package. In this step, you pull the package-examples/helloworld-set/v0.5.0 version from upstream.

    kpt pkg update .@package-examples/helloworld-set/v0.5.0 \
      --strategy resource-merge
  4. Observe that updates from upstream are applied to your local package:

    git diff service.yaml

    In this case, the update changed the service type from LoadBalancer to NodePort.

  5. Commit your changes:

    git commit -am "Update to package-examples/helloworld-set/v0.5.0"
  6. Apply the new version of the package:

    kpt live apply .
  7. Observe that the service is now of type NodePort:

    kubectl get svc helloworld-gke

Removing a resource and a package

Because kpt tracks the resources that it creates, it can prune resources from the cluster when you delete resources from the package. It can also completely remove a package from the cluster. In this section, you remove a resource from the package, and then remove the package.

  1. Remove the service.yaml file from the package:

    git rm service.yaml
    git commit -m "remove service"
  2. Preview the changes that kpt will make:

    kpt live preview .

    The output looks like the following:

    deployment.apps/helloworld-gke configured (preview)
    1 resource(s) applied. 0 created, 0 unchanged, 1 configured (preview)
    service/helloworld-gke pruned (preview)
    1 resource(s) pruned, 0 skipped (preview)
  3. Apply the change, and then verify that kpt pruned the helloworld-gke service:

    kpt live apply .
    kubectl get svc
  4. Remove the rest of the package from the cluster, and then verify that you have nothing left in the cluster:

    kpt live destroy .
    kubectl get deployment

Using kpt to harden your GKE

The kpt live command is not the only way that you can apply a package to a Kubernetes cluster. In this section, you use kpt with Config Sync in a simple, but real scenario. The Config Sync tool lets you manage your configuration centrally, uniformly, and declaratively for all your Kubernetes clusters from a Git repository. Config Sync is available as a standalone product for GKE. Anthos (which you use in this tutorial) includes Anthos Config Management.

The Anthos security blueprints provide you with a range of prepackaged security settings for your Anthos-based workloads. In this section, you use the restricting traffic blueprint (and its directory in the GitHub repository). You use kpt to download the default-deny package. The package uses Kubernetes network policies to deny any traffic in your GKE cluster by default (except for DNS resolution). To apply the configurations, you then commit the configurations to the Config Sync Git repository.

Install Config Sync

In this section, you create the Git repository that Config Sync needs, and then you install Config Sync on your GKE cluster.

  1. In Cloud Shell, use Cloud Source Repositories to create a Git repository for Config Sync:

    cd ~
    gcloud source repos create config-management
  2. Generate an SSH key pair to authenticate against the Git repository:

    cd ~
    ssh-keygen -t rsa -b 4096  -N '' \
       -f cloud_source_repositories_key
  3. Create the Kubernetes Secret that contains the SSH private key to access the Git repository:

    kubectl create ns config-management-system && \
    kubectl create secret generic git-creds \
      --namespace=config-management-system \
  4. Display the public key, and then copy it:

  5. Go to the Cloud Source Repositories Manage SSH Keys page.

  6. In the Register SSH Key dialog that appears, enter the following values:

    1. In the Key name field, enter config-management.
    2. In the Key field, paste the public key.
    3. Click Register.
  7. Clone the Git repository to Cloud Shell:

    gcloud source repos clone config-management
    cd config-management
    git checkout -b main
  8. Download the Config Sync command-line tool called nomos:

    mkdir -p ~/bin
    gsutil cp gs://config-management-release/released/latest/linux_amd64/nomos ~/bin/nomos
    chmod +x ~/bin/nomos
    export PATH=${HOME}/bin:${PATH}
  9. Initialize the Config Sync repository:

    nomos init
    git add .
    git commit -m "Config Management directory structure"
    git push -u origin main
  10. Deploy the Config Sync operator:

    gsutil cp gs://config-management-release/released/latest/config-sync-operator.yaml /tmp/config-sync-operator.yaml
    kubectl apply -f /tmp/config-sync-operator.yaml

Configure Config Sync

  1. Create a ConfigManagement custom resource to configure Config Sync:

    PROJECT=$(gcloud config get-value project)
    EMAIL=$(gcloud config get-value account)
    cat <<EOF > /tmp/config-management.yaml
    kind: ConfigManagement
      name: config-management
      clusterName: kpt-tutorial
        syncRepo: ssh://${EMAIL}${PROJECT}/r/config-management
        syncBranch: main
        secretType: ssh
    kubectl -n config-management-system \
        apply -f /tmp/config-management.yaml

    For more installation options, see the Config Sync installation documentation.

  2. In Cloud Shell, verify that Config Sync is working properly:

    nomos status --contexts=$(kubectl config current-context)

    This command returns the status as SYNCED. Config Sync might take some time to initialize. If the status isn't updated, wait a few minutes and then rerun the command.

Apply the Anthos security blueprint

In this section, you use kpt to download the default-deny package of the restricting traffic Anthos security blueprint. Then you use Config Sync to apply the package to the default namespace only.

  1. Download the default-deny package:

    cd ~
    kpt pkg get ./

    You can explore the content of the package: the default-deny/Kptfile file contains the metadata of the package, and the default-deny/default-deny.yaml file contains a Kubernetes NetworkPolicy, which is the only configuration from this blueprint.

  2. Create the default namespace directory and the associated namespace.yaml file in the Config Sync repository:

    mkdir -p ~/config-management/namespaces/default
    cat >> ~/config-management/namespaces/default/namespace.yaml <<EOF
    apiVersion: v1
    kind: Namespace
      name: default

    The default namespace exists in your GKE cluster, but Config Sync doesn't manage it. When you create the directory and the file in this step, you make Config Sync manage the namespace. To apply the package to multiple namespaces at once, you can create an abstract namespace.

  3. Use kpt to copy the package's content in the Config Sync repository, and then add labels to customize it:

    kpt fn source default-deny/ | \
        kpt fn run -- \
        annotation_name=anthos-security-blueprint \
        annotation_value=restricting-traffic | \
        kpt fn sink ~/config-management/namespaces/default/

    As shown in this example, you can use pipes to chain kpt fn commands together. Chaining kpt fn commands lets you read configurations, modify them as you want, and write the result. You can chain as many kpt fn commands as you want.

  4. Verify that the Kubernetes NetworkPolicies were written to the Anthos Config Management repository, and that they are annotated with anthos-security-blueprint: restricting-traffic:

    cat config-management/namespaces/default/default-deny.yaml

    The output looks like the following:

    kind: NetworkPolicy
      name: default-deny
        anthos-security-blueprint: restricting-traffic
      - Ingress
      - Egress
      podSelector: {}
      - to:
        - namespaceSelector:
              k8s-namespace: kube-system
            - key: k8s-app
              operator: In
              - kube-dns
              - node-local-dns
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
  5. Commit and push the changes:

    cd ~/config-management/
    git add namespaces/default/
    git commit -m "Default deny"
    git push
  6. Verify that the new policy is applied:

    kubectl get networkpolicies

    If the new policy isn't present, wait a few seconds and then run the command again. Config Sync updates the configurations every 15 seconds by default. If you need to do some additional troubleshooting, run the following command to get information about any potential Config Sync errors:

    nomos status --contexts=$(kubectl config current-context)
  7. To test the new policy, get a shell in a pod running inside the default namespace:

    kubectl -n default run -i --tty --rm test \
            --image=busybox --restart=Never -- sh
  8. Try to ping, and see that it doesn't work, as expected:

    ping -c 3 -W 1

    The output looks like the following:

    PING ( 56 data bytes
    --- ping statistics ---
    3 packets transmitted, 0 packets received, 100% packet loss
  9. Try to query the Kubernetes API server, and see that it doesn't work, as expected:

    wget --timeout=3 https://${KUBERNETES_SERVICE_HOST}

    The output looks like the following:

    Connecting to (
    wget: download timed out
  10. Exit the pod:


Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Google Cloud project that you used in this tutorial, you can delete the Git repository and the GKE cluster. In Cloud Shell, run the following commands:

gcloud source repos delete config-management --quiet
gcloud container clusters delete kpt-tutorial \
    --async --quiet --zone=us-central1-a

What's next