Use Config Sync with Kustomize and Helm

---

In this tutorial, you add Kustomize configurations that reference Helm charts to your repository and then use Config Sync to sync your cluster to your repository.

When you use Config Sync, the Kustomize configurations and Helm charts you place in your Git repository are automatically rendered. Automated rendering provides you with the following benefits:

• You no longer need an external hydration pipeline. Without automated rendering, you have to manually render the configurations using Kustomize and Helm on your workstation, or set up a step to trigger the hydration process in your CI systems. With automated rendering, Config Sync handles the execution.

• Your maintenance costs are reduced. Without automated rendering, you have to maintain one Git repository with the original Kustomize configurations and Helm charts and another Git repository with the output generated by the external hydration. You then have to configure Config Sync to sync from the Git repository with the rendered output. With automated rendering, you only need to maintain one repository with the original configs.

• Your development workflow is simplified. Without automated rendering, changes made to your original configs need to be reviewed twice before being merged; once in the original repository and again in the rendered repository. With automated rendering, the rendered configs are generated by Config Sync, and you only need to review the changes to the original configs.

Note: Config Sync supports all of the native Kustomize configurations with the exception of custom Kustomize plugins. Config Sync does not support pulling Kustomize remote bases from private repositories. Kustomize plugins are not supported due to security concerns.

Objectives

• Configure your repository with Kustomize configurations that reference an off-the-shelf Helm chart for cert-manager. cert-manager is a tool for Kubernetes that helps you manage your certificates.
• Preview and validate the configs that you create.
• Use Config Sync to automatically render your chart and sync your cluster to your repository.
• Verify that your installation succeeded.

Costs

In this document, you use the following billable components of Google Cloud:

• Google Kubernetes Engine (GKE) Enterprise edition

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project.

3. Create, or ensure that you have access to, a cluster that meets the requirements for Config Sync and uses the following Config Sync settings:
   • Uses an unstructured repository.
   • RootSync and RepoSync APIs are enabled.
   • spec.override.enableShellInRendering is set to true to let Config Sync support pulling remote bases from public repositories in the rendering process.
4. Register your cluster to a fleet.
5. Install the nomos command-line tool. If you've already installed the nomos tool, make sure you upgrade it to version 1.9.0 or later.
6. Install Helm.

It's also helpful to have some familiarity with Git, Kustomize, and Helm.

Configure your repository

The following tasks show you how to prepare a Git repository with configs that combine Kustomize configurations with Helm charts:

1. Create, or make sure you have access to, a Git repository. Since your repository uses Kustomize and Helm, this should be an unstructured repository.

2. In the root of your Git repository, create a file named kustomization.yaml and paste the following code into it:

   # ./kustomization.yaml
   resources:
   - base
   
   patches:
   - path: ignore-deployment-mutation-patch.yaml
     target:
       kind: Deployment
   

   This file is a Kustomize overlay that points to the Kustomize base. This overlay includes a patch for the Helm chart base that adds the client.lifecycle.config.k8s.io/mutation: ignore annotation to all Deployment objects. The annotation causes Config Sync to ignore any conflicting changes to this object in the cluster after you have created it.

3. In your Git repository, create a directory named base:

   mkdir base
   

4. In the base directory, create another file named kustomization.yaml and paste the following code into it:

   # ./base/kustomization.yaml
   helmCharts:
   - name: cert-manager
     repo: https://charts.jetstack.io
     version: v1.5.3
     releaseName: my-cert-manager
     namespace: cert-manager
   

   This file is the Kustomize base, which renders the remote Helm chart.

5. Navigate back to the root of your Git repository, create a file named ignore-deployment-mutation-patch.yaml and paste the following code into it:

   # ./ignore-deployment-mutation-patch.yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
    name: any
    annotations:
      client.lifecycle.config.k8s.io/mutation: ignore
   

   This file is a patch that is applied to the base Helm chart. It adds the client.lifecycle.config.k8s.io/mutation: ignore annotation to all Deployments in the base directory.

6. Commit the changes to your repository:

   git add .
   git commit -m 'Set up manifests.'
   git push
   

The samples repository has an example of what such a repository would look like.

Preview and validate rendered configs

Before Config Sync renders the configs and syncs them to the cluster, ensure that the configs are accurate by running nomos hydrate to preview the rendered configuration and running nomos vet to validate that the format is correct.

1. Run the following nomos hydrate with the following flags:

   nomos hydrate \
       --source-format=unstructured \
       --output=OUTPUT_DIRECTORY
   

   In this command:

   • --source-format=unstructured lets nomos hydrate work on an unstructured repository. Since you are using Kustomize configurations and Helm charts, you need to use an unstructured repository and add this flag.
   • --output=OUTPUT_DIRECTORY lets you define a path to the rendered configs. Replace OUTPUT_DIRECTORY with the location that you want the output to be saved in.

2. Check the syntax and validity of your configs by running nomos vet with the following flags:

   nomos vet \
       --source-format=unstructured \
       --keep-output=true \
       --output=OUTPUT_DIRECTORY
   

   In this command:

   • --source-format=unstructured lets nomos vet work on an unstructured repository.
   • --keep-output=true saves the rendered configs.
   • --output=OUTPUT_DIRECTORY is the path to the rendered configs.

Configure syncing from the Git repository

Now that you have created a repository with the configs that you want to use, you can configure syncing from your cluster to your repository. If you have already installed Config Sync, proceed to Verify the sync status.

1. In the Google Cloud console, enable the GKE Hub API.

   Go to GKE Hub API

2. In the Google Cloud console, go to the Config page under the Features section.

   Go to Config

3. Click Install Config Sync.

4. Select Auto-upgrades to enable Config Sync to upgrade versions automatically.

5. Under Installation options, select Install Config Sync on individual clusters.

6. In the Available clusters table, select cs-cluster and click Install Config Sync. In the Settings tab, you should see the status for the cs-cluster as Enabled after a few minutes.

7. On the Config Sync dashboard, click Deploy Package.

8. In the Select clusters for package deployment table, select cs-cluster and then click Continue.

9. Leave Package hosted on Git selected and then click Continue.

10. In the Package name field, enter sample-repository.

11. In the Repository URL field, enter https://github.com/GoogleCloudPlatform/anthos-config-management-samples.

12. In the Path field, enter config-sync-quickstart/multirepo/root.

13. Leave all other fields with their default values.

14. Click Deploy Package.

    After a few minutes, you should see Synced in the Sync status column for cs-cluster.

Verify the installation

After you have installed and configured Config Sync, you can verify that the installation completed successfully.

1. Verify that there no other errors by using nomos status:

   nomos status
   

   Example output:

   *CLUSTER_NAME
   --------------------
   <root>   https:/github.com/GoogleCloudPlatform/anthos-config-management-samples.git/helm-component/manifests@init
   SYNCED   fd17dd5a
   

2. Verify if the Helm component is successfully installed:

   kubectl get all -n cert-manager
   

   Example output:

   NAME                                              READY   STATUS    RESTARTS   AGE
   pod/my-cert-manager-54f5ccf74-wfzs4               1/1     Running   0          10m
   pod/my-cert-manager-cainjector-574bc8678c-rh7mq   1/1     Running   0          10m
   pod/my-cert-manager-webhook-7454f4c77d-rkct8      1/1     Running   0          10m
   
   NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
   service/my-cert-manager           ClusterIP   10.76.9.35     <none>        9402/TCP   10m
   service/my-cert-manager-webhook   ClusterIP   10.76.11.205   <none>        443/TCP    10m
   
   NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/my-cert-manager              1/1     1            1           10m
   deployment.apps/my-cert-manager-cainjector   1/1     1            1           10m
   deployment.apps/my-cert-manager-webhook      1/1     1            1           10m
   
   NAME                                                    DESIRED   CURRENT   READY   AGE
   replicaset.apps/my-cert-manager-54f5ccf74               1         1         1       10m
   replicaset.apps/my-cert-manager-cainjector-574bc8678c   1         1         1       10m
   replicaset.apps/my-cert-manager-webhook-7454f4c77d      1         1         1       10m
   

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

Delete the manifests in your repository

In order to help prevent accidental deletion, Config Sync does not let you remove all namespaces or cluster-scoped resources in a single commit. Follow these instructions to gracefully uninstall the component and remove the namespace in separate commits:

1. Remove the cert-manager component from your repository:

   git rm -rf manifests/cert-manager \
       && git commit -m "uninstall cert-manager" \
       && git push origin BRANCH
   

   Replace BRANCH with the branch that you created your repository in.

2. Delete the cert-manager namespace:

   git rm manifests/namespace-cert-manager.yaml \
       && git commit -m "remove the cert-manager namespace" \
       && git push origin BRANCH
   

3. Verify that the cert-manager namespace does not exist:

   kubectl get namespace cert-namespace
   

   Example output:

   Error from server (NotFound): namespaces "cert-namespace" not found
   

Delete the cluster

To delete the cluster, complete the following commands:

Console

To delete a cluster using the Google Cloud console, complete the following tasks:

1. In the Google Cloud console, go to the GKE page.

   Go to GKE

2. Next to the cluster you want to delete, click Actions, then click Delete.

3. When prompted to confirm, click Delete again.

gcloud

To delete a cluster using the Google Cloud CLI, run the following command:

gcloud container clusters delete CLUSTER_NAME

For more information, refer to the gcloud container clusters delete documentation.

What's next

• Learn more about working with unstructured repositories.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.