Add configs to a source of truth

This page explains how to add and organize configurations stored in a source of truth.

About configs

Config Sync is designed for cluster operators who manage many clusters. You can ensure that your clusters meet business and compliance standards by letting Config Sync manage namespaces, Roles, RoleBindings, ResourceQuotas, and other important Kubernetes objects, across your fleet.

When Config Sync manages these resources, it keeps your enrolled clusters in sync using configs. A config is a YAML or JSON file that is stored in a source of truth. Config Sync supports Git repositories, OCI images, and Helm charts as the source of truth. Configs contain the same type of configuration details that you can manually apply to a cluster using the kubectl apply command. You can create a config for any Kubernetes object that can exist in a cluster. However, some Kubernetes objects, such as Secrets, contain sensitive information that might be inappropriate to store in a source of truth. Use your judgment when considering whether to manage these types of objects using Config Sync.

You can also use Config Sync with Config Connector to sync configs for Google Cloud resources. To learn more about working with Config Connector, see managing Google Cloud resources using Config Connector. You can also simplify the installation of Config Sync and Config Connector by setting up Config Controller.

Limitations

  • Some Kubernetes resources contain immutable fields, like Pod selectors in a Deployment object. You can't change any immutable field in a config by changing the value in the source of truth. Attempting such a change causes a KNV2009 error. If you need to change an immutable field, delete the object from your source of truth, wait for Config Sync to delete the object from the cluster, and then re-create the object in your source of truth with your updates.

  • If you use Git submodules, you must grant Config Sync access to all repositories, including any submodules.

  • You can't use Config Sync to directly manage built-in Kubernetes ClusterRoles. GKE comes with some built-in user facing roles, such as cluster-admin, admin, edit, and view. You can't directly manage these ClusterRoles with Config Sync, otherwise Config Sync fights with GKE. To add permissions to the built-in ClusterRoles, use role aggregation to modify them indirectly. To modify the roles, create a uniquely named ClusterRole in your source of truth with the appropriate annotations.

Select how to organize your configs

Config Sync uses a source of truth for config storage and version control. There are two different formats you can choose for your source of truth; unstructured and hierarchical.

The unstructured source format lets you organize configs in whatever way is most convenient. This format can be especially useful if you are organizing or generating configs using a tool such as Kustomize, kpt, or Helm. For an example of how you might organize your configs, see Example format for an unstructured repository.

The hierarchical, or structured, source format separates configs into distinct categories to help you organize the configs. The categories are system configuration, cluster metadata, cluster-level configuration, and namespace configuration. For more information about the hierarchical source format, see Structure of the hierarchical repo.

Unstructured is the recommended format for most users. In addition, when you configure RepoSync objects, you must use the unstructured source format.

Supported features for unstructured and hierarchical formats

The following table highlights the differences between the unstructured and hierarchical formats:

Features Unstructured format (recommended) Hierarchical format
Used as the format for a RootSync object or the central source of truth Supported Supported
Used as the format for a RepoSync object Supported Not supported
ClusterSelector Supported Supported
NamespaceSelector Supported Supported
The nomos hydrate command Supported with the --source-format=unstructured flag Supported
The nomos init command Not supported Supported
The nomos vet command Supported with the --source-format=unstructured flag Supported
All other nomos commands Supported Supported
Abstract namespaces Not supported Supported
Repo objects Not supported Supported
HierarchyConfig objects Not supported Supported

When to add configs to the source

If you are creating an unstructured format, you can start adding configs to it as soon as it's created. If you are creating a hierarchical format, use the nomos init command to initialize the source of truth, or create the directory structure manually.

Empty directories cannot be committed to a Git repository, so before you configure Config Sync, you must create configs and add them into your repository.

After you have created the source of truth and added configs to it, use the nomos vet command to verify the structure of your source of truth and check the syntax and validity of your configs.

Configure Config Sync to read from the source of truth

After you've created a source of truth and placed your configs into it, you can configure Config Sync to read from the source. After you've completed this step, Config Sync syncs configs from your source of truth to your clusters.

You configure the location of the source of truth when you install Config Sync, and you can edit Config Sync's configuration later. In addition to the location of the source of truth, you can specify a branch or subdirectory to watch, if the source has contents other than configs.

If you are using a hierarchical format, and installing Config Sync manually with kubectl, don't place the Operator's config in the system/ directory, or any of the other reserved directories such as cluster/ or namespaces/. Placing the config in one of the reserved directories causes nomos vet to fail and logs an error such as KNV1033: IllegalSystemResourcePlacementError, KNV1038: IllegalKindInNamespacesError, or KNV1039: IllegalKindInClusterError.

You can grant people access to a given product team's deployment source of truth. However, when you grant a person access to a deployment source of truth, that person is also granted the same RBAC as the reconciler running for that source of truth.

To configure authentication and authorization between Config Sync and the source of truth, see the installation step about configuring the git-creds Secret.

Ignore object mutations

If you don't want Config Sync to maintain the state of the object in the cluster after it exists, you can add the client.lifecycle.config.k8s.io/mutation: ignore annotation to the object that you want Config Sync to ignore mutations in.

To use the annotation, you must enable the RootSync and RepoSync APIs.

The following example shows you how to add the annotation to an object:

metadata:
  annotations:
    client.lifecycle.config.k8s.io/mutation: ignore

You cannot manually modify this annotation on managed objects in the cluster.

What's next