Release notes

This page documents production updates to Config Sync. Check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, or browse and filter all release notes in the Google Cloud Console.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly:

April 05, 2021


The ability to sync from multiple Git repositories is now a generally-available feature. To learn more, see Syncing from multiple repositories.

Preview versions of multi-repo occasionally used excessive CPU usage and sent unnecessary queries to the apiserver master node, resulting in an unhealthy cluster. This issue has been corrected.

A memory leak in the Config Sync Operator Pod that led to high memory utilization or Pod restarts due to out-of-memory errors has been corrected.

Config Sync configured with sourceFormat: unstructured will have errors during syncing if the Git repository includes a "Repo" resource.

Config Sync configured with sourceFormat: unstructured will have errors during syncing if the Git repository specifies a ClusterSelector with an invalid field.

March 11, 2021


Config Sync multi-repo mode can't sync Git repositories using ssh as the authentication method. To workaround the issue, see Syncing from multiple repositories.

February 25, 2021


Hierarchy Controller now includes a preview of Hierarchical Resource Quotas (HRQs). HRQs are drop-in replacements for Kubernetes Resource Quotas, but apply to resources in both a namespace as well as all of its descendants. To learn more, see Using hierarchical resource quotas.

This release note was updated on March 10, 2021. The update removed information about a feature that is not yet available.

January 28, 2021


Hierarchy Controller is upgraded to include HNC v0.7.0. This release introduces Exceptions. Exceptions let you use Kubernetes label selectors to precisely control where certain objects are propagated.

This release also removes support for the v1alpha1 API. If you were using Hierarchy Controller 1.5.1 or earlier, you must either update to Hierarchy Controller 1.5.2 or 1.6.0, and follow the HNC v0.6.0 directions to upgrade to v1alpha2.

The nomos status output has been fixed for multi-repo clusters to show git.syncBranch when git.syncRev is not specified (git.syncRev defaults to HEAD) to provide a consistent experience with mono-repo clusters.

The nomos status output has been fixed for multi-repo clusters to distinctly show status of multiple namespace repos synced to the clusters.

January 22, 2021


Fixed an issue affecting the generation of Config Sync release notes.

January 08, 2021


Config Sync unintentionally started using the absolute path in the file system with spec.git.policyDir. This has no effect on Config Sync running on the cluster, but breaks validation when running nomos vet manually against hierarchical repositories. The issue will be corrected in 1.6.1.

December 10, 2020


Support for Git submodules has been fixed in this version.

November 16, 2020


A leading forward slash "/" in spec.git.policyDir will not match anything in Config Sync versions 1.5.2 and 1.6.0. For example:

policyDir: "/dirname"

will not match a top-level directory dirname. Because policyDir is always applied at the top of the directory structure, the workaround is to remove the leading forward slash.

This issue has been corrected in Config Sync 1.6.1; the forward slash will be ignored as it was in 1.5.1 and previous versions.

There is an issue where enabling referential constraints does not take effect for the audit container. This affects Config Sync versions 1.5.0, 1.5.1, and 1.5.2.

Hierarchy Controller is upgraded to include HNC v0.6.0. This release introduces support for v1alpha2, and will automatically update all your existing HNC objects. We recommend backing up these objects before upgrading in case there are any problems with the upgrade process. For more information, see the release notes for HNC v0.6.0.

October 29, 2020


Config Sync now includes the ability to sync from multiple Git repositories. This is a preview feature. To learn more, see Syncing from multiple repositories.

When the enableLegacyFields is set to true, the Config Sync operator will create a RootSync resource automatically, but any subsequent changes to the RootSync resource will not be noticed by the operator. This will be fixed in a subsequent release. As a workaround, if the RootSync resource resource is modified, add or modify an unused annotation on the ConfigManagement resource to cause the operator to reconcile changes in the RootSync resource.

The nomos status output has been modified significantly to provide a consistent experience for both mono-repo and multi-repo clusters.

September 24, 2020


The syncer and importer Containers now both run in the git-importer Pod in the importer Container.

The nomos CLI tool is now available via gcloud. Please see the downloads page for more information.

This release includes several logging and performance improvements.

August 27, 2020


This release includes several logging and performance improvements.

An issue with git submodule support is preventing syncing of configuration stored in submodule repositories. If this affects you, please contact support so we can suggest ways to handle your required use cases while we correct this.

July 30, 2020


Updated the git-sync image to fix security vulnerability CVE-2019-5482.

July 23, 2020


Config Sync now includes Hierarchy Controller as a beta feature. For more information on this component, see the Hierarchy Controller overview.

This release includes several logging and performance improvements.

This release includes several fixes and improvements for the nomos command line utility.

The use of unsecured HTTP for GitHub repo connections or in an http_proxy is now discouraged, and support for unsecured HTTP will be removed in a future release. HTTPS will continue to be supported for GitHub repo and local proxy connections.

This release improves the handling of GitHub repositories with very large histories.

Prior to this release, Config Sync and kubectl controllers and processes used the same annotation ( to calculate three-way merge patches. The shared annotation sometimes resulted in resource fights, causing unnecessary removal of each other's fields. Config Sync now uses its own annotation, which prevents resource clashes.

In most cases, this change will be transparent to you. However, there are two cases where some previously unspecified behavior will change.

The first case is when you have run kubectl apply on an unmanaged resource in a cluster, and you later add that same resource to the GitHub repo. Previously, Config Sync would have pruned any fields that were previously applied but not declared in the GitHub repo. Now, Config Sync writes the declared fields to the resource and leaves undeclared fields in place. If you want to remove those fields, do one of the following:

  • Get a local copy of the resource from GitHub and kubectl apply it.
  • Use kubectl edit --save-config to remove the fields directly.

The second case is when you stop managing a resource on the cluster or even stop all of Config Sync on a cluster. In this case, if you want to prune fields from a previously managed resource, you will see different behavior. Previously, you could get a local copy of the resource from GitHub, remove the unwanted fields, and kubectl apply it. Now, kubectl apply no longer prunes the missing fields. If you want to remove those fields, do one of the following:

  • Call kubectl apply set-last-applied with the unmodified resource from GitHub, then remove unwanted fields and kubectl apply it again without the set-last-applied flag.
  • Use kubectl edit --save-config to remove the fields directly.

In error messages, links to error docs are now more concise.

June 25, 2020


The nomos CLI tool now supports the KUBECONFIG environment variable in a way that matches the kubectl behavior with multiple delimited configuration files.

Config Sync no longer gets into a continuous PATCH loop when encountering unmanaged resources with config-management annotations and a missing last-applied-configuration annotation.

Config Sync is not issuing errors when it encounters certain types of malformed configurations in a resource definition. This may result in the Kubernetes API Server ignoring the malformed fields and applying the default value for the field instead.

This release includes several logging and performance improvements.

June 15, 2020


A regression in Config Sync 1.3.2 results in unnecessary patches to the API server for the gatekeeper-system namespace and spurious logging for error KNV2005. This "fight" results when the gatekeeper-system namespace is managed in the Git repo, and two Config Sync components (the operator and syncer) are both trying to reconcile the state of the namespace with the API server. The only workaround at this time is to unmanage the gatekeeper-system namespace. The issue will be fixed in Config Sync 1.4.1.

May 21, 2020


This release includes several performance and memory improvements.

In order to help prevent accidental deletion, Config Sync will no longer allow a user to remove all namespaces or cluster-scoped resources in a single commit. If you wish to delete the full set of resources under management, it now requires two steps: remove all but one in a first commit, allow Config Sync to sync those changes, then remove the final resource in a second commit.

Error documentation has been updated to add more information on error codes. Errors that are no longer encountered in the product have been removed. Most error references have been embellished with examples and steps for remediation.

Config Sync now supports a GKE-only authentication mechanism based on the service account of the cluster's node pool. Documentation on its use is here.

Config Sync will now attempt to detect when resources that it manages are also managed by other controllers. Documentation on this behavior is available in error knv2005 which Config Sync will log in that case.

April 23, 2020


Config Sync v1.3.1 now supports Kubernetes v1.16 and higher. Earlier versions of Config Sync relied on APIs that have been deprecated in Kubernetes v1.16.

The Config Sync Syncer Pod now reports when it detects that it is fighting with another process over a resource.

Config Sync no longer allows managing resources in unmanaged Namespaces.

If you define a CRD with an integer field that has min/max values, Config Sync will be unable to update the CRD.

Config Sync no longer overwrites undeclared labels and annotations on Namespaces.

March 24, 2020


Config Sync now supports the use of a Personal Access Tokens for authentication against supported Git providers. More information can be found in Installing Config Sync.

Config Sync now supports the use of an HTTP or HTTPS proxy to connect with your Git host. More information can be found at Installing Config Sync.

February 21, 2020


Minor updates and bug fixes.

February 10, 2020


Config Sync v1.2.1 is generally available for use in production.

Git repos with submodules are now also supported by Config Sync out of the box without additional configuration. This allows delegation of config authoring in a Git-idiomatic way. For more information, please see Git's documentation on submodules.

A new CLI subcommand is available. nomos bugreport bundles up Config Sync log information into a Zip file, which can be attached to a Google support ticket.

Previously, adding an APIService to the repo will leave Config Sync in a bad state, with the error message " KNV2002: failed to get server resources: unable to retrieve the complete list of server APIs." This issue has been mitigated; Config Sync will now sync APIService objects correctly.

Config Sync now can optionally support an unstructured repository, though some features that relied on hierarchical namespaces are disabled in this mode. More information can be found here.

December 20, 2019


Config Sync v1.2.0 is generally available for use in production. This is the first version of the product.

This release has minor bug fixes and performance improvements.

Adding an APIService to the repo will leave Config Sync in a bad state, with the error message KNV2002: failed to get server resources: unable to retrieve the complete list of server APIs. This issue will affect both new and existing clusters syncing from this repo. To correct the issue:

  • find the name of the git-importer and syncer pods using kubectl get pods -n config-management-system
  • copy those names and restart the pods with kubectl delete -n config-management-system pods git-importer-xxxx-xxxx syncer-xxxx-xxxx
  • These steps are required once for each cluster.

Once the pods for a cluster are restarted, syncing will be back to normal.

nomos view can fail to parse CRDs (Custom Resource Definitions) that exist in the local clone of the repo but have not yet been synced to a cluster.

To work around this issue, use nomos hydrate instead of nomos view.