Config Sync lets cluster operators manage Kubernetes resources by using files, called configs, stored in Git repositories.
This page shows you how to enable and configure Config Sync as part of Anthos Config Management. Follow these steps to install and configure Anthos Config Management in each cluster that you want to manage.
Before you begin
Before you install Config Sync, make sure that you have prepared your environment, clusters, and roles and enabled Anthos Config Management.
Preparing your local environment
Prepare your local environment by completing the following tasks:
Enable the Anthos API for your project:
Console
In the Google Cloud Console, go to the Anthos API page.
- Click Enable.
gcloud
Run the following command:
gcloud services enable anthos.googleapis.com
Install and initialize the Cloud SDK, which provides the
gcloud
,gsutil
,kubectl
, andnomos
commands used in these instructions. If you use Cloud Shell, the Cloud SDK comes pre-installed.kubectl
is not installed by default by the Cloud SDK. To installkubectl
, use the following command:gcloud components install kubectl
Use the
gcloud auth login
command to authenticate to Google Cloud so that you can download components of Anthos Config Management.
Preparing your cluster
Prepare your cluster by completing the following tasks:
Ensure that your cluster is on an Anthos supported platform and version.
Register your cluster to an Anthos environ by using Connect. Your project's environ provides a unified way to view and manage your clusters and their workloads as part of Anthos, including clusters outside Google Cloud. Anthos charges apply only to your registered clusters.
Preparing permissions
Depending on the installation method that you use, you need to grant the user that is installing Config Sync different permissions. To learn how to grant Identity and Access Management (IAM) roles, see Granting, changing, and revoking access to resources.
Console
The Google Cloud user installing Config Sync needs the
GKE Hub Admin
(roles/gkehub.admin
) role to administer your cluster through the
Hub API.
gcloud
The Google Cloud user installing Config Sync needs the
GKE Hub Admin
(roles/gkehub.admin
) role to administer your cluster through the
Hub API.
kubectl
If you are using GKE, the Google Cloud user installing Config Sync needs IAM permissions to create new roles in your cluster.
For example:
gcloud container clusters get-credentials CLUSTER_NAME kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin --user USER_ACCOUNT
Replace the following:
CLUSTER_NAME
: your cluster nameUSER_ACCOUNT
: your Google Cloud account's email address
Depending on how you configured the gcloud
command-line tool on your local system,
you might need to add the --project
and --zone
fields.
Enabling Anthos Config Management
If you are using Anthos Config Management for the first time, enable the feature by completing the following steps.
Console
In the Google Cloud Console, go to the Anthos Features page.
In the Config Management row, click Enable.
In the confirmation window, click Enable Config Management.
gcloud
Run the following command:
gcloud alpha container hub config-management enable
Installing Config Sync
In the following sections, you grant Config Sync access to your repository. After you have granted access, you configure the installation.
Granting Config Sync read-only access to Git
Config Sync needs read-only access to your Git repository so that it
can read the configs committed to the repository and apply them to your
clusters. If credentials are required, they are stored in the git-creds
Secret on each enrolled cluster.
If your repository does not require authentication for read-only access, you can
continue to configure Config Sync
and set secretType
to none
. For example, if you can browse the repository
using a web interface without logging in, or if you can use git clone
to
create a clone of the repository locally without providing credentials or using
saved credentials, then you do not need to authenticate. In this case, you do not
need to create a git-creds
Secret.
However, most users need to create credentials because read access to their repository is restricted. Config Sync supports the following mechanisms for authentication:
- SSH key pair
cookiefile
- Token
- Google Service Account (Google Source Repositories only)
The mechanism that you choose depends on what your repository supports. If all mechanisms are available, we recommend using an SSH key pair. GitHub, Cloud Source Repositories, and Bitbucket all support using an SSH key pair. If your organization hosts your repository and you don't know which authentication methods are supported, contact your administrator.
SSH key pair
An SSH key pair consists of two files, a public key and a private key. The
public key typically has a .pub
extension.
To use an SSH key pair, complete the following steps:
Create an SSH key pair to allow Config Sync to authenticate to your Git repository. This step is necessary if you need to authenticate to the repository to clone it or read from it. Skip this step if a security administrator provides you with a key pair. You can use a single key pair for all clusters, or a key pair per cluster, depending on your security and compliance requirements.
The following command creates a 4096-bit RSA key. Lower values are not recommended:
ssh-keygen -t rsa -b 4096 \ -C "GIT_REPOSITORY_USERNAME" \ -N '' \ -f /path/to/KEYPAIR_FILENAME
Replace the following:
GIT_REPOSITORY_USERNAME
: the username that you want Config Sync to use to authenticate to the repository/path/to/KEYPAIR_FILENAME
: a path to the key pair
If you are using a third-party Git repository host such as GitHub, or you want to use a service account with Cloud Source Repositories, we recommend that you use a separate account.
Configure your repository to recognize the newly created public key. Refer to the documentation for your Git hosting provider. Instructions for some popular Git hosting providers are included for convenience:
- Cloud Source Repositories
- Bitbucket
- GitHub. We recommend that you create separate deploy keys to provide read-only access to a single GitHub repository.
- Gitlab
Add the private key to a new Secret in the cluster:
kubectl create ns config-management-system && \ kubectl create secret generic git-creds \ --namespace=config-management-system \ --from-file=ssh=/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
Replace
/path/to/KEYPAIR_PRIVATE_KEY_FILENAME
with the name of the private key (the one without the.pub
suffix).Delete the private key from the local disk or otherwise protect it.
cookiefile
The process for acquiring a cookiefile
depends on the configuration of your
repository. For an example, see
Generate static credentials
in the Cloud Source Repositories documentation.
The credentials are usually stored in the .gitcookies
file in your home
directory, or they might be provided to you by a security administrator.
To use a cookiefile
, complete the following steps:
After you create and obtain the
cookiefile
, add it to a new Secret in the cluster.kubectl create ns config-management-system && \ kubectl create secret generic git-creds \ --namespace=config-management-system \ --from-file=cookie_file=/path/to/COOKIEFILE
Replace
/path/to/COOKIEFILE
with the appropriate path and filename.Protect the contents of the
cookiefile
if you still need it locally. Otherwise, delete it.
Token
If your organization does not permit the use of SSH keys, you might prefer to use a token. With Config Sync, you can use GitHub's personal access tokens (PATs) or Bitbucket's app password as your token.
To create a Secret using your token, complete the following steps:
Create a token using GitHub or Bitbucket:
GitHub: Create a PAT. Grant the token the
repo
scope so that it can read from private repositories. Because you bind a PAT to a GitHub account, we also recommend that you create a machine user and bind your PAT to the machine user.Bitbucket: Create an app password.
After you create and obtain the token, add it to a new Secret in the cluster:
kubectl create ns config-management-system && \ kubectl create secret generic git-creds \ --namespace="config-management-system" \ --from-literal=username=USERNAME \ --from-literal=token=TOKEN
Replace the following:
USERNAME
: the username that you want to useTOKEN
: the token that you created in the previous step
Protect the token if you still need it locally. Otherwise, delete it.
Google Service Account
If your repository is in Cloud Source Repositories,
you can use secretType: gcenode
to give Config Sync access to a
repository in the same project as your managed cluster.
Prerequisites
Before you begin, ensure that the following prerequisites are met:
Access scopes for the nodes in the cluster must include
cloud-source-repos-ro
. By default, the Compute Engine default service accountPROJECT_ID-compute@developer.gserviceaccount.com
hassource.reader
access to the repository for the same project, but if needed, you can add the role with the following command:gcloud projects add-iam-policy-binding PROJECT_ID \ --member serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \ --role roles/source.reader
Replace the following:
PROJECT_ID
: your organization's project IDPROJECT_NUMBER
: your organization's project number
Access scopes for the nodes in the cluster must include
cloud-source-repos-ro
. You can add this scope by includingcloud-source-repos-ro
in the--scopes
list specified at cluster creation time, or by using thecloud-platform
scope at cluster creation time:gcloud container clusters create example-cluster --scopes=cloud-platform
Using Cloud Source Repositories as your SyncRepo
After these prerequisites are met, set spec.git.syncRepo
for the URL of the
wanted repository in Cloud Source Repositories when you
configure Config Sync.
To use a repository in Cloud Source Repositories as your SyncRepo, complete the following steps:
List all repositories:
gcloud source repos list
From the output, copy the URL from the repository that you want to use:
REPO_NAME PROJECT_ID URL my-repo my-project https://source.developers.google.com/p/my-project/r/my-repo-csr
Add the URL as your
syncRepo
. For example:spec.git.syncRepo: https://source.developers.google.com/p/my-project/r/my-repo-csr
Using Cloud Source Repositories with Workload Identity
If Workload Identity is
enabled on your cluster, additional steps are required to use secretType:
gcenode
. After completing the preceding steps, and
configuring Config Sync (which
you do in the following section) create an
IAM policy binding
between the Kubernetes service account and the Google service account. The
Kubernetes service account is not created until you configure
Config Sync for the first time.
This binding allows the Config Sync Kubernetes service account to act as the Compute Engine default service account:
gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[config-management-system/importer]" \ PROJECT_NUMBER-compute@developer.gserviceaccount.com
Replace the following:
* PROJECT_ID
: your organization's project ID
* PROJECT_NUMBER
: your organization's project number
After you've created the binding, add an annotation
to the
Config Sync Kubernetes service account using the email address of
the Compute Engine default service account:
kubectl annotate serviceaccount -n config-management-system importer \ iam.gke.io/gcp-service-account=PROJECT_NUMBER-compute@developer.gserviceaccount.com
Replace PROJECT_NUMBER
with your organization's project number.
Configuring Config Sync
The Google Cloud Console guides you through the installation process and
automates many of the steps. You can also use the gcloud
command-line tool or kubectl
commands to complete the installation.
If you are using a GKE cluster, we recommend that you use the
Cloud Console to configure Config Sync. If you are using an
Anthos clusters on VMware, you should use kubectl
commands to install
Config Sync because private registries aren't supported in the
Cloud Console or the gcloud
tool.
To configure Config Sync, complete the following steps.
Console
In the Cloud Console, go to the Anthos Config Management page.
Select your registered clusters and click Configure.
In the Git Repository Authentication for ACM section, select one of the following options:
- None
- SSH
- Cookiefile
- Token
- Google Cloud Repository
If you have not done so already, grant Config Sync read-only access to Git.
Click Continue.
In the ACM settings for your clusters section, complete the following:
- In the Version field, select the version for Anthos Config Management.
- Select the Enable Config Sync checkbox.
- In the URL field, add the URL of the Git repository to use as the source of truth. This field is required.
- In the Branch field, add the branch of the repository to sync from. The default is the main (master) branch.
- In the Tag/Commit field, add the Git revision (tag or
hash) to check out. The default is
HEAD
. - In the Policy directory field, add the path within the repository to the top of the policy hierarchy to sync. The default is the root directory of the repository.
- In the Sync wait field, add the period in seconds between consecutive syncs. The default is 15 seconds.
- In the Git proxy field, enter the URL for the HTTPS proxy to be used when communicating with the Git repository. This is an optional field, and if it's left blank, no proxy is used.
- In the Source format field, choose either hierarchy (default) or unstructured (recommended). We recommend that you select unstructured as this format lets you organize your configs in the way that is most convenient to you.
Click Done. You are taken back to the Anthos Config Management page. After a few minutes, you should see
Synced
in the status column next to the clusters that you configured. If you seeError
, click on the wordError
for more information.
gcloud
Create a file named
config-management.yaml
and copy the following YAML file into it:# config-management.yaml apiVersion: configmanagement.gke.io/v1 kind: ConfigManagement metadata: name: config-management spec: sourceFormat: FORMAT git: syncRepo: REPO syncBranch: BRANCH secretType: TYPE policyDir: "DIRECTORY"
Replace the following:
FORMAT
: addunstructured
to use an unstructured repository or addhierarchy
to use a hierarchical repository. These values are case-sensitive. This field is optional and the default is value ishierarchy
. We recommend that you addunstructured
as this format lets you organize your configs in the way that is most convenient to you.REPO
: the URL of the Git repository to use as the source of truth. This field is required.BRANCH
: the branch of the repository to sync from. The default is the main (master) branch.TYPE
: one of the followingSecretTypes
:none
ssh
cookiefile
token
gcenode
DIRECTORY
: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.For a complete list of fields that you can add to the
spec
field, see ConfigManagement fields.
Apply the
config-management.yaml
file:gcloud alpha container hub config-management apply \ --membership=CLUSTER_NAME \ --config=CONFIG_YAML_PATH \ --project=PROJECT_ID
Replace the following:
CLUSTER_NAME
: the name of the registered cluster that you want to apply this configuration toCONFIG_YAML_PATH
: the path to yourconfig-management.yaml
filePROJECT_ID
: your project ID
kubectl
When you install Config Sync with kubectl
, you need to deploy the
Operator before you can configure it. The Operator
is a controller that manages Config Sync in a Kubernetes cluster.
Deploying the Operator
After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.
Download the latest version of the Operator CustomResourceDefinition (CRD) by using the following command. To download a specific version instead, see Downloads.
gsutil cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yaml
Apply the CRD:
kubectl apply -f config-management-operator.yaml
If this command fails, see Troubleshooting.
Configuring Anthos Config Management
To configure the behavior of Anthos Config Management, create a
configuration file for the ConfigManagement
CustomResource,
and then apply it by using the kubectl apply
command:
Create a file named
config-management.yaml
and copy the following YAML file into it.# config-management.yaml apiVersion: configmanagement.gke.io/v1 kind: ConfigManagement metadata: name: config-management spec: # clusterName is required and must be unique among all managed clusters clusterName: CLUSTER_NAME sourceFormat: FORMAT git: syncRepo: REPO syncBranch: BRANCH secretType: TYPE policyDir: "DIRECTORY"
Replace the following:
CLUSTER_NAME
: the name of the registered cluster that you want to apply this configuration toFORMAT
: addunstructured
to use an unstructured repository or addhierarchy
to use a hierarchical repository. These values are case-sensitive. This field is optional and the default is value ishierarchy
. We recommend that you addunstructured
as this format lets you organize your configs in the way that is most convenient to you.REPO
: the URL of the Git repository to use as the source of truth. This field is required.BRANCH
: the branch of the repository to sync from. The default is the main (master) branch.TYPE
: one of the followingSecretTypes
:none
ssh
cookiefile
token
gcenode
DIRECTORY
: the path in the Git repository to the root directory that contains the configuration that you want to sync to. The default is the root directory of the repository.
For a complete list of fields that you can add to the
spec
field, see ConfigManagement fields.Apply the configuration with the
kubectl apply
command:kubectl apply -f config-management.yaml
You might need to create separate configuration files for each cluster or each type of cluster. You can specify the cluster using the
--context
option:kubectl apply -f config-management1.yaml --context=CLUSTER_NAME
Replace
CLUSTER_NAME
with the name of the cluster that you want to use.
Verifying the installation
After you have installed and configured Config Sync, you can verify that the installation completed successfully.
Console
In the Cloud Console, go to the Anthos Config Management page.
View the Status column. A successful installation has a status of
Synced
.
For a detailed view of your cluster status:
- Select the cluster that you want to explore. The Cluster details page appears. On this page, you can see details for your cluster and details for your Config Sync installation.
gcloud
Run the following command:
gcloud alpha container hub config-management status \
--project=PROJECT_ID
Replace PROJECT_ID
with your project's ID.
A successful installation has a status of SYNCED
. If you see an error after
running the preceding command, make sure that you created the git-creds
Secret. If you have created the Secret, try rerunning the following command:
gcloud alpha container hub config-management apply
kubectl
To check whether Config Sync is installed successfully, you can use
the nomos status
command. A valid installation with no problems has a status of
PENDING
or SYNCED
. An invalid or incomplete installation has a status of
NOT INSTALLED
or NOT CONFIGURED
. The output also includes any reported
errors.
When Config Sync is deployed successfully, it runs in a Pod
whose name begins with config-management-operator
in the kube-system
namespace. The Pod might take a few moments to initialize.
Verify that the Pod is running:
kubectl -n kube-system get pods | grep config-management
If the Pod is running, the command's response is similar to the following example:
config-management-operator-6f988f5fdd-4r7tr 1/1 Running 0 26s
You can also verify that the config-management-system
namespace exists:
kubectl get ns | grep 'config-management-system'
The command's output is similar to the following example:
config-management-system Active 1m
If the commands don't return output similar to the preceding example, view the logs to see what went wrong:
kubectl -n kube-system logs -l k8s-app=config-management-operator
You can also use kubectl get events
to check whether Config Sync
has created any events.
kubectl get events -n kube-system
It's possible to have an invalid configuration that isn't detected right away,
such as a missing or invalid git-creds
Secret. For troubleshooting steps, see
Valid but incorrect ConfigManagement object
in the Troubleshooting section of this page.
Upgrading Config Sync versions
Config Sync is upgraded whenever you upgrade Anthos Config Management.
To upgrade Anthos Config Management, you can use the Google Cloud Console or
kubectl
. Before upgrading, check the release notes
for any specific instructions.
Console
In the Cloud Console, go to the Anthos Config Management page.
Select the clusters that you want to upgrade.
Click Configure.
Click ACM settings for your clusters.
From the Version dropdown, select the version that you want to upgrade to.
Click Done.
kubectl
Run these commands for each enrolled cluster:
Download the Anthos Config Management manifest and
nomos
commands for the new version.Apply the Anthos Config Management manifest:
kubectl apply -f config-management-operator.yaml
This command updates the Anthos Config Management image. Kubernetes retrieves the new version and restarts the Anthos Config Management Pod using the new version. When Anthos Config Management starts, it runs a reconcile loop that applies the set of manifests bundled in the new image. This updates and restarts each component Pod.
Replace the
nomos
ornomos.exe
command on all clients with the new version. This change ensures that thenomos
command can always get the status of all enrolled clusters and can validate configs for them.
Uninstalling Config Sync from a cluster
Follow these instructions to uninstall Config Sync from a cluster. You must follow these steps for each cluster that you no longer want to manage with Config Sync.
Console
To disable Anthos Config Management, complete the following steps:
In the Google Cloud Console, go to the Anthos Features page.
In the Config Management row of the Features table, click Details. The Status summary page appears.
Click Disable config management. A confirmation page appears.
On the confirmation page, click Disable config management.
gcloud
Run the following command:
gcloud alpha container hub config-management delete \
--project=PROJECT_ID \
--membership=CLUSTER_NAME
Replace the following:
CLUSTER_NAME
: the name of the registered cluster that you want to remove this configuration fromPROJECT_ID
: your project ID
To delete all Config Sync configurations and statuses, run the following command:
gcloud alpha container hub config-management disable \
--project=PROJECT_ID
Replace PROJECT_ID
with your project ID.
kubectl
Delete the ConfigManagement object from the cluster:
kubectl delete configmanagement --all
The following things happen:
- Any ClusterRoles and ClusterRoleBindings created in the cluster by Config Sync are deleted from the cluster.
- Any admission controller configurations installed by Config Sync are deleted.
- The contents of the
config-management-system
namespace are deleted, with the exception of thegit-creds
Secret. Config Sync cannot function without theconfig-management-system
namespace. Any CustomResourceDefinitions (CRDs) created or modified by Config Sync are removed from the clusters where they were created or modified. The CRD required to run Config Sync still exists because from the point of view of Kubernetes, they were added by the user who installed Config Sync. Information about removing these components is covered in the next step.
At this point, the Operator still exists in your cluster, but does nothing. If you are using Anthos clusters on VMware, you cannot remove the Operator manually.
If you are using GKE and decide that you no longer want to use Config Sync at all, you can uninstall Config Sync by following these steps:
Verify that the
config-management-system
namespace is empty after deleting the ConfigManagement object in the previous step. Wait until thekubectl -n config-management-system get all
command returnsNo resources found.
Delete the
config-management-system
namespace:kubectl delete ns config-management-system
Delete the ConfigManagement CustomResourceDefinition:
kubectl delete crd configmanagements.configmanagement.gke.io
Delete all Config Sync objects from the
kube-system
namespace:kubectl -n kube-system delete all -l k8s-app=config-management-operator
Troubleshooting
The following sections help you troubleshoot your Config Sync installation.
Using Audit Logs
Audit Logs can be a useful debugging tool.
If you installed Config Sync using the Cloud Console or the
gcloud
command-line tool, complete the following steps to use Audit Logs to
investigate Config Sync.
Console
Enable the GKE Connect/Hub APIs Audit Logs.
In the Cloud Console, go to the IAM Audit Logs page.
In the table, select the GKE Connect/Hub APIs checkbox.
Select the following checkboxes:
- Admin Read
- Data Read
- Data Write
Click Save.
Go to the Logs Explorer page.
In the Query builder text box, add the following filters:
resource.type="audited_resource" resource.labels.service="gkehub.googleapis.com"
Click Run Query.
In the Queries results section, select entries to learn more about the events.
Insufficient CPU
The output of kubectl get events
might include an event with the type
FailedScheduling
. The event looks like the following example:
LAST SEEN TYPE REASON OBJECT MESSAGE
9s Warning FailedScheduling pod/config-management-operator-74594dc8f6 0/1 nodes are available: 1 Insufficient cpu.
To fix this error, choose one of the following options:
- Add a node to an existing GKE node pool.
- Create a node pool with larger nodes.
Error: attempt to grant extra privileges
If you receive the following error when you try to apply the
config-management-operator.yaml
file, you might have fewer permissions than
are required for the installation:
Error from server (Forbidden): error when creating "config-management-operator.yaml": clusterroles.rbac.authorization.k8s.io "config-management-operator" is forbidden: attempt to grant extra privileges: [...] ruleResolutionErrors=[]
To ensure that you have the required permissions, review Preparing permissions.
Valid but incorrect ConfigManagement object
If installation fails due to a problem with the ConfigManagement object that is
not due to a YAML or JSON syntax error, the ConfigManagement object might be
instantiated in the cluster, but might not work correctly. In this situation, you
can use the nomos status
command to check for errors in the ConfigManagement
object.
A valid installation with no problems has a status of PENDING
or SYNCED
.
An invalid installation has a status of NOT CONFIGURED
and lists one of the
following errors:
missing git-creds Secret
missing required syncRepo field
git-creds Secret is missing the key specified by secretType
To fix the problem, correct the configuration error. Depending on the type of error, you might need to re-apply the ConfigManagement manifest to the cluster.
If the problem is that you forgot to create the git-creds
Secret,
Config Sync detects the Secret as soon as you create it, and you do
not need to re-apply the configuration.
What's next
- Learn more about the
gcloud
commands for configuring Config Sync with Anthos Config Management. - Learn about creating configs.
- Use the
nomos
command.