Sync to a hierarchical repository

---

This tutorial shows you how to use a Config Sync hierarchical root repository to manage the configuration of a Kubernetes cluster shared by two different teams, team-1 and team-2.

Objectives

• Learn about best practices for using a hierarchical repository.
• Sync a cluster to the example hierarchical repository.

Costs

In this document, you use the following billable components of Google Cloud:

• GKE

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

3. Have access to a cluster with Config Sync already installed. If you don't have such a cluster, follow the instructions in the "Before you begin" and "Preparing your environment" sections of the Syncing from Git repositories tutorial

   1. Configure kubectl command line access by running the following command:

   gcloud container clusters get-credentials CLUSTER_NAME \
       --zone ZONE \
       --project PROJECT_ID
   

   Replace the following:

   • CLUSTER_NAME: the name of the registered cluster that you want to apply this configuration to
   • ZONE: the zone that you created your cluster in
   • PROJECT_ID: your project ID

Explore the repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the config/ directory of the hierarchical-format/ repository. The config/ directory contains the following directories and files:

├── cluster
│   ├── clusterrolebinding-namespace-reader.yaml
│   ├── clusterrole-namespace-reader.yaml
│   ├── clusterrole-secret-admin.yaml
│   ├── clusterrole-secret-reader.yaml
│   └── crontab-crd.yaml
├── namespaces
│   ├── limit-range.yaml
│   ├── team-1
│   │   ├── crontab.yaml
│   │   ├── namespace.yaml
│   │   ├── network-policy-default-deny-egress.yaml
│   │   ├── resource-quota-pvc.yaml
│   │   ├── rolebinding-secret-reader.yaml
│   │   └── sa.yaml
│   └── team-2
│       ├── crontab.yaml
│       ├── namespace.yaml
│       ├── network-policy-default-deny-all.yaml
│       ├── resource-quota-pvc.yaml
│       ├── rolebinding-secret-admin.yaml
│       └── sa.yaml
├── README.md
└── system
    └── repo.yaml

A valid Config Sync hierarchical root repository must include three subdirectories: cluster/,namespaces/, and system/.

The cluster/ directory contains configs that apply to entire clusters (such as CRDs, ClusterRoles, and ClusterRoleBindings), rather than to namespaces.

The namespaces/ directory contains configs for the namespace objects and the namespace-scoped objects. Each subdirectory under namespaces/ includes the configs for a namespace object and all the namespace-scoped objects under the namespace. The name of a subdirectory should be the same as the name of the namespace object. Namespace-scoped objects which need to be created in each namespace, can be put directly under namespaces/ (for example, namespaces/limit- range.yaml).

In this tutorial, each team has its own Kubernetes namespace, Kubernetes service account, resource quotas, network policies, rolebindings. The cluster admin sets up a policy in namespaces/limit-range.yaml to constrain resource allocations (to Pods or Containers) in both namespaces. The cluster admin also sets up ClusterRoles and ClusterRoleBinidngs.

The system/ directory contains configs for the Config Sync Operator.

---

The compiled/ directory (which is not required for using Config Sync) contains the output of nomos hydrate, which compiles the configs under the cluster/, namespaces/, system/ directories to the exact form that would be sent to the APIServer to apply. The cluster-scoped resources are under this directory directly. Each subdirectory includes all the configs for the resources under a namespace. The compiled/ directory contains the following directories and files:

.
├── clusterrolebinding_namespace-reader.yaml
├── clusterrole_namespace-reader.yaml
├── clusterrole_secret-admin.yaml
├── clusterrole_secret-reader.yaml
├── customresourcedefinition_crontabs.stable.example.com.yaml
├── namespace_team-1.yaml
├── namespace_team-2.yaml
├── team-1
│   ├── crontab_my-new-cron-object.yaml
│   ├── limitrange_limits.yaml
│   ├── networkpolicy_default-deny-egress.yaml
│   ├── resourcequota_pvc.yaml
│   ├── rolebinding_secret-reader.yaml
│   └── serviceaccount_sa.yaml
└── team-2
    ├── crontab_my-new-cron-object.yaml
    ├── limitrange_limits.yaml
    ├── networkpolicy_default-deny-all.yaml
    ├── resourcequota_pvc.yaml
    ├── rolebinding_secret-admin.yaml
    └── serviceaccount_sa.yaml

Sync your cluster to the root repository using Config Sync

In this section, you sync your cluster to the hierarchical repository using Config Sync and Google Cloud CLI.

1. Create a file named apply-spec.yaml and paste the following text into it:

   # apply-spec.yaml
   
   applySpecVersion: 1
   spec:
     configSync:
       enabled: true
       sourceFormat: hierarchy
       syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples/
       syncBranch: init
       secretType: none
       policyDir: hierarchical-format/config
   

2. Apply the apply-spec.yaml file using the Google Cloud CLI:

    gcloud alpha container fleet config-management apply \
        --membership=CLUSTER_NAME \
        --config=CONFIG_YAML_PATH \
        --project=PROJECT_ID
   

   Replace the following:

   • CLUSTER_NAME: the name of the registered cluster that you want to apply this configuration to
   • CONFIG_YAML_PATH: the path to your apply-spec.yaml file
   • PROJECT_ID: your project ID

3. Check if Config Sync successfully syncs all configs to your cluster:

   gcloud alpha container fleet config-management status
       --project=PROJECT_ID
   

   Example output:

   Name          Status  Last_Synced_Token  Sync_Branch  Last_Synced_Time      Policy_Controller                          Hierarchy_Controller
   CLUSTER_NAME  SYNCED  6bfc9be            init         2021-06-08T17:26:32Z  GatekeeperControllerManager NOT_INSTALLED  PENDING
   

   A successful installation has a status of SYNCED.

Examine your configs

The config/ directory includes the following resources:

• ClusterRoles
• ClusterRoleBindings
• CRDs
• Namespaces
• RoleBindings
• ServiceAccounts
• ResourceQuotas
• NetworkPolicies
• LimitRanges
• CRs

These configs are applied as soon as the Config Sync is configured to read from the repository. In this section, you check to see if Config Sync is managing the namespaces, CRDs,and rolebindings in the directory.

All objects managed by Config Sync have the app.kubernetes.io/managed-by label set to configmanagement.gke.io and you can use this label to query your resources.

1. List namespaces managed by Config Sync:

   kubectl get ns -l app.kubernetes.io/managed-by=configmanagement.gke.io
   

   Example output:

   NAME        STATUS   AGE
   team-1      Active   28m
   team-2      Active   28m
   

2. List CRDs managed by Config Sync:

   kubectl get crds -A -l app.kubernetes.io/managed-by=configmanagement.gke.io
   

   Example output:

   NAME                          CREATED AT
   crontabs.stable.example.com   2021-05-04T14:58:14Z
   

3. List rolebindings managed by Config Sync:

   kubectl get rolebindings -A -l app.kubernetes.io/managed-by=configmanagement.gke.io
   

   Example output:

   NAMESPACE   NAME                            ROLE                        AGE
   team-1      secret-reader                   ClusterRole/secret-reader   29m
   team-2      secret-admin                    ClusterRole/secret-admin    29m
   

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

To stop Config Sync from managing your cluster, run the following command:

gcloud alpha container fleet config-management unmanage \
    --project=PROJECT_ID \
    --membership=CLUSTER_NAME

To delete your cluster, run the following command:

gcloud container clusters delete CLUSTER_NAME

What's next

• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.