Syncing to a hierarchical repository

This tutorial shows you how to use a Config Sync hierarchical root repository to manage the configuration of a Kubernetes cluster shared by two different teams, team-1 and team-2.

Objectives

  • Learn about best practices for using a hierarchical repository.
  • Sync a cluster to the example hierarchical repository.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

New Google Cloud users might be eligible for a free trial.

Before you begin

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  3. Have access to a cluster with Config Sync already installed. If you don't have such a cluster, follow the instructions in the "Before you begin" and "Preparing your environment" sections of the quickstart for Anthos or the quickstart for GKE.
  4. Configure kubectl command line access by running the following command:

    gcloud container clusters get-credentials CLUSTER_NAME \
        --zone ZONE \
        --project PROJECT_ID
    

    Replace the following:

    • CLUSTER_NAME: the name of the registered cluster that you want to apply this configuration to
    • ZONE: the zone that you created your cluster in
    • PROJECT_ID: your project ID

Exploring the repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the config/ directory of the hierarchical-format/ repository. The config/ directory contains the following directories and files:

├── cluster
│   ├── clusterrolebinding-namespace-reader.yaml
│   ├── clusterrole-namespace-reader.yaml
│   ├── clusterrole-secret-admin.yaml
│   ├── clusterrole-secret-reader.yaml
│   └── crontab-crd.yaml
├── namespaces
│   ├── limit-range.yaml
│   ├── team-1
│   │   ├── crontab.yaml
│   │   ├── namespace.yaml
│   │   ├── network-policy-default-deny-egress.yaml
│   │   ├── resource-quota-pvc.yaml
│   │   ├── rolebinding-secret-reader.yaml
│   │   └── sa.yaml
│   └── team-2
│       ├── crontab.yaml
│       ├── namespace.yaml
│       ├── network-policy-default-deny-all.yaml
│       ├── resource-quota-pvc.yaml
│       ├── rolebinding-secret-admin.yaml
│       └── sa.yaml
├── README.md
└── system
    └── repo.yaml

A valid Config Sync hierarchical root repository must include three subdirectories: cluster/,namespaces/, and system/.

The cluster/ directory contains configs that apply to entire clusters (such as CRDs, ClusterRoles, and ClusterRoleBindings), rather than to namespaces.

The namespaces/ directory contains configs for the namespace objects and the namespace-scoped objects. Each subdirectory under namespaces/ includes the configs for a namespace object and all the namespace-scoped objects under the namespace. The name of a subdirectory should be the same as the name of the namespace object. Namespace-scoped objects which need to be created in each namespace, can be put directly under namespaces/ (for example, namespaces/limit- range.yaml).

In this tutorial, each team has its own Kubernetes namespace, Kubernetes service account, resource quotas, network policies, rolebindings. The cluster admin sets up a policy in namespaces/limit-range.yaml to constrain resource allocations (to Pods or Containers) in both namespaces. The cluster admin also sets up ClusterRoles and ClusterRoleBinidngs.

The system/ directory contains configs for the Config Sync Operator.


The compiled/ directory (which is not required for using Config Sync) contains the output of nomos hydrate, which compiles the configs under the cluster/, namespaces/, system/ directories to the exact form that would be sent to the APIServer to apply. The cluster-scoped resources are under this directory directly. Each subdirectory includes all the configs for the resources under a namespace. The compiled/ directory contains the following directories and files:

.
├── clusterrolebinding_namespace-reader.yaml
├── clusterrole_namespace-reader.yaml
├── clusterrole_secret-admin.yaml
├── clusterrole_secret-reader.yaml
├── customresourcedefinition_crontabs.stable.example.com.yaml
├── namespace_team-1.yaml
├── namespace_team-2.yaml
├── team-1
│   ├── crontab_my-new-cron-object.yaml
│   ├── limitrange_limits.yaml
│   ├── networkpolicy_default-deny-egress.yaml
│   ├── resourcequota_pvc.yaml
│   ├── rolebinding_secret-reader.yaml
│   └── serviceaccount_sa.yaml
└── team-2
    ├── crontab_my-new-cron-object.yaml
    ├── limitrange_limits.yaml
    ├── networkpolicy_default-deny-all.yaml
    ├── resourcequota_pvc.yaml
    ├── rolebinding_secret-admin.yaml
    └── serviceaccount_sa.yaml

Syncing your cluster to the root repository using Config Sync

In this section, you sync your cluster to the hierarchical repository using Config Sync and gcloud command-line tool.

  1. Create a file named apply-spec.yaml and paste the following text into it:

    # apply-spec.yaml
    
    applySpecVersion: 1
    spec:
      configSync:
        enabled: true
        sourceFormat: hierarchy
        syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples/
        syncBranch: init
        secretType: none
        policyDir: hierarchical-format/config
    
  2. Apply the apply-spec.yaml file using the gcloud command-line tool:

     gcloud alpha container hub config-management apply \
         --membership=CLUSTER_NAME \
         --config=CONFIG_YAML_PATH \
         --project=PROJECT_ID
    

    Replace the following:

    • CLUSTER_NAME: the name of the registered cluster that you want to apply this configuration to
    • CONFIG_YAML_PATH: the path to your apply-spec.yaml file
    • PROJECT_ID: your project ID
  3. Check if Config Sync successfully syncs all configs to your cluster:

    gcloud alpha container hub config-management status
        --project=PROJECT_ID
    

    Example output:

    Name          Status  Last_Synced_Token  Sync_Branch  Last_Synced_Time      Policy_Controller                          Hierarchy_Controller
    CLUSTER_NAME  SYNCED  6bfc9be            init         2021-06-08T17:26:32Z  GatekeeperControllerManager NOT_INSTALLED  PENDING
    

    A successful installation has a status of SYNCED.

Examining your configs

The config/ directory includes the following resources:

  • ClusterRoles
  • ClusterRoleBindings
  • CRDs
  • Namespaces
  • RoleBindings
  • ServiceAccounts
  • ResourceQuotas
  • NetworkPolicies
  • LimitRanges
  • CRs

These configs are applied as soon as the Config Sync is configured to read from the repository. In this section, you check to see if Config Sync is managing the namespaces, CRDs,and rolebindings in the directory.

All objects managed by Config Sync have the app.kubernetes.io/managed-by label set to configmanagement.gke.io and you can use this label to query your resources.

  1. List namespaces managed by Config Sync:

    kubectl get ns -l app.kubernetes.io/managed-by=configmanagement.gke.io
    

    Example output:

    NAME        STATUS   AGE
    team-1      Active   28m
    team-2      Active   28m
    
  2. List CRDs managed by Config Sync:

    kubectl get crds -A -l app.kubernetes.io/managed-by=configmanagement.gke.io
    

    Example output:

    NAME                          CREATED AT
    crontabs.stable.example.com   2021-05-04T14:58:14Z
    
  3. List rolebindings managed by Config Sync:

    kubectl get rolebindings -A -l app.kubernetes.io/managed-by=configmanagement.gke.io
    

    Example output:

    NAMESPACE   NAME                            ROLE                        AGE
    team-1      secret-reader                   ClusterRole/secret-reader   29m
    team-2      secret-admin                    ClusterRole/secret-admin    29m
    

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

To stop Config Sync from managing your cluster, run the following command:

gcloud alpha container hub config-management unmanage \
    --project=PROJECT_ID \
    --membership=CLUSTER_NAME

To delete your cluster, run the following command:

gcloud container clusters delete CLUSTER_NAME

What's next