Create policies for a multi-tenant cluster

---

This tutorial shows you how to use Config Sync and Kustomize to configure policies for namespaces in a multi-tenant cluster.

In Kubernetes, a tenant can be a team or a workload. For multi-tenant clusters, the best practice is to create a namespace for each tenant. Then, each namespace can have policies suited to their different needs. You can use Config Sync to consistently apply these policies to each of your different tenants.

Repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the namespace-specific-policy/ directory of the Anthos Config Management samples repository. This directory contains the following directories and files:

├── configsync
│   ├── tenant-a
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   ├── tenant-b
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   └── tenant-c
│       ├── ~g_v1_namespace_default.yaml
│       ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│       ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│       └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
├── configsync-src
│   ├── base
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   ├── networkpolicy.yaml
│   │   ├── rolebinding.yaml
│   │   └── role.yaml
│   ├── tenant-a
│   │   └── kustomization.yaml
│   ├── tenant-b
│   │   └── kustomization.yaml
│   └── tenant-c
│       └── kustomization.yaml
├── README.md
└── scripts
    └── render.sh

This repository is an unstructured repository. This gives you the flexibility to organize your configs in the way that you want, which is particularly useful when you are working with Kustomize.

In this repository, there are three namespaces for three different tenants; tenant-a, tenant-b, and tenant-c. Each of the namespace directories contains configs for Roles, RoleBindings, and NetworkPolicies. The configsync- src directory contains the configuration in kustomize format. The directory configsync-src contains the configuration in kustomize format.There is one base and three overlays tenant-a, tenant-b, and tenant-c.

Objectives

• Update a Kustomize overlay.

• Create a cluster that you can use with Config Sync.

• Sync the policies in your Git repository to a cluster.

• Verify that the cluster is syncing to the configs in your repository.

Costs

In this document, you use the following billable components of Google Cloud:

• Anthos Config Management
• GKE

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

Before you start this tutorial, complete the following tasks:

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

3. Create, or have access to, a GKE cluster that meets the requirements for Config Sync. For details on how to create such a cluster, see Get started with Config Sync.
4. Create, or have access to, a GitHub account.

5. Install Kustomize by running the following command:

   gcloud components install kustomize
   

It's also helpful to have some familiarity with Git and Kustomize.

Update a Kustomize file

In the following section, you update a Kustomize file in your repository to update the policies applied to tenant-a.

Prepare your environment

To prepare your environment, complete the following steps:

1. Fork the repository:

   1. Go to the Anthos Config Management samples directory repository in GitHub.
   2. Click Fork.

2. Clone the forked repository:

   git clone https://github.com/GITHUB_USERNAME/anthos-config-management-samples
   

   Replace GITHUB_USERNAME with your GitHub username.

3. Navigate to the directory that contains the examples used in this tutorial:

   cd anthos-config-management-samples/namespace-specific-policy
   

Update an overlay

In this section you update an overlay. An overlay is a kustomization that depends on another kustomization.

The following steps show you how to give a new role to tenant-a by updating its overlay; acm-samples/namespace-specific-policy/configsync-src/tenant-a. Since you are only updating one configuration, you only need to update one overlay.

To give tenant-a this new role, complete the following steps:

1. Create a new file by navigating to the following page in GitHub:

   github.com/GITHUB_USERNAME/anthos-config-management-samples/new/main/namespace-specific-policy/configsync-src/tenant-a
   

2. Name your file another-role.yaml.

3. In the Edit new file window, paste the following YAML manifest for the new role:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: pod-reader
   rules:
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["get", "watch", "list"]
   

4. Click Commit new file.

   Now you can include the config for the new role in the kustomization.yaml file.

5. In your /namespace-specific-policy/configsync-src/tenant-a folder, open kustomization.yaml and click the edit icon (which looks like a pencil).

6. Under the resources section of kustomization.yaml, add - another-role.yaml:

   namespace: tenant-a
   
   resources:
   - ../base
   - another-role.yaml
   # The rest of the file is omitted
   

7. Click Commit changes.

8. After the update, rebuild the Kustomize output for each namespace by running the render.sh script:

   ./scripts/render.sh
   

9. Commit and push the update:

   git add .
   git commit -m 'update configuration'
   git push
   

Note: In this tutorial, the Kustomize output is written into a different directory on the same branch in the same Git repository. You can also write the Kustomize output into a different Git repository if you want.

Configure Config Sync

In this section, you configure Config Sync so that it syncs a cluster to the policies in the namespace-specific-policy/ directory.

To configure Config Sync, complete the following steps:

Console

1. In the Google Cloud console, go to the Config Sync dashboard.

   Go to Config Sync dashboard

2. In the Config Sync settings box, click Install Config Sync.

3. In the Available Clusters table, select your cluster and click Next.

4. Clear the Enable Policy Controller checkbox and click Next.

5. Leave the Enable Config sync checkbox enabled.

6. In the Repository list, select Custom.

7. In the URL field, add https://github.com/GITHUB_USERNAME/anthos-config-management-samples

8. Click Show advanced settings.

9. In the Configuration directory field, add namespace-specific-policy/configsync

10. Leave all other fields with their default values.

11. Click Complete.

You are taken back to the Config Sync Settings page. After a few minutes, you should see Synced in the status column next to the cluster that you configured.

gcloud

To configure Config Sync with the Google Cloud CLI, perform the following tasks:

1. Create a file named apply-spec.yaml and copy the following YAML file into it:

   # apply-spec.yaml
   
   applySpecVersion: 1
   spec:
     configSync:
       # Set to true to install and enable Config Sync
       enabled: true
       sourceFormat: unstructured
       syncRepo: https://github.com/GITHUB_USERNAME/anthos-config-management-samples
       syncBranch: main
       secretType: none
       policyDir: namespace-specific-policy/configsync
   

   Replace GITHUB_USERNAME with your GitHub username.

2. Apply the apply-spec.yaml file:

    gcloud beta container fleet config-management apply \
        --membership=MEMBERSHIP_NAME \
        --config=CONFIG_YAML_PATH \
        --project=PROJECT_ID
   

   Replace the following:

   • MEMBERSHIP_NAME: the membership name that you chose when you registered your cluster, you can find the name with gcloud container fleet memberships list.
   • CONFIG_YAML_PATH: the path to your apply-spec.yaml file
   • PROJECT_ID: your project ID

Verify that namespace specific policies are synced

Now that you have installed Config Sync, you can verify that the namespace specific policies are synced to your cluster.

1. Get authentication credentials to interact with the cluster:

   gcloud container clusters get-credentials CLUSTER_NAME
       --zone ZONE
   

   Replace the following:

   • CLUSTER_NAME: the name of your cluster
   • ZONE: the zone that you created your cluster in

2. Get the status of you cluster with the nomos status command:

   nomos status
   

   The output is similar to the following:

   gke_PROJECT_ID_us-central1-c_CLUSTER_NAME
   --------------------
   <root>   https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main
   SYNCED   bf8655aa
   Managed resources:
      NAMESPACE   NAME                                                             STATUS
                  namespace/foo                                                    Current
                  namespace/istio-system                                           Current
                  namespace/tenant-a                                               Current
                  namespace/tenant-b                                               Current
                  namespace/tenant-c                                               Current
      tenant-a    networkpolicy.networking.k8s.io/deny-all                         Current
      tenant-a    role.rbac.authorization.k8s.io/tenant-admin                      Current
      tenant-a    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
      tenant-b    networkpolicy.networking.k8s.io/deny-all                         Current
      tenant-b    role.rbac.authorization.k8s.io/tenant-admin                      Current
      tenant-b    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
      tenant-c    networkpolicy.networking.k8s.io/deny-all                         Current
      tenant-c    role.rbac.authorization.k8s.io/tenant-admin                      Current
      tenant-c    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
   

3. Confirm that the resources exist on the cluster. The following examples show you how to check some of the resources for tenant-a.

   1. Verify the RoleBinding for tenant-a exists:

      kubectl get RoleBinding/tenant-admin-rolebinding -n tenant-a
      

      The output is similar to the following:

      NAME                       ROLE                AGE
      tenant-admin-rolebinding   Role/tenant-admin   23h
      

   2. Verify that the Role for tenant-a exists:

      kubectl get Role/tenant-admin -n tenant-a
      

      The output is similar to the following:

      NAME           CREATED AT
      tenant-admin   2021-05-24T21:49:06Z
      

   3. Verify the NetworkPolicy for tenant-a exists:

      kubectl get NetworkPolicy/deny-all -n tenant-a
      

      The output is similar to the following:

      NAME       POD-SELECTOR   AGE
      deny-all   <none>         23h
      

   By using these commands, you can see that each namespace has its own set of policies, following the best practice for multi-tenant clusters.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the directory

To clean up the tenant namespaces and policies for them, we recommend that you remove the directories that contain their configuration from your Git repository:

rm -r acm-samples/namespace-specific-policy/configsync/tenant-*/*
git add .
git commit -m 'clean up'
git push

When the last commit from the root repository is synced, the three namespaces tenant-a, tenant-b, and tenant-c are deleted from the cluster.

What's next

• Discover Best practices for enterprise multi-tenancy.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.