Creating policies for a multi-tenant cluster

This tutorial shows you how to use Config Sync and Kustomize to configure policies for namespaces in a multi-tenant cluster.

In Kubernetes, a tenant can be a team or a workload. For multi-tenant clusters, the best practice is to create a namespace for each tenant. Then, each namespace can have policies suited to their different needs. You can use Config Sync to consistently apply these policies to each of your different tenants.

Repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the namespace-specific-policy/ directory of the Anthos Config Management samples repository. This directory contains the following directories and files:

├── configsync
│   ├── tenant-a
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   ├── tenant-b
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   └── tenant-c
│       ├── ~g_v1_namespace_default.yaml
│       ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│       ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│       └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
├── configsync-src
│   ├── base
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   ├── networkpolicy.yaml
│   │   ├── rolebinding.yaml
│   │   └── role.yaml
│   ├── tenant-a
│   │   └── kustomization.yaml
│   ├── tenant-b
│   │   └── kustomization.yaml
│   └── tenant-c
│       └── kustomization.yaml
├── README.md
└── scripts
    └── render.sh

This repository is an unstructured repository. This gives you the flexibility to organize your configs in the way that you want, which is particularly useful when you are working with Kustomize.

In this repository, there are three namespaces for three different tenants; tenant-a, tenant-b, and tenant-c. Each of the namespace directories contains configs for Roles, RoleBindings, and NetworkPolicies. The configsync- src directory contains the configuration in kustomize format. The directory configsync-src contains the configuration in kustomize format.There is one base and three overlays tenant-a, tenant-b, and tenant-c.

Objectives

• Update a Kustomize overlay.

• Create a cluster that you can use with Config Sync.

• Sync the policies in your Git repository to a cluster.

• Verify that the cluster is syncing to the configs in your repository.

Costs

This tutorial uses the following billable components of Google Cloud:

• GKE

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

New Google Cloud users might be eligible for a free trial.

Before you begin

Before you start this tutorial, complete the following tasks:

1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

3. Create, or have access to, a GitHub account.

4. Install Kustomize by running the following command:

   gcloud components install kustomize
   

It's also helpful to have some familiarity with Git and Kustomize.

Updating a Kustomize file

In the following section, you update a Kustomize file in your repository to update the policies applied to tenant-a.

Prepare your environment

To prepare your environment, complete the following steps:

1. Fork the repository:

   1. Go to the Anthos Config Management samples directory repository in GitHub.
   2. Click Fork.

2. Clone the forked repository:

   git clone https://github.com/GITHUB_USERNAME/anthos-config-management-samples
   

   Replace GITHUB_USERNAME with your GitHub username.

3. Navigate to the directory that contains the examples used in this tutorial:

   cd anthos-config-management-samples/namespace-specific-policy
   

Update an overlay

In this section you update an overlay. An overlay is a kustomization that depends on another kustomization.

The following steps show you how to give a new role to tenant-a by updating its overlay; acm-samples/namespace-specific-policy/configsync-src/tenant-a. Since you are only updating one configuration, you only need to update one overlay.

To give tenant-a this new role, complete the following steps:

1. Create a new file by navigating to the following page in GitHub:

   github.com/GITHUB_USERNAME/anthos-config-management-samples/new/main/namespace-specific-policy/configsync-src/tenant-a
   

2. Name your file another-role.yaml.

3. In the Edit new file window, paste the following YAML manifest for the new role:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: pod-reader
   rules:
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["get", "watch", "list"]
   

4. Click Commit new file.

   Now you can include the config for the new role in the kustomization.yaml file.

5. In your /namespace-specific-policy/configsync-src/tenant-a folder, open kustomization.yaml and click the edit icon (which looks like a pencil).

6. Under the resources section of kustomization.yaml, add - another-role.yaml:

   namespace: tenant-a
   
   resources:
   - ../base
   - another-role.yaml
   # The rest of the file is omitted
   

7. Click Commit changes.

8. After the update, rebuild the Kustomize output for each namespace by running the render.sh script:

   ./scripts/render.sh
   

9. Commit and push the update:

   git add .
   git commit -m 'update configuration'
   git push
   

Note: In this tutorial, the Kustomize output is written into a different directory on the same branch in the same Git repository. You can also write the Kustomize output into a different Git repository if you want.

Creating and registering a GKE cluster

In this section you create and configure a cluster that you can use with Config Sync.

Create a cluster

Console

To create a cluster with the Google Cloud Console, perform the following tasks:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit Google Kubernetes Engine

2. Click Create.

3. In the Standard section, click Configure.

4. In the Cluster basics section, complete the following:

   1. Enter mt-cluster as the Name for your cluster.
   2. From the Zone drop-down list, select us-central1-c.
   3. Leave all other fields with their recommended defaults.

5. In the left-hand menu, click default-pool and in the drop-down list that appears, click Nodes.

6. In the Nodes section, complete the following:

   1. In the Machine type drop-down list, select e2-standard-4.
   2. Leave all other fields as their default values.

7. In the left-hand menu, select Security.

8. In the Security section, select the Enable Workload Identity checkbox.

9. Click Create. It can take several minutes for your cluster to be created.

gcloud

To create a cluster with the gcloud command-line tool, perform the following task:

gcloud container clusters create mt-cluster \
    --project PROJECT_ID \
    --zone us-central1-c \
    --release-channel regular \
    --machine-type "e2-standard-4" \
    --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your project ID.

Get authentication credentials for the cluster

After creating your cluster, you need to get authentication credentials to interact with the cluster:

gcloud container clusters get-credentials mt-cluster --zone us-central1-c

Give yourself admin permissions

Once your cluster is created, grant yourself the GKE Hub Admin role that you need to Config Sync.

Note: If you are an owner of your project, you already have this permission and don't need to complete the following steps.

Console

To give yourself admin permissions with the Google Cloud Console, perform the following task:

1. In the Cloud Console, go to the IAM page.

   Go to the IAM page

2. Click Add.

3. In the New members field, enter an email address. You can add individuals, service accounts, or Google Groups as members.

4. In the Select a role drop-down list, search for and select GKE Hub Admin.

5. Click Save.

gcloud

To give yourself admin permission with the gcloud command-line tool, perform the following tasks:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=MEMBER \
    --role=roles/gkehub.admin

Replace the following:

• PROJECT_ID: your project ID
• MEMBER: An identifier for the member, which usually has the following form: member-type:id For example, user:[e-mail]. For a full list of the values that member can have, see the Policy Binding reference.

Note: For the member type user, the domain name in the identifier must be a Google Workspace domain or a Cloud Identity domain. To learn how to set up a Cloud Identity domain, see the overview of Cloud Identity.

Register your cluster

After your cluster is created, you can register your cluster to a fleet.

Console

To register your cluster with the Google Cloud Console, perform the following tasks:

1. In the Google Cloud Console, go to the Anthos Clusters page.

   Go to the Anthos Clusters page

2. Click Register existing cluster.

3. Next to mt-cluster, click Register. If you don't see this cluster in the list, make sure that it's created.

   Example output:

   Cluster mt-cluster registered successfully as mt-cluster in project PROJECT_NAME.
   

gcloud

To register your cluster with the gcloud command-line tool, perform the following task:

gcloud beta container hub memberships register MEMBERSHIP_NAME \
 --gke-cluster=us-central1-c/mt-cluster \
 --enable-workload-identity

Replace the following:

• MEMBERSHIP_NAME: the membership name that you choose to uniquely represent the cluster being registered to the fleet.

Configuring Config Sync

In this section, you configure Config Sync so that it syncs to the policies in the namespace-specific-policy/ directory.

To configure Config Sync, complete the following steps:

Console

To configure Config Sync with the Google Cloud Console, perform the following tasks:

1. In the Cloud Console, go to the Anthos Config Management page.

   Go to Anthos Config Management

2. Select mt-cluster and click Configure.

3. In the Git Repository Authentication for ACM section, select None and click Continue.

4. In the ACM settings for your clusters section, complete the following:

   1. In the Version field, select a version of 1.7 or later.
   2. Select the Enable Config Sync checkbox and complete the following fields:
      1. In the URL field, add https://github.com/GITHUB_USERNAME/anthos-config-management-samples
      2. In the Branch field, add main
      3. In the Tag/Commit field, add HEAD
      4. In the Policy directory field, add namespace-specific-policy/configsync
      5. Leave the Sync wait field blank.
      6. Leave the Git proxy field blank.
      7. In the Source format drop-down list, select unstructured.

5. Click Done. You are taken back to the Anthos Config Management page. After a few minutes, you should see Synced in the status column next to the cluster that you configured.

gcloud

To configure Config Sync with the gcloud command-line tool, perform the following tasks:

1. Create a file named apply-spec.yaml and copy the following YAML file into it:

   # apply-spec.yaml
   
   applySpecVersion: 1
   spec:
     configSync:
       # Set to true to install and enable Config Sync
       enabled: true
       sourceFormat: unstructured
       syncRepo: https://github.com/GITHUB_USERNAME/anthos-config-management-samples
       syncBranch: main
       secretType: none
       policyDir: namespace-specific-policy/configsync
   

   Replace GITHUB_USERNAME with your GitHub username.

2. Apply the apply-spec.yaml file:

    gcloud beta container hub config-management apply \
        --membership=MEMBERSHIP_NAME \
        --config=CONFIG_YAML_PATH \
        --project=PROJECT_ID
   

   Replace the following:

   • MEMBERSHIP_NAME: the membership name that you chose when you registered your cluster, you can find the name with gcloud container hub memberships list.
   • CONFIG_YAML_PATH: the path to your apply-spec.yaml file
   • PROJECT_ID: your project ID}

Verifying that namespace specific policies are synced

Now that you have installed Config Sync, you can verify that the namespace specific policies are synced to your cluster by using the nomos status command:

nomos status

You should see output similar the following example:

gke_PROJECT_ID_us-central1-c_mt-cluster
  --------------------
  <root>   https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main
  SYNCED   bf8655aa
  Managed resources:
     NAMESPACE   NAME                                                             STATUS
                 namespace/foo                                                    Current
                 namespace/istio-system                                           Current
                 namespace/tenant-a                                               Current
                 namespace/tenant-b                                               Current
                 namespace/tenant-c                                               Current
     tenant-a    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-a    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-a    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
     tenant-b    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-b    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-b    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
     tenant-c    networkpolicy.networking.k8s.io/deny-all                         Current
     tenant-c    role.rbac.authorization.k8s.io/tenant-admin                      Current
     tenant-c    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current

Next, confirm that the resources exist on the cluster. The following examples show you how to check some of the resources for tenant-a.

1. Verify the RoleBinding for tenant-a exists:

   kubectl get RoleBinding/tenant-admin-rolebinding -n tenant-a
   

   Example output:

   NAME                       ROLE                AGE
   tenant-admin-rolebinding   Role/tenant-admin   23h
   

2. Verify that the Role for tenant-a exists:

   kubectl get Role/tenant-admin -n tenant-a
   

   Example output:

   NAME           CREATED AT
   tenant-admin   2021-05-24T21:49:06Z
   

3. Verify the NetworkPolicy for tenant-a extsts:

   kubectl get NetworkPolicy/deny-all -n tenant-a
   

   Example output:

   NAME       POD-SELECTOR   AGE
   deny-all   <none>         23h
   

By using these commands, you can see that each namespace has its own set of policies, following the best practice for multi-tenant clusters.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the directory

To clean up the tenant namespaces and policies for them, we recommend that you remove the directories that contain their configuration from your Git repository:

rm -r acm-samples/namespace-specific-policy/configsync/tenant-*/*
git add .
git commit -m 'clean up'
git push

When the last commit from the root repository is synced, the three namespaces tenant-a, tenant-b, and tenant-c are deleted from the cluster.

What's next

• Discover Best practices for enterprise multi-tenancy.
• Look through other Config Sync tutorials.
• Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.