Create policies for a multi-tenant cluster


This tutorial shows you how to use Config Sync and Kustomize to configure policies for namespaces in a multi-tenant cluster.

In Kubernetes, a tenant can be a team or a workload. For multi-tenant clusters, the best practice is to create a namespace for each tenant. Then, each namespace can have policies suited to their different needs. You can use Config Sync to consistently apply these policies to each of your different tenants.

Repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the namespace-specific-policy/ directory of the Anthos Config Management samples repository. This directory contains the following directories and files:

├── configsync
│   ├── tenant-a
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   ├── tenant-b
│   │   ├── ~g_v1_namespace_default.yaml
│   │   ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│   │   ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│   │   └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│   └── tenant-c
│       ├── ~g_v1_namespace_default.yaml
│       ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│       ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│       └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
├── configsync-src
│   ├── base
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   ├── networkpolicy.yaml
│   │   ├── rolebinding.yaml
│   │   └── role.yaml
│   ├── tenant-a
│   │   └── kustomization.yaml
│   ├── tenant-b
│   │   └── kustomization.yaml
│   └── tenant-c
│       └── kustomization.yaml
├── README.md
└── scripts
    └── render.sh

This repository is an unstructured repository. This gives you the flexibility to organize your configs in the way that you want, which is particularly useful when you are working with Kustomize.

In this repository, there are three namespaces for three different tenants; tenant-a, tenant-b, and tenant-c. Each of the namespace directories contains configs for Roles, RoleBindings, and NetworkPolicies. The configsync- src directory contains the configuration in kustomize format. The directory configsync-src contains the configuration in kustomize format.There is one base and three overlays tenant-a, tenant-b, and tenant-c.

Objectives

  • Update a Kustomize overlay.

  • Create a cluster that you can use with Config Sync.

  • Sync the policies in your Git repository to a cluster.

  • Verify that the cluster is syncing to the configs in your repository.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

Before you start this tutorial, complete the following tasks:

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

  3. Create, or have access to, a GKE cluster that meets the requirements for Config Sync. For details on how to create such a cluster, see Get started with Config Sync.
  4. Create, or have access to, a GitHub account.
  5. Install Kustomize by running the following command:

    gcloud components install kustomize
    

It's also helpful to have some familiarity with Git and Kustomize.

Update a Kustomize file

In the following section, you update a Kustomize file in your repository to update the policies applied to tenant-a.

Prepare your environment

To prepare your environment, complete the following steps:

  1. Fork the repository:

    1. Go to the Anthos Config Management samples directory repository in GitHub.
    2. Click Fork.
  2. Clone the forked repository:

    git clone https://github.com/GITHUB_USERNAME/anthos-config-management-samples
    

    Replace GITHUB_USERNAME with your GitHub username.

  3. Navigate to the directory that contains the examples used in this tutorial:

    cd anthos-config-management-samples/namespace-specific-policy
    

Update an overlay

In this section you update an overlay. An overlay is a kustomization that depends on another kustomization.

The following steps show you how to give a new role to tenant-a by updating its overlay; acm-samples/namespace-specific-policy/configsync-src/tenant-a. Since you are only updating one configuration, you only need to update one overlay.

To give tenant-a this new role, complete the following steps:

  1. Create a new file by navigating to the following page in GitHub:

    github.com/GITHUB_USERNAME/anthos-config-management-samples/new/main/namespace-specific-policy/configsync-src/tenant-a
    
  2. Name your file another-role.yaml.

  3. In the Edit new file window, paste the following YAML manifest for the new role:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: pod-reader
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["get", "watch", "list"]
    
  4. Click Commit new file.

    Now you can include the config for the new role in the kustomization.yaml file.

  5. In your /namespace-specific-policy/configsync-src/tenant-a folder, open kustomization.yaml and click the edit icon (which looks like a pencil).

  6. Under the resources section of kustomization.yaml, add - another-role.yaml:

    namespace: tenant-a
    
    resources:
    - ../base
    - another-role.yaml
    # The rest of the file is omitted
    
  7. Click Commit changes.

  8. After the update, rebuild the Kustomize output for each namespace by running the render.sh script:

    ./scripts/render.sh
    
  9. Commit and push the update:

    git add .
    git commit -m 'update configuration'
    git push
    

Configure Config Sync

In this section, you configure Config Sync so that it syncs a cluster to the policies in the namespace-specific-policy/ directory.

To configure Config Sync, complete the following steps:

Console

  1. In the Google Cloud console, go to the Config Sync dashboard.

    Go to Config Sync dashboard

  2. In the Config Sync settings box, click Install Config Sync.

  3. In the Available Clusters table, select your cluster and click Next.

  4. Clear the Enable Policy Controller checkbox and click Next.

  5. Leave the Enable Config sync checkbox enabled.

  6. In the Repository list, select Custom.

  7. In the URL field, add https://github.com/GITHUB_USERNAME/anthos-config-management-samples

  8. Click Show advanced settings.

  9. In the Configuration directory field, add namespace-specific-policy/configsync

  10. Leave all other fields with their default values.

  11. Click Complete.

You are taken back to the Config Sync Settings page. After a few minutes, you should see Synced in the status column next to the cluster that you configured.

gcloud

To configure Config Sync with the Google Cloud CLI, perform the following tasks:

  1. Create a file named apply-spec.yaml and copy the following YAML file into it:

    # apply-spec.yaml
    
    applySpecVersion: 1
    spec:
      configSync:
        # Set to true to install and enable Config Sync
        enabled: true
        sourceFormat: unstructured
        syncRepo: https://github.com/GITHUB_USERNAME/anthos-config-management-samples
        syncBranch: main
        secretType: none
        policyDir: namespace-specific-policy/configsync
    

    Replace GITHUB_USERNAME with your GitHub username.

  2. Apply the apply-spec.yaml file:

     gcloud beta container fleet config-management apply \
         --membership=MEMBERSHIP_NAME \
         --config=CONFIG_YAML_PATH \
         --project=PROJECT_ID
    

    Replace the following:

    • MEMBERSHIP_NAME: the membership name that you chose when you registered your cluster, you can find the name with gcloud container fleet memberships list.
    • CONFIG_YAML_PATH: the path to your apply-spec.yaml file
    • PROJECT_ID: your project ID

Verify that namespace specific policies are synced

Now that you have installed Config Sync, you can verify that the namespace specific policies are synced to your cluster.

  1. Get authentication credentials to interact with the cluster:

    gcloud container clusters get-credentials CLUSTER_NAME
        --zone ZONE
    

    Replace the following:

    • CLUSTER_NAME: the name of your cluster
    • ZONE: the zone that you created your cluster in
  2. Get the status of you cluster with the nomos status command:

    nomos status
    

    The output is similar to the following:

    gke_PROJECT_ID_us-central1-c_CLUSTER_NAME
    --------------------
    <root>   https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main
    SYNCED   bf8655aa
    Managed resources:
       NAMESPACE   NAME                                                             STATUS
                   namespace/foo                                                    Current
                   namespace/istio-system                                           Current
                   namespace/tenant-a                                               Current
                   namespace/tenant-b                                               Current
                   namespace/tenant-c                                               Current
       tenant-a    networkpolicy.networking.k8s.io/deny-all                         Current
       tenant-a    role.rbac.authorization.k8s.io/tenant-admin                      Current
       tenant-a    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
       tenant-b    networkpolicy.networking.k8s.io/deny-all                         Current
       tenant-b    role.rbac.authorization.k8s.io/tenant-admin                      Current
       tenant-b    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
       tenant-c    networkpolicy.networking.k8s.io/deny-all                         Current
       tenant-c    role.rbac.authorization.k8s.io/tenant-admin                      Current
       tenant-c    rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding   Current
    
  3. Confirm that the resources exist on the cluster. The following examples show you how to check some of the resources for tenant-a.

    1. Verify the RoleBinding for tenant-a exists:

      kubectl get RoleBinding/tenant-admin-rolebinding -n tenant-a
      

      The output is similar to the following:

      NAME                       ROLE                AGE
      tenant-admin-rolebinding   Role/tenant-admin   23h
      
    2. Verify that the Role for tenant-a exists:

      kubectl get Role/tenant-admin -n tenant-a
      

      The output is similar to the following:

      NAME           CREATED AT
      tenant-admin   2021-05-24T21:49:06Z
      
    3. Verify the NetworkPolicy for tenant-a exists:

      kubectl get NetworkPolicy/deny-all -n tenant-a
      

      The output is similar to the following:

      NAME       POD-SELECTOR   AGE
      deny-all   <none>         23h
      

    By using these commands, you can see that each namespace has its own set of policies, following the best practice for multi-tenant clusters.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the directory

To clean up the tenant namespaces and policies for them, we recommend that you remove the directories that contain their configuration from your Git repository:

rm -r acm-samples/namespace-specific-policy/configsync/tenant-*/*
git add .
git commit -m 'clean up'
git push

When the last commit from the root repository is synced, the three namespaces tenant-a, tenant-b, and tenant-c are deleted from the cluster.

What's next