This tutorial shows you how to use Config Sync and Kustomize to configure policies for namespaces in a multi-tenant cluster.
In Kubernetes, a tenant can be a team or a workload. For multi-tenant clusters, the best practice is to create a namespace for each tenant. Then, each namespace can have policies suited to their different needs. You can use Config Sync to consistently apply these policies to each of your different tenants.
Repository architecture
In this tutorial, you configure Config Sync to sync to the configs in the
namespace-specific-policy/
directory
of the Anthos Config Management samples repository. This directory contains
the following directories and files:
├── configsync
│ ├── tenant-a
│ │ ├── ~g_v1_namespace_default.yaml
│ │ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ │ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ │ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│ ├── tenant-b
│ │ ├── ~g_v1_namespace_default.yaml
│ │ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ │ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ │ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│ └── tenant-c
│ ├── ~g_v1_namespace_default.yaml
│ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
├── configsync-src
│ ├── base
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ ├── networkpolicy.yaml
│ │ ├── rolebinding.yaml
│ │ └── role.yaml
│ ├── tenant-a
│ │ └── kustomization.yaml
│ ├── tenant-b
│ │ └── kustomization.yaml
│ └── tenant-c
│ └── kustomization.yaml
├── README.md
└── scripts
└── render.sh
This repository is an unstructured repository. This gives you the flexibility to organize your configs in the way that you want, which is particularly useful when you are working with Kustomize.
In this repository, there are three namespaces for three different tenants;
tenant-a
, tenant-b
, and tenant-c
. Each of the namespace directories
contains configs for Roles, RoleBindings, and NetworkPolicies. The configsync-
src
directory contains the configuration in kustomize
format. The directory
configsync-src contains the configuration in kustomize format.There is one base
and three overlays tenant-a
, tenant-b
, and tenant-c
.
Objectives
Update a Kustomize overlay.
Create a cluster that you can use with Config Sync.
Sync the policies in your Git repository to a cluster.
Verify that the cluster is syncing to the configs in your repository.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.
Before you begin
Before you start this tutorial, complete the following tasks:
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.
- Create, or have access to, a GKE cluster that meets the requirements for Config Sync. For details on how to create such a cluster, see Get started with Config Sync.
- Create, or have access to, a GitHub account.
Install Kustomize by running the following command:
gcloud components install kustomize
It's also helpful to have some familiarity with Git and Kustomize.
Update a Kustomize file
In the following section, you update a Kustomize file in your repository
to update the policies applied to tenant-a
.
Prepare your environment
To prepare your environment, complete the following steps:
Fork the repository:
- Go to the Anthos Config Management samples directory repository in GitHub.
- Click Fork.
Clone the forked repository:
git clone https://github.com/GITHUB_USERNAME/anthos-config-management-samples
Replace
GITHUB_USERNAME
with your GitHub username.Navigate to the directory that contains the examples used in this tutorial:
cd anthos-config-management-samples/namespace-specific-policy
Update an overlay
In this section you update an overlay. An overlay is a kustomization that depends on another kustomization.
The following steps show you how to give a new role to tenant-a
by updating
its overlay;
acm-samples/namespace-specific-policy/configsync-src/tenant-a
. Since you are
only updating one configuration, you only need to update one overlay.
To give tenant-a
this new role, complete the following steps:
Create a new file by navigating to the following page in GitHub:
github.com/GITHUB_USERNAME/anthos-config-management-samples/new/main/namespace-specific-policy/configsync-src/tenant-a
Name your file
another-role.yaml
.In the Edit new file window, paste the following YAML manifest for the new role:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]
Click Commit new file.
Now you can include the config for the new role in the
kustomization.yaml
file.In your
/namespace-specific-policy/configsync-src/tenant-a
folder, openkustomization.yaml
and click the edit icon (which looks like a pencil).Under the resources section of
kustomization.yaml
, add- another-role.yaml
:namespace: tenant-a resources: - ../base - another-role.yaml # The rest of the file is omitted
Click Commit changes.
After the update, rebuild the Kustomize output for each namespace by running the
render.sh
script:./scripts/render.sh
Commit and push the update:
git add . git commit -m 'update configuration' git push
Configure Config Sync
In this section, you configure Config Sync so that it syncs a cluster to the
policies in the
namespace-specific-policy/
directory.
To configure Config Sync, complete the following steps:
Console
In the Google Cloud console, go to the Config Sync dashboard.
In the Config Sync settings box, click add Install Config Sync.
In the Available Clusters table, select your cluster and click Next.
Clear the Enable Policy Controller checkbox and click Next.
Leave the Enable Config sync checkbox enabled.
In the Repository list, select Custom.
In the URL field, add
https://github.com/GITHUB_USERNAME/anthos-config-management-samples
Click
Show advanced settings.In the Configuration directory field, add
namespace-specific-policy/configsync
Leave all other fields with their default values.
Click Complete.
You are taken back to the Config Sync Settings page. After a few
minutes, you should see Synced
in the status column next to the cluster
that you configured.
gcloud
To configure Config Sync with the Google Cloud CLI, perform the following tasks:
Create a file named
apply-spec.yaml
and copy the following YAML file into it:# apply-spec.yaml applySpecVersion: 1 spec: configSync: # Set to true to install and enable Config Sync enabled: true sourceFormat: unstructured syncRepo: https://github.com/GITHUB_USERNAME/anthos-config-management-samples syncBranch: main secretType: none policyDir: namespace-specific-policy/configsync
Replace
GITHUB_USERNAME
with your GitHub username.Apply the
apply-spec.yaml
file:gcloud beta container fleet config-management apply \ --membership=MEMBERSHIP_NAME \ --config=CONFIG_YAML_PATH \ --project=PROJECT_ID
Replace the following:
MEMBERSHIP_NAME
: the membership name that you chose when you registered your cluster, you can find the name withgcloud container fleet memberships list
.CONFIG_YAML_PATH
: the path to yourapply-spec.yaml
filePROJECT_ID
: your project ID
Verify that namespace specific policies are synced
Now that you have installed Config Sync, you can verify that the namespace specific policies are synced to your cluster.
Get authentication credentials to interact with the cluster:
gcloud container clusters get-credentials CLUSTER_NAME --zone ZONE
Replace the following:
CLUSTER_NAME
: the name of your clusterZONE
: the zone that you created your cluster in
Get the status of you cluster with the
nomos status
command:nomos status
The output is similar to the following:
gke_PROJECT_ID_us-central1-c_CLUSTER_NAME -------------------- <root> https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main SYNCED bf8655aa Managed resources: NAMESPACE NAME STATUS namespace/foo Current namespace/istio-system Current namespace/tenant-a Current namespace/tenant-b Current namespace/tenant-c Current tenant-a networkpolicy.networking.k8s.io/deny-all Current tenant-a role.rbac.authorization.k8s.io/tenant-admin Current tenant-a rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current tenant-b networkpolicy.networking.k8s.io/deny-all Current tenant-b role.rbac.authorization.k8s.io/tenant-admin Current tenant-b rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current tenant-c networkpolicy.networking.k8s.io/deny-all Current tenant-c role.rbac.authorization.k8s.io/tenant-admin Current tenant-c rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current
Confirm that the resources exist on the cluster. The following examples show you how to check some of the resources for
tenant-a
.Verify the RoleBinding for
tenant-a
exists:kubectl get RoleBinding/tenant-admin-rolebinding -n tenant-a
The output is similar to the following:
NAME ROLE AGE tenant-admin-rolebinding Role/tenant-admin 23h
Verify that the Role for
tenant-a
exists:kubectl get Role/tenant-admin -n tenant-a
The output is similar to the following:
NAME CREATED AT tenant-admin 2021-05-24T21:49:06Z
Verify the NetworkPolicy for
tenant-a
exists:kubectl get NetworkPolicy/deny-all -n tenant-a
The output is similar to the following:
NAME POD-SELECTOR AGE deny-all <none> 23h
By using these commands, you can see that each namespace has its own set of policies, following the best practice for multi-tenant clusters.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the project
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete the directory
To clean up the tenant namespaces and policies for them, we recommend that you remove the directories that contain their configuration from your Git repository:
rm -r acm-samples/namespace-specific-policy/configsync/tenant-*/*
git add .
git commit -m 'clean up'
git push
When the last commit from the root repository is synced, the three namespaces
tenant-a
, tenant-b
, and tenant-c
are deleted from the cluster.
What's next
- Discover Best practices for enterprise multi-tenancy.
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.