This tutorial shows you how to use Config Sync and Kustomize to configure policies for namespaces in a multi-tenant cluster.
In Kubernetes, a tenant can be a team or a workload. For multi-tenant clusters, the best practice is to create a namespace for each tenant. Then, each namespace can have policies suited to their different needs. You can use Config Sync to consistently apply these policies to each of your different tenants.
Repository architecture
In this tutorial, you configure Config Sync to sync to the configs in the
namespace-specific-policy/
directory
of the Anthos Config Management samples repository. This directory contains
the following directories and files:
├── configsync
│ ├── tenant-a
│ │ ├── ~g_v1_namespace_default.yaml
│ │ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ │ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ │ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│ ├── tenant-b
│ │ ├── ~g_v1_namespace_default.yaml
│ │ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ │ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ │ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
│ └── tenant-c
│ ├── ~g_v1_namespace_default.yaml
│ ├── networking.k8s.io_v1_networkpolicy_deny-all.yaml
│ ├── rbac.authorization.k8s.io_v1_rolebinding_tenant-admin-rolebinding.yaml
│ └── rbac.authorization.k8s.io_v1_role_tenant-admin.yaml
├── configsync-src
│ ├── base
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ ├── networkpolicy.yaml
│ │ ├── rolebinding.yaml
│ │ └── role.yaml
│ ├── tenant-a
│ │ └── kustomization.yaml
│ ├── tenant-b
│ │ └── kustomization.yaml
│ └── tenant-c
│ └── kustomization.yaml
├── README.md
└── scripts
└── render.sh
This repository is an unstructured repository. This gives you the flexibility to organize your configs in the way that you want, which is particularly useful when you are working with Kustomize.
In this repository, there are three namespaces for three different tenants;
tenant-a
, tenant-b
, and tenant-c
. Each of the namespace directories
contains configs for Roles, RoleBindings, and NetworkPolicies. The configsync-
src
directory contains the configuration in kustomize
format. The directory
configsync-src contains the configuration in kustomize format.There is one base
and three overlays tenant-a
, tenant-b
, and tenant-c
.
Objectives
Update a Kustomize overlay.
Create a cluster that you can use with Config Sync.
Sync the policies in your Git repository to a cluster.
Verify that the cluster is syncing to the configs in your repository.
Costs
This tutorial uses the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.
Before you begin
Before you start this tutorial, complete the following tasks:
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
- Create, or have access to, a GitHub account.
Install Kustomize by running the following command:
gcloud components install kustomize
It's also helpful to have some familiarity with Git and Kustomize.
Update a Kustomize file
In the following section, you update a Kustomize file in your repository
to update the policies applied to tenant-a
.
Prepare your environment
To prepare your environment, complete the following steps:
Fork the repository:
- Go to the Anthos Config Management samples directory repository in GitHub.
- Click Fork.
Clone the forked repository:
git clone https://github.com/GITHUB_USERNAME/anthos-config-management-samples
Replace
GITHUB_USERNAME
with your GitHub username.Navigate to the directory that contains the examples used in this tutorial:
cd anthos-config-management-samples/namespace-specific-policy
Update an overlay
In this section you update an overlay. An overlay is a kustomization that depends on another kustomization.
The following steps show you how to give a new role to tenant-a
by updating
its overlay;
acm-samples/namespace-specific-policy/configsync-src/tenant-a
. Since you are
only updating one configuration, you only need to update one overlay.
To give tenant-a
this new role, complete the following steps:
Create a new file by navigating to the following page in GitHub:
github.com/GITHUB_USERNAME/anthos-config-management-samples/new/main/namespace-specific-policy/configsync-src/tenant-a
Name your file
another-role.yaml
.In the Edit new file window, paste the following YAML manifest for the new role:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]
Click Commit new file.
Now you can include the config for the new role in the
kustomization.yaml
file.In your
/namespace-specific-policy/configsync-src/tenant-a
folder, openkustomization.yaml
and click the edit icon (which looks like a pencil).Under the resources section of
kustomization.yaml
, add- another-role.yaml
:namespace: tenant-a resources: - ../base - another-role.yaml # The rest of the file is omitted
Click Commit changes.
After the update, rebuild the Kustomize output for each namespace by running the
render.sh
script:./scripts/render.sh
Commit and push the update:
git add . git commit -m 'update configuration' git push
Create and register a GKE cluster
In this section you create and configure a cluster that you can use with Config Sync.
Create a cluster
Console
To create a cluster with the Google Cloud console, perform the following tasks:
Visit the Google Kubernetes Engine menu in Google Cloud console.
Click add_box Create.
In the Standard section, click Configure.
In the Cluster basics section, complete the following:
- Enter
mt-cluster
as the Name for your cluster. - From the Zone drop-down list, select
us-central1-c
. - Leave all other fields with their recommended defaults.
- Enter
In the left-hand menu, click default-pool and in the drop-down list that appears, click Nodes.
In the Nodes section, complete the following:
- In the Machine type drop-down list, select e2-standard-4.
- Leave all other fields as their default values.
In the left-hand menu, select Security.
In the Security section, select the Enable Workload Identity checkbox.
Click Create. It can take several minutes for your cluster to be created.
gcloud
To create a cluster with the Google Cloud CLI, perform the following task:
gcloud container clusters create mt-cluster \
--project PROJECT_ID \
--zone us-central1-c \
--release-channel regular \
--machine-type "e2-standard-4" \
--workload-pool=PROJECT_ID.svc.id.goog
Replace PROJECT_ID
with your project ID.
Get authentication credentials for the cluster
After creating your cluster, you need to get authentication credentials to interact with the cluster:
gcloud container clusters get-credentials mt-cluster --zone us-central1-c
Give yourself admin permissions
Once your cluster is created, grant yourself the GKE Hub Admin role that you need to Config Sync.
Console
To give yourself admin permissions with the Google Cloud console, perform the following task:
In the Google Cloud console, go to the IAM page.
Click Add.
In the New members field, enter an email address. You can add individuals, service accounts, or Google Groups as members.
In the Select a role drop-down list, search for and select GKE Hub Admin.
Click Save.
gcloud
To give yourself admin permission with the Google Cloud CLI, perform the following tasks:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member=MEMBER \
--role=roles/gkehub.admin
Replace the following:
PROJECT_ID
: your project IDMEMBER
: An identifier for the member, which usually has the following form: member-type:id For example,user:my-user@example.com
. For a full list of the values that member can have, see the Policy Binding reference.
Register your cluster
After your cluster is created, you can register your cluster to a fleet.
Console
To register your cluster with the Google Cloud console, perform the following tasks:
In the Google Cloud console, go to the Anthos Clusters page.
Click Register existing cluster.
Next to
mt-cluster
, click Register. If you don't see this cluster in the list, make sure that it's created.Example output:
Cluster mt-cluster registered successfully as mt-cluster in project PROJECT_NAME.
gcloud
To register your cluster with the Google Cloud CLI, perform the following task:
gcloud container fleet memberships register MEMBERSHIP_NAME \ --gke-cluster=us-central1-c/mt-cluster \ --enable-workload-identity
Replace the following:
- MEMBERSHIP_NAME: the membership name that you choose to uniquely represent the cluster being registered to the fleet.
Configure Config Sync
In this section, you configure Config Sync so that it syncs to the policies
in the namespace-specific-policy/
directory.
To configure Config Sync, complete the following steps:
Console
To configure Config Sync with the Google Cloud console, perform the following tasks:
In the Google Cloud console, go to the Anthos Config Management page.
Select
mt-cluster
and click Configure.In the Git Repository Authentication for ACM section, select None and click Continue.
In the ACM settings for your clusters section, complete the following:
- In the Version field, select a version of 1.7 or later.
- Select the Enable Config Sync checkbox and complete the following
fields:
- In the URL field, add
https://github.com/GITHUB_USERNAME/anthos-config-management-samples
- In the Branch field, add
main
- In the Tag/Commit field, add
HEAD
- In the Configuration directory field, add
namespace-specific-policy/configsync
- Leave the Sync wait field blank.
- Leave the Git proxy field blank.
- In the Source format drop-down list, select unstructured.
- In the URL field, add
Click Done. You are taken back to the Anthos Config Management page. After a few minutes, you should see
Synced
in the status column next to the cluster that you configured.
gcloud
To configure Config Sync with the Google Cloud CLI, perform the following tasks:
Create a file named
apply-spec.yaml
and copy the following YAML file into it:# apply-spec.yaml applySpecVersion: 1 spec: configSync: # Set to true to install and enable Config Sync enabled: true sourceFormat: unstructured syncRepo: https://github.com/GITHUB_USERNAME/anthos-config-management-samples syncBranch: main secretType: none policyDir: namespace-specific-policy/configsync
Replace
GITHUB_USERNAME
with your GitHub username.Apply the
apply-spec.yaml
file:gcloud beta container fleet config-management apply \ --membership=MEMBERSHIP_NAME \ --config=CONFIG_YAML_PATH \ --project=PROJECT_ID
Replace the following:
MEMBERSHIP_NAME
: the membership name that you chose when you registered your cluster, you can find the name withgcloud container fleet memberships list
.CONFIG_YAML_PATH
: the path to yourapply-spec.yaml
filePROJECT_ID
: your project ID
Verify that namespace specific policies are synced
Now that you have installed Config Sync, you can verify that the namespace
specific policies are synced to your cluster by using the nomos status
command:
nomos status
You should see output similar the following example:
gke_PROJECT_ID_us-central1-c_mt-cluster
--------------------
<root> https:/github.com/GITHUB_USERNAME/anthos-config-management-samples/namespace-specific-policy/configsync@main
SYNCED bf8655aa
Managed resources:
NAMESPACE NAME STATUS
namespace/foo Current
namespace/istio-system Current
namespace/tenant-a Current
namespace/tenant-b Current
namespace/tenant-c Current
tenant-a networkpolicy.networking.k8s.io/deny-all Current
tenant-a role.rbac.authorization.k8s.io/tenant-admin Current
tenant-a rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current
tenant-b networkpolicy.networking.k8s.io/deny-all Current
tenant-b role.rbac.authorization.k8s.io/tenant-admin Current
tenant-b rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current
tenant-c networkpolicy.networking.k8s.io/deny-all Current
tenant-c role.rbac.authorization.k8s.io/tenant-admin Current
tenant-c rolebinding.rbac.authorization.k8s.io/tenant-admin-rolebinding Current
Next, confirm that the resources exist on the cluster. The following examples
show you how to check some of the resources for tenant-a
.
Verify the RoleBinding for
tenant-a
exists:kubectl get RoleBinding/tenant-admin-rolebinding -n tenant-a
Example output:
NAME ROLE AGE tenant-admin-rolebinding Role/tenant-admin 23h
Verify that the Role for
tenant-a
exists:kubectl get Role/tenant-admin -n tenant-a
Example output:
NAME CREATED AT tenant-admin 2021-05-24T21:49:06Z
Verify the NetworkPolicy for
tenant-a
extsts:kubectl get NetworkPolicy/deny-all -n tenant-a
Example output:
NAME POD-SELECTOR AGE deny-all <none> 23h
By using these commands, you can see that each namespace has its own set of policies, following the best practice for multi-tenant clusters.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the project
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete the directory
To clean up the tenant namespaces and policies for them, we recommend that you remove the directories that contain their configuration from your Git repository:
rm -r acm-samples/namespace-specific-policy/configsync/tenant-*/*
git add .
git commit -m 'clean up'
git push
When the last commit from the root repository is synced, the three namespaces
tenant-a
, tenant-b
, and tenant-c
are deleted from the cluster.
What's next
- Discover Best practices for enterprise multi-tenancy.
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.