Configuring only a subset of clusters

Normally, Config Sync applies a config to each enrolled cluster. If the config is within the namespaces/ subdirectory, Config Sync first creates the namespace within each cluster and then applies all inherited configs to that namespace.

However, you have the ability to apply a config to a subset of clusters. Starting from version 1.6.1, you can add the configsync.gke.io/cluster-name-selector annotation to your configs. If you are using a version earlier than 1.6.1, you can use a ClusterSelector.

Using the cluster selector annotation

You can use the configsync.gke.io/cluster-name-selector annotation to apply a config to a set of clusters, denoted by cluster names. The value of the annotation is a comma-separated list of target cluster names. You can apply the annotation on cluster-scoped objects and namespace-scoped objects. Namespace objects are selected when the annotation matches the cluster name and when the namespace that the clusters belong to is also selected.

Selecting a single cluster

The following config creates a Role called namespace-reader that defines a set of permissions for reading namespaces. This Role is only instantiated on the cluster that has the name cluster-1.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: namespace-reader
  annotations:
    configsync.gke.io/cluster-name-selector: cluster-1
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch", "list"]

Selecting a list of clusters

The following config creates the same Role as the previous example, but this Role is only instantiated on the clusters that have the name cluster-1, cluster-2, or cluster-3.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: namespace-reader
  annotations:
    configsync.gke.io/cluster-name-selector: cluster-1,cluster-2,cluster-3
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch", "list"]

Using ClusterSelectors

A ClusterSelector is a special type of config that uses Kubernetes labelSelectors. You can use a ClusterSelector to limit which clusters a particular config applies to, based on the cluster's labels. You can also use ClusterSelectors to limit which clusters instantiate a namespace-scoped object.

Like other labelSelectors, ClusterSelectors operate using AND logic. Since ClusterSelectors use AND logic, objects are only selected when they match all of the labels that you define.

A ClusterSelector config does not persist in a cluster. Instead, you reference it in another config using an annotation, and that config only applies to clusters that match the ClusterSelector.

Before you can use ClusterSelectors, each cluster must have a unique name. You set the cluster's name when you configure the Config Sync Operator. You can apply labels to each cluster manually, but we recommend using a Cluster config.

Next, you create the ClusterSelector, then reference it in another config.

Adding labels to a cluster

To use ClusterSelectors, each cluster must have a set of labels that can be selected. You can label clusters manually, but it is recommended that you use a Cluster config for each cluster. Cluster configs are stored in the clusterregistry/ directory of your Git repo.

The clusterName field of your config-management.yaml file should match the metadata.name field of at least one of the Cluster configs.

The following example Cluster config declares that cluster-2 has the environment: prod and location: central labels.

kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: cluster-2
  labels:
    environment: prod
    location: central

You can also apply annotations using a Cluster config.

Creating a ClusterSelector

A ClusterSelector selects only clusters with a given label or combination of labels. ClusterSelectors are stored in the top level clusterregistry/ directory in the repo.

The following ClusterSelector selects only clusters with the environment: prod label.

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: selector-env-prod
spec:
  selector:
    matchLabels:
      environment: prod

The following ClusterSelector selects any clusters with the location: central or location: west labels.

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: selector-central-or-west
spec:
  selector:
    matchExpressions:
      - key: location
        operator: In
        values:
        - central
        - west

A ClusterSelector has no effect until you reference it in another config.

Referencing a ClusterSelector

To reference a ClusterSelector in another config, set the annotation configmanagement.gke.io/cluster-selector: CLUSTERSELECTOR-NAME.

The following config creates a ClusterRole called namespace-reader that defines a set of permissions for reading namespaces. This ClusterRole is only instantiated on clusters that match the selector-env-prod ClusterSelector.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader
  annotations:
    configmanagement.gke.io/cluster-selector: selector-env-prod
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch", "list"]

Limiting the clusters a namespace-scoped config affects

By default, Config Sync applies configs inherited by a namespace to that namespace in each cluster where it exists. You can use a ClusterSelector to apply the config to only a subset of those clusters.

For example, you could configure clusters differently based on their geographic location or the geographic location of their clients. This configuration could be useful for localization or for legal compliance. The examples in this section apply locale-specific configs to only clusters with the label location: france.

The following Cluster config adds an location: france label to a cluster called cluster-1:

kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: cluster-1
  labels:
    location: france

You could label each relevant cluster manually, but this process creates room for error and is not recommended.

Create a ClusterSelector config that references the labels you want to select. The following ClusterConfig selects the location: france label:

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: selector-location-france
spec:
  selector:
    matchLabels:
      location: france

A ClusterSelector has no effect until you reference it in another config. The following RoleBinding config selects only clusters that match the selector-location-france ClusterSelector. This configuration might be useful, for instance, if a compliance guideline only allowed auditors to audit information on clusters in their designated geographic region.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: viewers
  annotations:
    configmanagement.gke.io/cluster-selector: selector-location-france
subjects:
- kind: Group
  name: system:serviceaccounts:audit
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

If you place this config in an abstract namespace or a namespace directory, such as namespaces/online/shipping-app-backend/, it is applied to the namespaces that inherit it, but only on clusters with the location: france label.

What's next