Hierarchy Controller overview
Hierarchy Controller adds hierarchical features to Kubernetes namespaces, which lets you write more expressive policies, improve observability, and delegate cluster administration.
Hierarchy Controller is based on the Hierarchical Namespace Controller (HNC), an open source project. It is implemented using Kubernetes controllers and dynamic admission controllers which run in your cluster.
Hierarchical namespaces
In Kubernetes, namespaces are the fundamental unit of organization of most objects. They also form the fundamental unit of isolation and security in Kubernetes. Most policies and isolation objects operate at the namespace level, such as RBAC roles, secrets, service accounts, resource quotas, and network policies.
You can typically improve your cluster's security and isolation by scoping each namespace to contain the smallest number of resources, such as a single microservice, along with its resources and policies. However, this can lead to a large number of namespaces in your clusters, which can be difficult to manage.
To solve this, Hierarchy Controller introduces the concept of hierarchical namespaces, which let you group Kubernetes namespaces according to their owners, and manipulate those groups as cohesive units. They are especially useful in clusters that are shared by multiple teams, but the owners do not need to be people. For example, you might want to make an automated tool the owner of a set of namespaces, especially if it requires very broad permissions but only needs access to a small set of related namespaces.
To support multi-tenant use cases, hierarchical namespaces support several features in addition to those of regular Kubernetes namespaces:
- Policy enforcement. Hierarchical namespaces can inherit certain policies from their ancestors through the use of propagation. For example, this lets you grant RBAC permissions in a "root" namespace, and Hierarchy Controller ensures that those permissions are also in all descendant namespaces by copying (propagating) the RBAC objects to those descendants. Propagation can also be controlled at a fine-grained level using exceptions.
- Policy application. All hierarchical namespaces include trusted, well-known labels that reflect their ancestors. These labels can be used in label selectors for policies such as validating webhook configurations or network policies.
- Hierarchical quota. You can define a single hierarchical quota in an ancestor namespace, and the limits are automatically enforced on the aggregate usage of that namespace, as well as all its descendants.
- Hierarchical observability. The trusted labels used for policy application can also be used to observe hierarchical workloads, such as to filter logs or usage.
- Delegated namespace creation. Hierarchy Controller introduces the concept of a subnamespace, which can be created by users under an existing namespace even if that user does not have cluster-level namespace privileges.
Hierarchical namespaces also support extensive administration features, including delegation.
Hierarchical namespaces versus abstract namespaces
The hierarchical namespaces provided by Hierarchy Controller are similar to the abstract namespaces you can use if you are using a Config Sync hierarchical repository. However, they have several conceptual differences:
- Abstract namespaces can only be defined in a hierarchical repository, which enforces a particular structure on your configs. Hierarchical namespaces can be used in unstructured repositories, letting you arrange your config files however you want.
- Abstract namespaces only exist in Git repositories, while hierarchical namespaces are also regular Kubernetes namespaces and exist on the cluster. As a result, hierarchical namespaces can be defined either in a repository or directly on the cluster.
- Hierarchical namespaces do not require cluster-level permissions to create. Instead, you can use subnamespaces for deletegated namespace creation.
- Hierarchical Resource Quotas are only supported in hierarchical namespaces, not abstract namespaces.
- Abstract namespaces are only available to Config Sync, while hierarchical namespaces are based on open-source concepts.
Abstract and hierarchical namespaces also have some differing default behaviors. By default, all objects that are created in an abstract namespace are propagated to its descendants. By contrast, objects in hierarchical namespaces are only propagated if Hierarchy Controller has been configured to propagate that object type.
By default, only RBAC roles and role bindings are propagated, but any namespaced object can be propagated. We recommend that you only propagate policy objects (like roles) and resource objects (like config maps). Hierarchy Controller is not designed to propagate workload objects such as pods, deployments, or jobs, and may produce unintended consequences if used in this way. We recommend putting workloads into leaf namespaces.
Using Hierarchy Controller with Config Sync
Hierarchy Controller lets you define and apply policies using hierarchical concepts, which are suitable for many applications including clusters used by multiple teams. Config Sync lets you store those policies in a Git repository and apply them to groups of clusters. While different, these two features are complimentary and gives you a powerful way to apply GitOps in multi-team, multi-cluster environments.
When you are using Hierarchy Controller with Config Sync, we recommend
that you use an
unstructured repository
to disable Config Sync abstract namespaces, and use
Hierarchy Controller to define hierarchies and propagate policies. For the
namespaces you want to check in to your repo, we recommend only checking in the
configs for
full namespaces,
not subnamespaces, because you can update their hierarchical relationships by
modifying the HierarchicalConfiguration object.
It is possible to use Hierarchy Controller together with Config Sync abstract namespaces in a hierarchical repository, but only in limited ways. For example, Hierarchy Controller will not let you create a hierarchical relationship between two namespaces that were defined in a hierarchical repository, as this could conflict with abstract namespaces in that repository. However, you can use hierarchical namespaces with a hierarchical repository in the following ways:
- You can create self-service subnamespaces on the cluster, underneath a namespace created from a hierarchical repository. However, we do not recommend that you check in the configs for those subnamespaces.
- If you are using multiple repositories, you can create subnamespace anchors in the namespaces repositories, which will instruct Hierarchy Controller to create subnamespaces underneath the specified namespace. These subnamespaces will inherit all configured resources from its ancestors, and will be constrained by any hierarchical resource quotas. However, you cannot define any resources that should only exist in these subnamespaces; Config Sync will sync all objects in the namespace repository to the specified namespace.