Configuring Kubernetes objects

This topic demonstrates how to create configs, the files Config Sync reads from Git and applies to your clusters automatically.

Before you begin

  • You need a basic understanding of YAML or JSON syntax, because configs are written in one of these two formats. All the examples in this documentation use YAML, because it is easier for people to read.

  • Different types of Kubernetes objects have different configurable options. It is helpful to understand how you would achieve your desired configuration manually before writing a config for that type of object.

  • If you choose to use a hierarchical repo, make sure you understand the structure of the hierarchical repo. The location of a config within the repo impacts which clusters and namespaces it is applied to. This is especially important for the namespaces/ directory, because subdirectories of the namespaces/ directory can inherit configs from their abstract namespace directories.

  • We created a canonical example repo to illustrate how Config Sync works. It has various examples, including unstructured format and hierarchical format, multi-repo mode and non-multi-repo mode, root repository and namespace repositories.

    Examples in this topic are taken from that repo, so you might find it helpful to have the repo open in a browser or clone it to your local system.

Creating a config

When you create a config, you need to decide the best location in the repo and the fields to include.

Location in the repo

  • For unstructured repos, you can organize the configs arbitrarily and create subfolders of resources.

  • For hierarchical repos, the location of a config in the repo is one of the factors that determines which clusters it applies to:

    • Configs for cluster-scoped objects except for namespaces are stored in the cluster/ directory of the repo.
    • Configs for namespaces and namespace-scoped objects are stored in the namespaces/ directory of the repo.
    • Configs for Config Sync components are stored in the system/ directory of the repo.
    • The config for the Config Management Operator is not stored directly in the repo and is not synced.

Contents of the config

Configs use an additive approach, similar to kubectl. When creating new objects, you need to include all required fields. However, when updating existing objects, you only need to supply the fields you need to update.

The config, when applied, must result in a valid Kubernetes object.

Example configs

The following example configs are all taken from the example repo, and should get you started writing your own configs. This list is not exhaustive; you can configure any type of Kubernetes object using Config Sync.

Namespace config

This config creates a namespace called gamestore.

apiVersion: v1
kind: Namespace
metadata:
  name: gamestore

When you create a namespace config, you can also add labels or annotations to the namespace. Labels are required when using a NamespaceSelector.

The following example config creates a namespace called gamestore if it doesn't already exist or is not managed. The namespace has the label app: gamestore and the annotation retail: true. If someone manually modifies any of the object's metadata, Config Sync quickly resets it to the value in the config.

apiVersion: v1
kind: Namespace
metadata:
  name: gamestore
  labels:
    app: gamestore
  annotations:
    retail: "true"

For more information about working with namespaces, see Configuring namespaces and namespace-scoped objects.

ClusterRole config

This config creates a ClusterRole called namespace-reader, which provides the ability to read (get, watch, and list) all namespace objects in the cluster. A ClusterRole config is often used together with a ClusterRoleBinding config.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "watch", "list"]

ClusterRoleBinding config

This config creates a ClusterRoleBinding called namespace-readers, which grants user cheryl@example.com the namespace-reader ClusterRole.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: namespace-readers
subjects:
- kind: User
  name: cheryl@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: namespace-reader
  apiGroup: rbac.authorization.k8s.io

ClusterRoleBindings are cluster-scoped, and cannot be placed in namespace directories or abstract namespaces.

PodSecurityPolicy config

This example creates a PodSecurityPolicy called psp, which disallows running privileged containers, and allows containers to run as any valid user on the node.

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp
spec:
  privileged: false
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

PodSecurityPolicies are cluster-scoped, and cannot be placed in namespace directories or abstract namespaces.

NetworkPolicy config

This example creates a NetworkPolicy called default-deny-all-traffic.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-all-traffic
spec:
  podSelector: {}

NetworkPolicies are namespace-scoped, and can only be placed in namespace directories or abstract namespaces.

When you apply the above NetworkPolicy to a single namespace, it isolates any Pods in that namespace from ingress and egress traffic.

When you apply the same NetworkPolicy to multiple namespaces by placing it in an abstract namespace with descendant namespaces, each of those namespaces inherits the NetworkPolicy. In the namespace-inheritance example repo, the namespaces directory contains two abstract namespaces, eng and rnd, and each abstract namespace contains two actual namespaces, analytics and gamestore, incubator-1 and incubator-2 respectively. If you add the default-deny-all-traffic NetworkPolicy above to the abstract namespaces, the four actual namespaces each inherit the NetworkPolicy, so each of their Pods is protected from ingress and egress traffic.

You can use namespace inheritance to enforce a least-privilege approach to security. For example, if the previous NetworkPolicy example is applied to both abstract namespaces and the following NetworkPolicy is added to the eng abstract namespace, ingress traffic is allowed only to Pods in the descendant namespaces with the app:gamestore label. analytics, incubator-1, and incubator-2 namespaces are not affected by this NetworkPolicy.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-gamestore-ingress
spec:
  podSelector:
    matchLabels:
      app: gamestore
  ingress:
  - {}

ResourceQuota config

This example creates a ResourceQuota called quota, which sets a hard limit of 1 Pod, 100 milli-CPUs, and 100 mebibytes (Mi) of memory.

kind: ResourceQuota
apiVersion: v1
metadata:
  name: quota
spec:
  hard:
    pods: "1"
    cpu: "100m"
    memory: "100Mi"

If creating a new object of a given type would violate an existing ResourceQuota, Kubernetes cannot create that object until doing so would no longer violate the ResourceQuota.

RepoSync config

This example creates a RepoSync object in the root repository, which syncs from a namespace repository. For more information on configuring the RepoSync object, see Configuring syncing from namespace repositories.

apiVersion: configsync.gke.io/v1beta1
kind: RepoSync
metadata:
  name: repo-sync
  namespace: gamestore
spec:
  sourceFormat: unstructured
  git:
    repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
    branch: init
    dir: quickstart/multirepo/namespaces/gamestore
    auth: none

Config Sync creates a namespace reconciler to sync from the namespace repository.

What's next