Mutating resources

This page shows you how to mutate resources using Policy Controller.

This is useful for doing things like setting default values. For example, you might want to inject a label for all resources in a specific namespace, or default a Pod's imagePullPolicy to Always if it's not already set.

Before you begin

Mutation must be enabled by setting spec.policyController.mutation.enabled to true in the config-management resource:

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  policyController:
    enabled: true
    mutation:
      enabled: true

If you are using the gcloud tool command, you must use the alpha version to enable mutation as shown in the following example:

  # apply-spec.yaml

  applySpecVersion: 1
  spec:
    policyController:
      enabled: true
      mutationEnabled: true

After you have created the apply-spec.yaml file, run the following command to apply the configuration:

  gcloud alpha container hub config-management apply \
      --membership=MEMBERSHIP_NAME \
      --config=CONFIG_YAML_PATH \
      --project=PROJECT_ID

Replace the following:

  • MEMBERSHIP_NAME: the membership name of the registered cluster that has the Policy Controller settings you want to use
  • CONFIG_YAML_PATH: the path to the apply-spec.yaml file
  • PROJECT_ID: your project ID

Definitions

  • mutator: A Kubernetes resource that helps configure the mutation behavior of Policy Controller.
  • system: An arrangement of multiple mutators

Example

Before we examine the details, here is an example of a mutator that sets the imagePullPolicy for all containers in all Pods to Always:

# set-image-pull-policy.yaml

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: always-pull-image
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.containers[name: *].imagePullPolicy"
  parameters:
    assign:
      value: "Always"

Breaking it down

There are the standard Kubernetes metadata fields (apiVersion, kind, metadata.name), but spec is where the mutator's behavior is configured.

spec.applyTo binds the mutator to the specified resources. Because we are changing specific fields in an object, we are implicitly defining that object's schema. For example, the current mutator would make no sense if it were applied to a Namespace resource. Because of this, this field is required so Policy Controller knows what schemas this mutator is relevant for. GroupVersionKinds missing from this list are not mutated.

spec.location is telling us which field to modify. In this case, we are using a glob (*) to indicate that we want to modify all entries in the container list. Note that the only kinds of lists that might be traversed by the location field are map-type lists, and the key field for the map must be specified. Map-type lists are a Kubernetes construct. More details about them can be found in Kubernetes' documentation.

spec.parameters.assign.value is the value to assign to location. This field is un-typed and can take any value, though do note that Kubernetes still validates the request post-mutation, so inserting values with an incorrect schema for the object being modified results in the request being rejected.

Execution flow

Perhaps the most important concept to understand about Kubernetes mutation webhooks is their reinvocation policy, because the output of one mutator might change how another mutator behaves. For instance, if you add a new sidecar container, a mutator that sets the image pull policy for all containers now has a new container to mutate.

Practically, what this means is that for any given request Policy Controller's mutation webhook might be called more than one time.

In order to reduce latency, Policy Controller reinvokes itself to avoid additional HTTP requests. This means that sidecar injection and image pull policy have the expected result.

Policy Controller's mutation routine will continue to reinvoke itself until the resource "converges", which means additional iterations have no further effect.

Location syntax

The location syntax uses the dot (.) accessor to traverse fields. In the case of keyed lists, users can reference individual objects in a list using the syntax [<key>: <value>], or all objects in the list using [<key>: *].

Values and fields can be quoted with either single (') or double (") quotes. This is necessary when they have special characters, like dots or spaces.

In quoted values, special characters can be escaped by prefixing them with \. "Use \" to escape and \\\" to escape" turns into Use " to escape and \" to escape.

Some examples for the v1/Pod resource:

  • spec.priority references spec.priority
  • spec.containers[name: "foo"].volumeMounts[mountPath: "/my/mount"].readOnly references the readOnly field of the /my/mount mount of the foo container.
  • spec.containers[name: *].volumeMounts[mountPath: "/my/mount"].readOnly references the readOnly field of the /my/mount mount of all containers.

If you reference a location that doesn't currently exist on a resource, that location is created by default. This behavior can be configured via path testing.

Path testing

How can we perform defaulting, where we avoid modifying a value that already exists? Maybe we want to set /secure-mount to read-only for all containers, but we don't want to create a /secure-mount if one doesn't already exist. We can do either of these things via path testing.

Here is an example that avoids mutating imagePullPolicy if it's already set:

# set-image-pull-policy.yaml

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: always-pull-image
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.containers[name: *].imagePullPolicy"
  parameters:
    assign:
      value: "Always"
    pathTests:
    - subPath: "spec.containers[name: *].imagePullPolicy"
      condition: "MustNotExist"

Here is another example that avoids creating an empty sidecar container if it doesn't already exist:

# set-image-pull-policy.yaml

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: always-pull-image
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: 'spec.containers[name: "sidecar"].imagePullPolicy'
  parameters:
    assign:
      value: "Always"
    pathTests:
    - subPath: 'spec.containers[name: "sidecar"]'
      condition: "MustExist"

Multiple path tests can be specified, if necessary.

subPath must be a prefix of (or equal to) location.

The only valid values for condition are MustExist and MustNotExist.

Value testing

If you only want to mutate values that meet certain conditions, it is possible to do this using the spec.parameters.assignIf field.

For example, if you only wanted to mutate a Pod's security context if it explicitly configures the Pod to run as root, you would do the following:

# set-non-rootyaml

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: always-pull-image
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.securityContext.runAsUser"
  parameters:
    assign:
      value: 1111
    assignIf:
      in: [0]

assignIf has two types of tests in and notIn, both of which take a list of untyped values. If the original value is in (or not in for notIn) the list of values, then the value is mutated.

Matching

Mutators also allow for matching, using the same criteria as constraints.

Mutators

Currently there are two kinds of mutators: Assign and AssignMetadata.

Assign

Assign can change any value outside of the metadata field of a resource. Because all GroupVersionKinds have a unique schema, it must be bound to a set of particular GroupVersionKinds.

It has the following schema:

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: always-pull-image
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    kinds: # redundant because of `applyTo`, but left in for consistency
      - apiGroups: ["*"]
        kinds: ["*"]
    namespaces: ["my-namespace"]
    scope: "Namespaced" # or "Cluster"
    excludedNamespaces: ["some-other-ns"]
    labelSelector:
      matchLabels:
        mutate: "yes"
      matchExpressions:
      - key: "my-label"
        operator: "In" # or, "NotIn", "Exists", or "DoesNotExist"
        values: ["my-value"]
    namespaceSelector:
      matchLabels:
        mutate: "yes"
      matchExpressions:
      - key: "my-label"
        operator: "In" # or, "NotIn", "Exists", or "DoesNotExist"
        values: ["my-value"]
  location: "spec.containers[name: *].imagePullPolicy"
  parameters:
    pathTests:
    - subPath: 'spec.containers[name: "sidecar"]' # must be a prefix of `location`
      condition: "MustExist" # or "MustNotExist"
    - subPath: "spec.containers[name: *].imagePullPolicy"
      condition: "MustNotExist"
    assign:
      value: "Always" # any type can go here, not just a string
    assignIf:
      in: ["mutate-if-in-this-list"]
      notIn: ["don't-mutate-if-in-this-list"]

AssignMetadata

AssignMetadata can add new metadata labels. It cannot change the value of existing metadata labels. Otherwise, it would be possible to write a system of mutators that would recurse indefinitely, causing requests to time out.

Because all resources share the same metadata schema, there is no need to specify which resource an AssignMetadata applies to.

Also, because AssignMetadata isn't allowed to do as much, its schema is a bit simpler.

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignMetadata
metadata:
  name: set-team-name
spec:
  match:
    kinds:
      - apiGroups: ["*"]
        kinds: ["*"]
    namespaces: ["my-namespace"]
    scope: "Namespaced" # or "Cluster"
    excludedNamespaces: ["some-other-ns"]
    labelSelector:
      matchLabels:
        mutate: "yes"
      matchExpressions:
      - key: "my-label"
        operator: "In" # or, "NotIn", "Exists", or "DoesNotExist"
        values: ["my-value"]
    namespaceSelector:
      matchLabels:
        mutate: "yes"
      matchExpressions:
      - key: "my-label"
        operator: "In" # or, "NotIn", "Exists", or "DoesNotExist"
        values: ["my-value"]
  location: "metadata.labels.team" # must start with `metadata.labels`
  parameters:
    assign:
      value: "Always" # any type can go here, not just a string

Best practices

Kubernetes caveats

The Kubernetes documentation lists some important considerations around using mutation webhooks. Because Policy Controller operates as a Kubernetes admission webhook, that advice applies here.

Policy Controller's mutation syntax is designed to make it easier to comply with operational concerns around mutation webhooks, including idempotence.

Writing mutators

Atomicity

It is best practice to make each mutator as self-sufficient as possible. Because Kubernetes is eventually consistent, one mutator should not rely on a second mutator to have been recognized in order to do its job properly. For example, when adding a sidecar, add the whole sidecar, do not construct it piecemeal via multiple mutators.

Validation

If there is a condition that you want to enforce, it is a good idea for the mutator to have a matching constraint. This helps make sure violating requests get rejected and pre-existing violations are detected in audit.

Emergency recovery

Mutation is implemented as a Kubernetes mutating webhook. It can be stopped in the same manner as the validating webhook, but the relevant resource is a MutatingWebhookConfiguration called gatekeeper-mutating-webhook-configuration.

Additional resources

The open source documentation for Gatekeeper mutation has further explanations and examples.