Migrating from OpenShift to GKE Enterprise: Migrating OpenShift security context constraints to GKE Enterprise

Last reviewed 2022-01-24 UTC

This document helps you to plan migration of security policies from OpenShift security context constraints (SCCs) defined on a source OpenShift cluster to a target GKE cluster. The implementation uses Policy Controller constraints to define the migrated policies on the target cluster.

The document assumes that you are familiar with Migrating containers to Google Cloud: Migrating from OpenShift to GKE Enterprise. It assumes that you are familiar with OpenShift and Security Context Constraints and that you have access to a source OpenShift cluster and a target GKE cluster.

This document is part of a multi-part series about migrating to Google Cloud. For an overview of the series, see Migration to Google Cloud: Choosing your migration path.

This document is part of a series that discusses migrating containers to Google Cloud:

This document is useful if you're planning to migrate OpenShift SCCs to GKE Enterprise. This document is also useful if you're evaluating the opportunity to migrate and want to explore what it might look like.

This document relies on concepts covered in Migration to Google Cloud: Getting started, in Migrating containers to Google Cloud: Migrating Kubernetes to GKE, in Migrating containers to Google Cloud: Migrating from OpenShift to GKE Enterprise, and in Best practices for GKE networking. It includes links to those documents where appropriate.

OpenShift SCCs

SCCs are OpenShift-specific resources that are used to define policies for Pods that specify the actions that a Pod can perform and what resources it can access on nodes. When you make an API request to create a pod, SCCs evaluate Pod requests in terms of process privileges against a set of policies that are defined through the SCC. The SCCs evaluate the requests to allow or disallow the execution of the Pods according to configured policies. For more information about OpenShift SCCs, see Managing security context constraints.

Default OpenShift SCCs

OpenShift 4.x clusters contain a set of default SCCs that are described in the Managing SCCs in OpenShift blog post by Red Hat.

Config Sync and Policy Controller

This section describes Config Sync and Policy Controller. This section provides links to the relevant documentation and guidance to set up Policy Controller to perform migration tasks described later in this document.

Config Sync

Config Sync lets you use a common Git-compliant repository to centrally define the configuration of any resource that applies to any GKE Enterprise-managed Kubernetes cluster. You apply that configuration to multiple clusters.

Policy Controller

Policy Controller is a Kubernetes dynamic admission controller that checks, audits, and enforces the compliance of your clusters against centrally defined policies. Policy Controller is based on the Open Policy Agent (OPA) Gatekeeper open source project.

Config Sync and Policy Controller setup

To prepare to implement security policies that mirror the OpenShift SCCs, you enable Config Sync and Policy Controller components for each of your target clusters. This section describes how to set up these components. The Migrate OpenShift SCCs section later in this document describes how to use Policy Controller constraint templates to implement the security policies.

When you set up Policy Controller, it's a good practice to put system-related namespaces that don't run application Pods into the Exempt namespaces field. Exempting system-related namespaces helps you to avoid the risk of blocking any system Pod that requires elevated privileges. System-related namespaces on an GKE cluster can include the following:

  • kube-system
  • kube-public
  • gke-connect
  • gke-system
  • config-management-system
  • config-management-monitoring
  • gatekeeper-system
  • istio-system
  • cnrm-system
  • knative-serving
  • monitoring-system

After you set up Policy Controller with the preceding exceptions, exclude the namespaces from constraints application by adding the admission.gatekeeper.sh/ignore=true label to each namespace. If you don't add the label to each namespace, system Pods (and therefore your entire cluster) can be affected by restrictive policies.

Migrate OpenShift SCCs to Policy Controller constraints

This section describes how you can export SCCs from your OpenShift cluster and configure target GKE Enterprise Policy Controller constraints to match the required policies. This section also describes some differences between SCCs and Policy Controller constraints so that you can plan your migration accordingly.

Assess OpenShift SCCs

To export a list and configuration of the SCCs that are installed in your OpenShift cluster, you can use the following commands:

  1. Get a list of all SCCs:

    oc get scc
    
  2. Export the configuration of each SCC:

    oc get scc SCC_NAME > SCC_NAME.yaml
    

    Replace SCC_NAME with the name of the SCC for which you want to export the configuration.

After you export the configuration, you can analyze it and use the table in the following Map OpenShift SCCs section to configure Policy Controller constraints that match your application security requirements.

Map OpenShift SCCs to Policy Controller constraints templates

The following table provides the Policy Controller constraints and settings that correspond to OpenShift SCC fields and their possible values. Use the table to help configure target Policy Controller constraints that match the application security requirements that were implemented through OpenShift SCCs in the source environment. The Example end-to-end migration section later in this document provides an example of how to use the information in the table.

OpenShift SCC field Type / possible values Policy Controller constraint template Policy Controller constraint specification
allowPrivilegedContainer: Boolean K8sPSPPrivilegedContainer Prevents privileged containers if applied.
allowHostIPC: Boolean K8sPSPHostNamespace Prevents access to host pid and ipc namespace if applied.
allowHostPID: Boolean K8sPSPHostNamespace Prevents access to host pid and ipc namespace if applied.
allowHostNetwork: Boolean K8sPSPHostNetworkingPorts Has a boolean parameter to prevent access to host network and can define a range for accessible host ports.
allowHostPorts: Boolean K8sPSPHostNetworkingPorts Has a boolean parameter to prevent access to host network and can define a range for accessible host ports.
readOnlyRootFilesystem: Boolean K8sPSPReadOnlyRootFilesystem Lets you mount container root file system only as read-only if applied.
allowPrivilegeEscalation: true Boolean K8sPSPAllowPrivilegeEscalationContainer Prevents Pods with AllowPrivilegeEscalation security context set to true if applied.
allowHostDirVolumePlugin: Boolean K8sPSPVolumeTypes Has a parameter to define list of allowed volume types (as in the SCC) including host directory.
volumes: Array list with type of volumes allowed K8sPSPVolumeTypes Has a parameter to define list of allowed volume types (as in the SCC) including host directory.
allowedCapabilities: Array list of Linux capabilities that can be requested. K8sPSPCapabilities Has parameters to define Linux capabilities that can be requested (allowedCapabilities) and that are forbidden (requiredDropCapabilites).

Cannot be used to directly add or drop the listed capabilities as can be done in defaultAddCapabilities: and requiredDropCapabilities: in OpenShift SCCs.

defaultAddCapabilities: Array list of Linux capabilities that must be added to each container. K8sPSPCapabilities Has parameters to define Linux capabilities that can be requested (allowedCapabilities) and that are forbidden (requiredDropCapabilites).

Cannot be used to directly add or drop the listed capabilities as can be done in defaultAddCapabilities: and requiredDropCapabilities: in OpenShift SCCs.

requiredDropCapabilities: Array list of Linux capabilities that are automatically dropped from the Pod or container. K8sPSPCapabilities Has parameters to define Linux capabilities that can be requested (allowedCapabilities) and that are forbidden (requiredDropCapabilites).

Cannot be used to directly add or drop the listed capabilities as can be done in defaultAddCapabilities: and requiredDropCapabilities: in OpenShift SCCs.

fsGroup: Has a type: key that can be one of the following:
  • MustRunAs: Requires at least one range to be specified if not using pre-allocated values.
  • RunAsAny: Allows any fsGroup ID to be specified.
K8sPSPAllowedUsers Lets you define rules that have a similar function to the type: key in SCC and id ranges for the runAsUser, runAsGroup, supplementalGroups, and fsGroup parameters.

Can be used to define allowed ranges for User, Groups, Supplemental Groups, or FS Groups but not to set the user ID directly in the Pod as can be done in OpenShift SCCs.

runAsUser: Has a type: key that can be one of the following:
  • MustRunAs: Requires a runAsUser to be configured.
  • MustRunAsRange: Requires minimum and maximum values to be defined if not using pre-allocated values from the namespace.
  • MustRunAsNonRoot: Requires that the Pod be submitted with a non-zero runAsUser or have the USER directive defined in the image.
  • RunAsAny: Allows any runAsUser to be specified.
K8sPSPAllowedUsers Lets you define rules that have a similar function to the type: key in SCC and id ranges for the runAsUser, runAsGroup, supplementalGroups, and fsGroup parameters.

Can be used to define allowed ranges for User, Groups, Supplemental Groups, or FS Groups but not to set the user ID directly in the Pod as can be done in OpenShift SCCs.

supplementalGroups: Has a type: key that can be one of the following:
  • MustRunAs: Requires at least one range to be specified if not using pre-allocated values from the namespace
  • RunAsAny: Allows any supplementalGroups to be specified.
K8sPSPAllowedUsers Lets you define rules that have a similar function to the type: key in SCC and id ranges for the runAsUser, runAsGroup, supplementalGroups, and fsGroup parameters.

Can be used to define allowed ranges for User, Groups, Supplemental Groups, or FS Groups but not to set the user ID directly in the Pod as can be done in OpenShift SCCs.

seLinuxContext: Has a type: key that can be one of the following:
  • MustRunAs: Requires seLinuxOptions to be configured if not using pre-allocated values from the namespace.
  • RunAsAny: Allows any seLinuxOptions to be specified.
K8sPSPSELinuxV2 Has an allowedSELinuxOptions parameter where you can set the level, role, type and user seLinuxOptions allowed

Differences between OpenShift SCCs and Policy Controller constraints

This section describes some differences between Policy Controller constraints and OpenShift SCCs. Consider these differences before you use the preceding table to deploy constraints on your target environment.

How constraints are applied to resources

You can assign OpenShift SCCs to users and groups using the users: or group: specification that is present in the SCC object. With OpenShift 4.x and later versions, you can also assign SCCs to users or groups using role-based access control (RBAC). SCCs also have a priority: field that is used to order the SCCs that are applied to a Pod.

Policy Controller constraints are applied to target clusters, namespaces, or pods using specific resource selectors in the constraint, instead of targeting the user or service account. For more information, see the Policy Controller documentation. Using the specific resource selectors helps to ensure that the Pod behaves the same whether a low-privilege user runs it using a deployment tool or a cluster-admin launches it from the command line.

Policy Controller constraints also support a dry-run mode, which lets you test policies and audit violations before actual enforcement. Using this mode helps you to prevent impact on existing workloads, while SCCs are always enforced if applicable to the user.

SCC Pod mutation using pre-allocated values in OpenShift namespaces

In OpenShift SCCs, you can alter the related security context of each Pod to which the SCC is applied with a specific ID from a pre-allocated range provided by annotations in namespaces. You do this using the RunAsUser, fsGroup, supplementalGroups, and seLinuxContext fields with a MustRunAs or MustRunAsRange strategy type.

For example, consider a restricted SCC that has a RunAsUser field with a strategy type MustRunAsRange with no range defined in the SCC. In this scenario, each Pod the SCC applies to gets a RunAsUser ID from the range specified in the openshift.io/sa.scc.uid-range annotation in the namespace of the Pod.

Policy Controller constraints along with mutation capabilities provide both Pod validation and mutation. However, the constraints don't use annotations in namespaces to provide values for Pod security contexts. The next section, Example of end-to-end migration, provides an example of how your application delivery teams must explicitly configure Security Contexts on Pods to obey constraints that implement restrictions similar to those listed earlier.

Example of end-to-end migration

This section provides an example target manifest file that includes all the Policy Controller constraints and mutators that you need to map the following OpenShift default SCCs on your target GKE cluster:

  • privileged
  • anyuid
  • nonroot
  • restricted

Depending on the namespace that a Pod runs in, the Pod gets different policy controller constraints when you map the SCC policies that are defined in a source OpenShift environment:

  • Workloads that need the highest privileged access, such as the ability to run in privileged mode or access any host resource, should run in one of the exempt namespaces that are defined in Policy Controller configuration.

    No constraints are applied to exempt namespaces. Workloads that have this highest privileged access are typically system components or any workload to which the privileged SCC was applied in the source OpenShift environment.

  • All Pods that are created in any non-exempt namespace get the most restrictive constraints. These constraints deny access to all host features and require the Pods to run with a UID that is part of a specific range. This configuration matches the policies that are applied by OpenShift restricted SCC. Exceptions to this configuration include the following:

    • Pods that are created in a namespace that have the security=anyuid label get the preceding restrictive constraints, but are allowed to run with any UID and any GID. This matches the constraints of the anyuid SCC on OpenShift.
    • Pods that are created in a namespace that has the security=nonroot label get the preceding restrictive constraints. However, the Pods are allowed to run with any non-root UID. This matches the constraints of the nonroot SCC on OpenShift.

Example target manifest

The following is an example of a single manifest that includes a set of Policy Controller constraints and mutators that match the behavior described in the preceding end-to-end migration example. We recommend that you review and adjust the constraints or their scope in this example based on your organization's needs.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPHostNamespace
metadata:
  name: psp-host-namespace
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPHostNetworkingPorts
metadata:
  name: psp-host-network-ports
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    hostNetwork: false
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer
metadata:
  name: psp-privileged-container
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: restricted-capabilities
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["*"]
      kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: NotIn
          key: security
          values: ["anyuid"]
  location: "spec.containers[name:*].securityContext.capabilities.drop"
  parameters:
    assign:
      value: ["KILL","MKNOD","SYS_CHROOT"]
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPCapabilities
metadata:
  name: restricted-capabilities
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: NotIn
          key: security
          values: ["anyuid"]
  parameters:
    requiredDropCapabilities: ["KILL","MKNOD","SYS_CHROOT"]
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: anyuid-capabilities
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["*"]
      kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: In
          key: security
          values: ["anyuid"]
  location: "spec.containers[name:*].securityContext.capabilities.drop"
  parameters:
    assign:
      value: ["MKNOD"]
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPCapabilities
metadata:
  name: anyuid-capabilities
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: In
          key: security
          values: ["anyuid"]
  parameters:
    requiredDropCapabilities: ["MKNOD"]
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
  name: restricted-users-and-groups
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: NotIn
          key: security
          values: ["anyuid","nonroot"]
  parameters:
    runAsUser:
      rule: MustRunAs # MustRunAsNonRoot # RunAsAny
      ranges:
        - min: 1000
          max: 2000
    fsGroup:
      rule: MustRunAs # MayRunAs # RunAsAny
      ranges:
        - min: 1000
          max: 2000
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
  name: nonroot-users-and-groups
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaceSelector:
      matchExpressions:
        - operator: In
          key: security
          values: ["nonroot"]
  parameters:
    runAsUser:
      rule: MustRunAsNonRoot
    fsGroup:
      rule: MustRunAsNonRoot
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPVolumeTypes
metadata:
  name: psp-volume-types
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    volumes:
      - configMap
      - downwardAPI
      - emptyDir
      - nfs
      - persistentVolumeClaim
      - projected
      - secret

The restricted-users-and-groups constraint in the example manifest uses the K8sPSPAllowedUsers template to explicitly set an example range of 1000-2000 for runAsUser: and fsGroup: parameters. Any Pod that isn't set to use an ID in that range for runAsUser: and fsGroup: is blocked.

GKE Enterprise and Kubernetes don't use namespace annotations to automatically mutate Pods with a specific user or group ID. Therefore, to limit the UID range as in the preceding example, your application delivery teams need to explicitly set a compliant UID on the created pod, or you need to completely remove the constraint to allow any ID.

The following is an example of a Pod manifest that complies with the preceding constraints in any namespace that you create it in (the manifest is compliant with the restricted SCC):

apiVersion: v1
kind: Pod
metadata:
  name: restricted-pod-example
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1100
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo

What's next

  • Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.