Validating your app against company policies in a continuous integration pipeline

If your organization uses Anthos Config Management and Policy Controller to manage policies across its Anthos clusters, then you can validate an app's deployment configuration in its continuous integration (CI) pipeline. This tutorial demonstrates how to achieve this result. It's useful if you are a developer building a CI pipeline for an app, or a platform engineer building a CI pipeline template for multiple app teams.

Policies are an important part of the security and compliance of an organization. Policy Controller, which is part of Anthos Config Management, allows your organization to manage those policies centrally and declaratively for all your clusters. As a developer, you can take advantage of the centralized and declarative nature of those policies. You can use those characteristics to validate your app against those policies as early as possible in your development workflow. Learning about policy violations in your CI pipeline instead of during the deployment has two main advantages: it lets you shift left on security, and it tightens the feedback loop, reducing the time and cost necessary to fix those violations.

If you are in charge of Anthos Config Management and Policy Controller, and want to build a CI pipeline for them (and not for a specific app), see Using Policy Controller in a CI pipeline.

This tutorial uses Cloud Build as a CI tool and a sample GitHub repository containing policies for demonstrations.

Resources

This tutorial uses several tools from the Kubernetes ecosystem. This section explains what those tools are, how they interact with each other, and whether you can replace them with something else. The tools that you use in this tutorial are:

  • Policy Controller: Policy Controller is a Google Cloud product, part of Anthos Config Management. It is based on the open source project Open Policy Agent - Gatekeeper. Policy Controller enforces policies about the objects that are created in a Kubernetes cluster (for example, preventing the usage of a specific option or enforcing the usage of a specific label). Those policies are called constraints. Constraints are defined as Kubernetes Custom Resources. Config Sync (available both as a standalone product and as a part of Anthos Config Management) lets you declare those constraints in a Git repository, and to apply traditional development workflows to your policy management process. You can use Open Policy Agent - Gatekeeper instead of Policy Controller for your implementation.

  • GitHub: In this tutorial, we use GitHub to host the Git repositories: one for a sample app, and one for Anthos Config Management (which contains the constraints for Policy Controller). For simplicity, the two repositories are two different folders in a single Git repository. In reality, they would be different repositories. You can use any Git solution.

  • Cloud Build: Cloud Build is Google Cloud's CI solution. In this tutorial, we use it to run the validation tests. While the details of the implementation can vary from one CI system to another, the concepts outlined in this tutorial can be used with any container-based CI system.

  • Kustomize: Kustomize is a customization tool for Kubernetes configurations. It works by taking "base" configurations and applying customizations to them. It lets you have a DRY (Don't Repeat Yourself) approach to Kubernetes configurations. With kustomize you keep elements that are common to all your environments in the base configurations and create customizations per environment. In this tutorial, we keep the Kustomize configurations in the app repository, and we "build" (i.e apply the customizations) the configurations in the CI pipeline. You can use the concepts outlined in this tutorial with any tool that produces Kubernetes configurations that are ready to be applied to a cluster (for example, the helm template command).

  • Kpt: Kpt is a tool to build workflows for Kubernetes configurations. Kpt lets you fetch, display, customize, update, validate, and apply Kubernetes configurations. Because it works with Git and YAML files, it is compatible with most of the existing tools of the Kubernetes ecosystem. In this tutorial, we use kpt in the CI pipeline to fetch the constraints from the Anthos Config Management repository, and to validate the Kubernetes configurations against those constraints.

Pipeline

The CI pipeline we are using in this tutorial is shown in the diagram below. It runs in Cloud Build, and the commands are run in a directory containing a copy of the sample app repository. The pipeline starts by generating the final Kubernetes configurations with kustomize. Next, it fetches the constraints that we want to validate against from the Anthos Config Management repository using kpt. Finally, it uses kpt to validate the Kubernetes configurations against those constraints. To achieve this last step, we use a specific config function called gatekeeper-validate that performs this validation. In this tutorial, you trigger the CI pipeline manually, but in reality you would configure it to run after a git push to your Git repository.

CI pipeline for Policy Controller

Objectives

  • Run a CI pipeline for a sample app with Cloud Build.
  • Observe that the pipeline fails because of a policy violation.
  • Modify the sample app repository to comply with the policies.
  • Run the CI pipeline again successfully.

Costs

This tutorial uses the following billable components of Google Cloud:

  • Cloud Build

To generate a cost estimate based on your projected usage, use the pricing calculator.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See Cleaning up for more detail.

Before you begin

  1. Select or create a Cloud project.

    GO TO THE MANAGE RESOURCES PAGE

  2. Enable billing for your project.

    ENABLE BILLING

  3. Open Cloud Shell to execute the commands listed in this tutorial.

    GO TO CLOUD SHELL

  4. If the gcloud config get-value project command does not return the ID of the project you just selected, configure Cloud Shell to use your project.

    gcloud config set project PROJECT_ID
    
  5. In Cloud Shell, enable the required Cloud Build API.

    gcloud services enable cloudbuild.googleapis.com
    

Validate the sample app configurations

In this section, you run a CI pipeline with Cloud Build for a sample app repository that we provide. This pipeline validates the Kubernetes configuration available in that sample app repository against constraints available in a sample Anthos Config Management repository.

  1. In Cloud Shell, clone the sample app repository.

    git clone https://github.com/GoogleCloudPlatform/csp-config-management.git
    
  2. Run the CI pipeline with Cloud Build. Logs of the build are displayed directly in Cloud Shell.

    cd csp-config-management/ci-app/app-repo
    gcloud builds submit .
    

    The pipeline you run is defined in the following file.

    steps:
    - id: 'Prepare config'
      # This step builds the final manifests for the app
      # using kustomize and the configuration files
      # available in the repository.
      name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
      entrypoint: '/bin/sh'
      args: ['-c', 'mkdir hydrated-manifests && kubectl kustomize config/prod > hydrated-manifests/prod.yaml']
    - id: 'Download policies'
      # This step fetches the policies from the Anthos Config Management repository
      # and consolidates every resource in a single file.
      name: 'gcr.io/kpt-dev/kpt'
      entrypoint: '/bin/sh'
      args: ['-c', 'kpt pkg get https://github.com/GoogleCloudPlatform/csp-config-management.git/ci-app/acm-repo/cluster@1.0.0 constraints
                      && kpt fn source constraints/ hydrated-manifests/ > hydrated-manifests/kpt-manifests.yaml']
    - id: 'Validate against policies'
      # This step validates that all resources comply with all policies.
      name: 'gcr.io/config-management-release/policy-controller-validate'
      args: ['--input', 'hydrated-manifests/kpt-manifests.yaml']
  3. After a few minutes, observe that the pipeline fails with the following error:

    [...]
    Step #2 - "Validate against policies": gcr.io/config-management-release/policy-controller-validate:latest
    Step #2 - "Validate against policies": Error: Found 1 violations:
    Step #2 - "Validate against policies":
    Step #2 - "Validate against policies": [1] Deployment objects should have an 'owner' label indicating who created them.
    Step #2 - "Validate against policies":
    Step #2 - "Validate against policies": name: "nginx-deployment"
    Step #2 - "Validate against policies": path: prod.yaml
    [...]
    

    The constraint that the configuration is violating is defined in the following file. It's a Kubernetes custom resource called K8sRequiredLabels.

    apiVersion: constraints.gatekeeper.sh/v1beta1
    kind: K8sRequiredLabels
    metadata:
      name: deployment-must-have-owner
    spec:
      match:
        kinds:
          - apiGroups: ["apps"]
            kinds: ["Deployment"]
      parameters:
        labels:
          - key: "owner"
        message: "Deployment objects should have an 'owner' label indicating who created them."

    For the constraint template corresponding to this constraint, see this file on GitHub.

  4. Build the full Kubernetes configuration yourself, and observe that the owner label is indeed missing. To build the configuration:

    kubectl kustomize config/prod
    

Fix the app to comply with company policies

In this section, you fix the policy violation using Kustomize.

  1. In Cloud Shell, add a commonLabels section to the base kustomization file:

    cat <<EOF >> config/base/kustomization.yaml
    commonLabels:
      owner: myself
    EOF
    
  2. Build the full Kubernetes configuration, and observe that the owner label is now present:

    kubectl kustomize config/prod
    
  3. Rerun the CI pipeline with Cloud Build:

    gcloud builds submit .
    

    The pipeline now succeeds with the following output:

    [...]
    Finished Step #2 - "Validate against policies"
    PUSH
    DONE
    [...]
    

Cleaning up

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

What's next