Using Config Sync in multiple environments with automated rendering

This tutorial shows you how to set up Config Sync for Google Kubernetes Engine (GKE) across two environments, one for development and one for production, using best practices for Anthos Config Management.

In this scenario, you're part of a platform admin team at Foo Corp. The Foo Corp applications are deployed to GKE, with resources divided across two projects, dev and prod. The dev project contains a development GKE cluster, and the prod project contains the production GKE cluster. Your goal as the platform admin is to ensure that both environments stay within compliance of Foo Corp's policies, and that base level resources, like Kubernetes namespaces and service accounts, remain consistent across both environments.

The following diagram shows you an overview of the environments that you set up in this tutorial:

An overview of the environments that you set up in this tutorial.

As shown in the preceding diagram, in this tutorial you create the following resources:

  • Two Google Cloud projects representing the development and production environments.
  • Two GKE clusters, dev and prod, in the separate projects, with Config Sync installed.

Repository architecture

In this tutorial, you configure Config Sync to sync to the configs in the config-source/ directory of the Anthos Config Management samples repository. This directory contains the following directories and files:

config-source/
├── base
│   ├── foo
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   └── serviceaccount.yaml
│   ├── kustomization.yaml
│   ├── pod-creator-clusterrole.yaml
│   └── pod-creator-rolebinding.yaml
├── cloudbuild.yaml
├── overlays
│   ├── dev
│   │   └── kustomization.yaml
│   └── prod
│       └── kustomization.yaml
└── README.md

The config-source directory includes the base/ manifests and the dev/ and prod/ Kustomize overlays. Each directory contains a kustomization.yaml file, which lists the files Kustomize should manage and apply to the cluster. In dev/kustomization.yaml and prod/kustomization.yaml a series of patches are defined. These patches manipulate the base/ resources for that specific environment.

For example, the dev RoleBinding allows all Foo Corp developers deploy Pods to the dev cluster, whereas the prod RoleBinding only allows a Continuous Deployment agent, deploy-bot@foo-corp.com, to deploy Pods into production:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patches:
# ServiceAccount - make name unique per environ
- target:
    kind: ServiceAccount
    name: foo-ksa
  patch: |-
    - op: replace
      path: /metadata/name
      value: foo-ksa-dev
    - op: replace
      path: /metadata/namespace
      value: foo-dev
# Pod creators - give all Foo Corp developers access
- target:
    kind: RoleBinding
    name: pod-creators
  patch: |-
    - op: replace
      path: /subjects/0/name
      value: developers-all@foo-corp.com
commonLabels:
  environment: dev

Objectives

  • Set up Config Sync to automatically render and sync the configuration for the two separate environments.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

Before you begin this tutorial, make sure that you have completed the following steps:

  1. In the Google Cloud Console, on the project selector page, choose or create two Google Cloud projects.

    Go to project selector

  2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  3. Upgrade your gcloud command-line tool to the latest version.

  4. Install or upgrade the nomos command

Create and register your clusters

To let you focus on the workflow that you need to use when you configure Config Sync for multiple environments, the multi-environments-kustomize directory contains scripts that you can use to automate the configuration of Config Sync.

  1. Clone the Anthos Config Management sample repository:

    git clone https://github.com/GoogleCloudPlatform/anthos-config-management-samples.git
    
  2. Navigate to the folder that contains the resources that you need for this tutorial:

    cd anthos-config-management-samples/multi-environments-kustomize/
    
  3. To run the scripts used in this tutorial, set the following variables:

    export DEV_PROJECT="DEV_PROJECT_ID"
    export PROD_PROJECT="PROD_PROJECT_ID"
    export DEV_CLUSTER_ZONE="DEV_CLUSTER_ZONE"
    export PROD_CLUSTER_ZONE="PROD_CLUSTER_ZONE"
    export CM_CONFIG_DIR="config-sync-rendering"
    

    Replace the following:

    • DEV_PROJECT_ID: the project ID from the Google Cloud project that you want you use as your dev project
    • PROD_PROJECT_ID: the project ID from the Google Cloud project that you want you use as your prod project
    • DEV_CLUSTER_ZONE: the Compute Engine zone that you want to create your dev cluster in. For example, us-central1-c.
    • PROD_CLUSTER_ZONE: the Compute Engine zone that you want to create your prod cluster in
  4. To create two clusters, run the ./create-clusters.sh script:

    ./create-clusters.sh
    

    This script creates one GKE cluster named dev in the dev project and one GKE cluster named prod in the prod project. This script also enables the GKE and Anthos APIs, and connects to your dev and prod clusters so that you can access their APIs with kubectl.

    Example output:

    kubeconfig entry generated for dev.
    Fetching cluster endpoint and auth data.
    kubeconfig entry generated for prod.
    ⭐️ Done creating clusters.
    
  5. To register your clusters to two separate Anthos fleets, run the register-clusters.sh script:

    ./register-clusters.sh
    

    This script creates a Google Cloud service account and key for Anthos cluster registration, then uses the gcloud container hub memberships register command to register the dev and prod clusters to Anthos in their own projects.

    Example output:

    Waiting for Feature Config Management to be created...done.
    ⭐️ Done registering clusters.
    

Setting up Config Sync

Now that you have created and registered your clusters, you can install Config Sync and verify the installation.

Install Config Sync

To install Config Sync, on the dev and prod cluster, run the install-config-sync.sh script:

./install-config-sync.sh

Expected output:

🔁 Installing ConfigSync on the dev cluster...
Updated property [core/project].
Switched to context "DEV_CLUSTER".
Waiting for Feature Config Management to be updated...done.
🔁 Installing ConfigSync on the prod cluster...
Updated property [core/project].
Switched to context "PROD_CLUSTER".
Waiting for Feature Config Management to be updated...done.

Config Sync is now syncing to the configs in your repositories.

Verify your configuration

In this section you check to make sure that your clusters are syncing to the configs in your repository:

  1. To check the state of your Config Sync installation, run the nomos status command:

    nomos status
    

    You should see that both your dev and prod clusters are now synced to their respective repos:

    gke_DEV_PROJECT_ID_us-central1-c_dev
      --------------------
      <root>   https://github.com/GoogleCloudPlatform/anthos-config-management-samples/multi-environments-kustomize/config-source/overlays/dev@main
      SYNCED   8f2e196f
      Managed resources:
         NAMESPACE   NAME                                                 STATUS
                     clusterrole.rbac.authorization.k8s.io/pod-creator    Current
                     namespace/default                                    Current
                     namespace/foo                                        Current
         default     rolebinding.rbac.authorization.k8s.io/pod-creators   Current
         foo         serviceaccount/foo-ksa-dev                           Current
    
    *gke_PROD_PROJECT_ID_us-central1-c_prod
       --------------------
       <root>   https://github.com/GoogleCloudPlatform/anthos-config-management-samples/multi-environments-kustomize/config-source/overlays/prod@main
       SYNCED   c91502ee
       Managed resources:
          NAMESPACE   NAME                                                 STATUS
                      clusterrole.rbac.authorization.k8s.io/pod-creator    Current
                      namespace/default                                    Current
                      namespace/foo                                        Current
          default     rolebinding.rbac.authorization.k8s.io/pod-creators   Current
          foo         serviceaccount/foo-ksa-prod                          Current
    
      ```
    
  2. Use kubectl to switch to the dev cluster:

    kubectl config use-context "gke_${DEV_PROJECT}_${DEV_CLUSTER_ZONE}_dev"
    
  3. To verify that the resources are synced, get namespaces. You should see the foo namespace appear.

    kubectl get namespace
    

    Example output:

    NAME                           STATUS   AGE
    config-management-monitoring   Active   9m38s
    config-management-system       Active   9m38s
    default                        Active   47h
    foo                            Active   9m5s
    kube-node-lease                Active   47h
    kube-public                    Active   47h
    kube-system                    Active   47h
    resource-group-system          Active   9m30s
    

    You have now set up automated config rendering for a dev and prod environment, across multiple Google Cloud projects and environments.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete all resources

To delete the resources that you created in this tutorial, but to keep both the dev and prod projects intact, run the cleanup script:

./cleanup.sh

Delete the projects

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next