Learning Path: Scalable applications - Centralize changes with Config Sync

This set of tutorials is for IT administrators and Operators that want to deploy, run, and manage modern application environments that run on Google Kubernetes Engine (GKE) Enterprise edition. As you progress through this set of tutorials you learn how to configure monitoring and alerts, scale workloads, and simulate failure, all using the Cymbal Bank sample microservices application:

Create a cluster and deploy a sample application
Monitor with Google Cloud Managed Service for Prometheus
Scale workloads
Simulate a failure
Centralize change management (this tutorial)

Overview and objectives

As you build new services and applications, you might want to test changes in different environments. As your organization grows, you might need different cluster configurations for different teams. Managing multiple clusters with different configurations can be challenging. You can use GitOps tools like Config Sync to help you manage these challenges.

In the Create a cluster tutorial, you created a cluster and deployed the Cymbal Bank application to that cluster.

In this tutorial, you learn how to store the Kubernetes manifests for your application in a centralized Git repository and how to use tools like Config Sync to deploy an application to multiple clusters in a fleet. You learn how to complete the following tasks:

Create a Git repository and connect it to Cloud Build
Create a cluster, register it to a fleet, and install Config Sync on your fleet of clusters
Use a fleet package to deploy Cymbal Bank and other resources on a cluster or across a fleet

Create clusters

In the first tutorial in this series, you created one cluster and deployed the Cymbal Bank application to that cluster. In a real scenario, it's unlikely that you would have only one cluster to manage. GKE lets you group together clusters in a fleet: a logical group of clusters that can be managed together. Within a fleet, you might further group your clusters with some clusters representing different environments or belonging to different teams. For example, you might have a development cluster, a staging cluster, and a production cluster. In a large organization, individual teams might have their own clusters for different environments. As IT administrators or Operators, this might mean you have to manage dozens of clusters!

When it comes to deploying applications, or individual resources like custom policies, across all of these clusters, GKE Enterprise features like Config Sync can help you manage those deployments at scale.

To help demonstrate how to deploy resources to different environments or across a fleet of clusters, you create a new cluster and deploy the Cymbal Bank application to it:

Create a GKE cluster that simulates a development environment:
```
gcloud container clusters create-auto scalable-apps-dev \
  --project=PROJECT_ID \
  --region=REGION \
  --fleet-project=PROJECT_ID \
  --release-channel=rapid
```
Replace the following:
- PROJECT_ID with the automatically-generated ID of the project that you created in the previous section. The project ID is often different from the project name. For example, your project might be scalable-apps, but your project ID might be scalable-apps-567123.
- REGION with the region that you want to create your cluster in, such as us-central1.
Labels are key-value pairs that you can add to GKE resources to help organize them. For fleet packages, you can use fleet membership labels to customize which clusters the fleet package targets. Other types of labels aren't supported.

Add a label to the fleet membership:
```
gcloud container fleet memberships update scalable-apps-dev \
    --update-labels=env=dev
```
When you create the fleet package later in this tutorial, this label ensures that the resources are only deployed on the scalable-apps-dev cluster and not the scalable-apps cluster from the first tutorial in this series.

Install Config Sync

Install Config Sync on both clusters:

In the Google Cloud console, go to the Config page under the Features section.

Go to Config

Click Install Config Sync.
Keep Manual upgrades selected.
The version menu should default to the latest version of Config Sync. If needed, select the most recent version.
Under Installation options, select Install Config Sync on entire fleet (recommended).
Click Install Config Sync.
In the Settings tab for Config Sync, look at the Status field of the cluster list. After a few minutes, the status shows as "Pending" until the cluster is correctly configured for Config Sync. It can take up to 10 minutes for the status to change to Enabled.

Deploy Cymbal Bank

In the first tutorial in this series, you used kubectl commands to apply the application configurations to your cluster. In this tutorial, you use Config Sync's fleet packages feature to deploy those same configurations to a new cluster. In a later section, you'll see an example of how to add a new resource to your Git repository and deploy those resources across a fleet of clusters.

Set up a service account for Cloud Build

A service account is a special kind of account typically used by an application, rather than a person. As a best practice when creating service accounts, you should create service accounts for a single, specific service or task, and give the service account granular roles. In this tutorial, you create a service account to grant Cloud Build permissions to fetch Kubernetes resources from your Git repository and deploy them to your clusters.

To create the service account and grant the required permissions, complete the following steps:

Create the service account:

gcloud iam service-accounts create "cymbal-bank-service-account"

Grant the service account permission to fetch resources from your Git repository by adding an IAM policy binding for the Resource Bundle Publisher role:
```
gcloud projects add-iam-policy-binding PROJECT_ID \
   --member="serviceAccount:cymbal-bank-service-account@PROJECT_ID." \
   --role='roles/configdelivery.resourceBundlePublisher'
```
If prompted, select None as the condition for the policy.

Grant the service account permission to write logs by adding an IAM policy binding for the Logs Writer role:

gcloud projects add-iam-policy-binding PROJECT_ID \
   --member="serviceAccount:cymbal-bank-service-account@PROJECT_ID." \
   --role='roles/logging.logWriter'

If prompted, select None as the condition for the policy.

Create a release for the Cymbal Bank application

A fleet package requires a semantic version tag to know which version of your repository to deploy from. It's a good idea to create new releases when you make changes to your repository, as this helps with version control and makes it easier to roll back to a stable version if needed.

In a web browser window of your GitHub fork of Cymbal Bank, under the Releases section of the sidebar, click Create a new release.
Select the Choose a tag menu and type v1.0.0 as the tag. Click Create new tag.
Click Publish release.

Set up authentication

Just like the first tutorial in this series, you must create a JWT to handle user authentication and a Kubernetes Secret to store the JWT for the new cluster that you created. It's a good idea for each cluster to have a unique JWT for authentication.

In your Cloud Shell, create the JWT:

openssl genrsa -out jwtRS256.key 4096
openssl rsa -in jwtRS256.key -outform PEM -pubout -out jwtRS256.key.pub

Create the Kubernetes Secret:

kubectl create secret generic jwt-key --from-file=./jwtRS256.key --from-file=./jwtRS256.key.pub

Deploy the Cymbal Bank application with a fleet package

A FleetPackage resource is a declarative API to deploy multiple Kubernetes manifests to a fleet of clusters.

To create a fleet package, you define a FleetPackage spec that points to the repository with your Kubernetes resources that you connected to Cloud Build. Then you apply the FleetPackage resource, which fetches the resources from Git and deploys them across the fleet.

In your Cloud Shell, set the default location for the configdelivery Google Cloud CLI commands. As with connecting your repository to Cloud Build, you must use us-central1 while the fleet packages feature is in preview:
```
gcloud config set config_delivery/location us-central1
```
Create a file named fleetpackage-spec.yaml with the following content:
```
resourceBundleSelector:
  cloudBuildRepository:
    name: projects/PROJECT_ID/locations/us-central1/connections/cymbal-bank-connection/repositories/REPOSITORY_NAME
    tag: v1.0.0
    serviceAccount: projects/PROJECT_ID/serviceAccounts/cymbal-bank-service-account@PROJECT_ID.
    path: kubernetes-manifests
target:
  fleet:
    project: projects/PROJECT_ID
    selector:
      matchLabels:
        env: dev
rolloutStrategy:
  rolling:
    maxConcurrent: 1
```
Replace REPOSITORY_NAME with the name of your repository, as it appears in the Cloud Build connection.

The selector fields match the fleet membership label that you created earlier. This ensures that the Cymbal Bank application is only deployed on the scalable-apps-dev cluster. The scalable-apps cluster from the first tutorial is unaffected. In the next section, you'll see an example of a fleet package that targets all clusters in a fleet.

The rollout strategy field controls how resources are deployed across clusters. In this example, you're deploying only to one cluster so this field doesn't change how the rollout proceeds. But if you had many clusters, this setting ensures that the resource files are all applied to one cluster before moving on to the next cluster. This lets you monitor how the rollout is progressing.

Create the fleet package:

gcloud alpha container fleet packages create cymbal-bank-fleet-package \
    --source=fleetpackage-spec.yaml \
    --project=PROJECT_ID

Verify that the fleet package was created:
```
gcloud alpha container fleet packages list
```
The output lists the status of the build trigger. After a few seconds, the MESSAGE field updates with output similar to the following:
```
MESSAGES: Build status: WORKING. The release is still being built; see the build status on the following page:
```
You can click the link provided to view the streaming logs for the Cloud Build job. It can take a few minutes for Cloud Build to process the build trigger.
When the build trigger successfully completes, the output of gcloud alpha container fleet packages list is similar to the following:
```
NAME: cymbal-bank-fleet-package
STATE: ACTIVE
CREATE_TIME: 2024-07-09T15:15:56
ACTIVE_ROLLOUT: rollout-20240709-153621
LAST_COMPLETED_ROLLOUT:
MESSAGES:
```
The fleet package starts rolling out the Kubernetes resources across your fleet.
In the Google Kubernetes Engine page of the Google Cloud console, select your scalable-apps-dev cluster, then go to the Workloads page to see an aggregated view of the workloads that are being deployed on all your GKE clusters:

Open the Workloads page

You might see some errors while Autopilot adjusts your Pods to meet resource requests. After a few minutes, you should see your Pods start running with the status OK.
To view the Cymbal Bank web interface, complete the following steps:
1. In the Google Kubernetes Engine page of the Google Cloud console, go to the Gateways, Services & Ingress page.
  
  Go to the Gateways, Services & Ingress page
2. To find the Cymbal Bank ingress, click the tab for "Services" and find the service with the name frontend.
3. Click the Endpoint link for the frontend ingress, such as 198.51.100.143:80, to open to the Cymbal Bank web interface.

Deploy resources across a fleet

Next, imagine a scenario where you want to extend your Cymbal Bank application with a new microservice. You want to deploy this microservice across all your current clusters and any future clusters added to the fleet. By using a fleet package, you can deploy to multiple clusters and get automatic deployment on new clusters.

Add new resources to your Git repository

To demonstrate adding a new service to your application, you create a basic nginx deployment and add it to your clusters:

In a web browser window of your GitHub fork of Cymbal Bank, click Add file and then Create new file.

Name your file new-resource/nginx.yaml and paste the following contents into it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.14.2
        name: nginx
        ports:
        - containerPort: 80

Click Commit changes...
In the confirmation dialog, keep Commit directly to the main branch selected and then click Commit changes.

Note: In a real scenario, you'd probably create a new pull request instead so that your change can be reviewed and approved.
On the main page of your forked Cymbal Bank repository, select Releases from the sidebar.
At the top of the page, choose Draft a new release.
Select the Choose a tag menu and type v1.1.0 as the tag. Click Create new tag.
Click Publish release.

Deploy a resource to clusters with a fleet package

To deploy the new resource, create a new fleet package:

This fleet package targets all of the clusters in your fleet since it doesn't contain a selector field. This also means any future clusters added to the fleet will have the nginx deployment automatically added.

In your Cloud Shell, create a file named new-deployment-fleet-package.yaml with the following content:

resourceBundleSelector:
  cloudBuildRepository:
    name: projects/PROJECT_ID/locations/us-central1/connections/cymbal-bank-connection/repositories/REPOSITORY_NAME
    tag: v1.1.0
    serviceAccount: projects/PROJECT_ID/serviceAccounts/cymbal-bank-service-account@PROJECT_ID.
    path: kubernetes-manifests/new-resource
target:
  fleet:
    project: projects/PROJECT_ID
rolloutStrategy:
  rolling:
    maxConcurrent: 1

Create the fleet package to start the rollout:

gcloud alpha container fleet packages create new-deployment-fleet-package \
    --source=new-deployment-fleet-package.yaml \
    --project=PROJECT_ID

Verify that the fleet package was created:
```
gcloud alpha container fleet packages list
```
You can click the link provided to view the streaming logs for the Cloud Build job.

The fleet package starts rolling out the Kubernetes resources across your fleet. It can take a few minutes for the rollout to begin and complete.
In the Google Kubernetes Engine page of the Google Cloud console, go to the Workloads page to see an aggregated view of the workloads that are being deployed on all your GKE clusters:

Open the Workloads page

You can continue to explore different deployment strategies with fleet packages. For example, you can try adding different types of resources to your forked repository and using different fleet package configurations to deploy them. You can also use a fleet package to delete any resources you deployed across your clusters. For more information about fleet packages, see Deploy fleet packages in the Config Sync documentation.

Delete resources across a fleet

Just like you can deploy resources across a fleet, you can delete resources across a fleet with a fleet package.

To remove individual resources, the high-level steps are the following:

Delete the resource from your Git repository.
Create a new Git release and create a new tag.
Update the fleet package tag field.
Run the fleet package update command.

Alternatively, you can delete the fleet package itself, which also deletes any resources that were managed by the fleet package.

For example, if you want to remove the nginx deployment from the previous section, run the following command:

gcloud alpha container fleet packages delete new-deployment-fleet-package --force