Modern CI/CD with Anthos: Apply the developer workflow

---

This tutorial shows how to onboard a new application, develop a feature for the application, and deploy the application to production using modern continuous integration/continuous delivery (CI/CD) techniques with Anthos.

This document is part of a series:

• Modern CI/CD with Anthos: A software delivery framework
• Modern CI/CD with Anthos: Build a CI/CD system (reference architecture)
• Modern CI/CD with Anthos: Apply the developer workflow (this document)

In this tutorial, you use tools such as Skaffold, kustomize, Artifact Registry, and GitLab to develop and deploy your application.

This document is intended for enterprise architects and application developers, as well as IT security, DevOps, and Site Reliability Engineering (SRE) teams. Some experience with automated deployment tools and processes is useful for understanding the concepts in this document.

Architecture

In this tutorial, you use Skaffold to develop a new feature and then deploy it to pre-production and production environments by using a GitOps-based process. The reference architecture contains the infrastructure and tools needed to implement the GitOps-style code promotion and deployment workflow shown in the following diagram:

Starting at the code repository for CI, the workflow includes the following steps:

1. You share your application source code through your application repositories.
2. Using the development cluster, you develop and test your changes in an iterative development loop.
3. You merge your code changes from the feature branch into the main code line, triggering CI. The CI process creates and pushes a container image to Artifact Registry and generates and pushes fully rendered Kubernetes manifests.
4. The rendered manifests are pushed into the staging branch of the application's environment repository, triggering the deployment to the staging cluster.
5. The rendered manifests are merged from the staging branch to the main branch, triggering the deployment of the code changes to the production clusters.

For more information about the tools and infrastructure used in this workflow, see Modern CI/CD with Anthos: Build a CI/CD system.

Objectives

• Onboard a new application.
• Use Skaffold to develop a new feature and then test it live.
• Deploy the new feature to a pre-production environment.
• Use GitOps to deploy the new feature to multiple production clusters.

Costs

In this document, you use the following billable components of Google Cloud:

• Google Kubernetes Engine
• Cloud SQL
• Memorystore
• Artifact Registry

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

• For this tutorial, deploy the reference architecture in this series.

Prepare your environment

1. If you're continuing directly from Modern CI/CD with Anthos: Build a CI/CD system, go to the next step. However, if you have a new session or your session expired, in Cloud Shell, define the environment variables that you set in the reference architecture:

   export PROJECT_ID=PROJECT_ID
   export REGION="us-central1"
   
   gcloud config set compute/region ${REGION}
   gcloud config set core/project ${PROJECT_ID}
   
   export GITLAB_HOSTNAME=gitlab.endpoints.${PROJECT_ID}.cloud.goog
   export GITLAB_USER=$(gcloud secrets versions access latest --secret="gitlab-user")
   export GITLAB_PASSWORD=$(gcloud secrets versions access latest --secret="gitlab-password")
   

   Replace PROJECT_ID with your Google Cloud project ID.

2. If you've never used Git in Cloud Shell, configure Git with your name and email address. Git uses this information to identify you as the author of the commits that you create in Cloud Shell.

   git config --global user.email "GIT_EMAIL_ADDRESS"
   git config --global user.name "GIT_USERNAME"
   

   Replace the following:

   • GIT_EMAIL_ADDRESS: the email address associated with your Git account
   • GIT_USERNAME: the username associated with your Git account

3. Install kustomize.

Onboard a new application

The reference architecture contains a command-line interface (CLI), anthos-platform-cli, that streamlines application management processes, such as creating and deleting applications. anthos-platform-cli uses starter repositories to initialize the new application and pull in best practices that the operation and platform administration teams manage. The CLI also establishes the landing zone for your new application by defining the namespace and base policies in the anthos-config-management repository.

1. In Cloud Shell, build the anthos-platform-cli CLI:

   cd $HOME/solutions-modern-cicd-anthos/cli
   make release
   mv release/anthos-platform-cli-vlatest-linux-amd64 $HOME/anthos-platform-cli
   

2. Create an application:

   ./anthos-platform-cli add app \
       --name hello-world-golang \
       --gitlab-hostname $GITLAB_HOSTNAME \
       --gitlab-token $GITLAB_PASSWORD \
       --template-name golang-template
   mv hello-world-golang hello-world-golang.bak
   

Add a new feature to the application

When you develop a new feature, you need to quickly deploy your changes into a development sandbox in order to test and iterate on them. In this tutorial, you use Skaffold to actively monitor your changes and deploy them to a development sandbox.

Skaffold generates a configuration file named skaffold.yaml. This file defines the Docker images and Kubernetes manifests that you use to deploy the application. When you run skaffold dev, the continuous development loop begins. As you make changes to the application, Skaffold automatically rebuilds the necessary Docker images and deploys the latest version of your development code to the development cluster.

In their work, developers build many images and need a place in a container registry to store them. We recommend that you give each developer access to a subpath in the container registry where they store their development images. For example, your project might give them access to the gcr.io/my-project/alice repository in Container Registry. After you give them administrative access to push and pull images from that repository, they can set the default-repo Skaffold configuration to use that repository for their development images. For more information about image repository handling, see Image Repository Handling.

Connect to the development cluster

1. In Cloud Shell, get credentials to the development cluster:

   gcloud container clusters --region us-west1 get-credentials dev-us-west1
   

2. Rename your context:

   kubectx development=gke_${PROJECT_ID}_us-west1_dev-us-west1
   

3. Switch to your development cluster context:

   kubectx development
   

Set up your development environment

1. In Cloud Shell, clone your application repository:

   git clone git@$GITLAB_HOSTNAME:hello-world-golang/hello-world-golang.git
   cd hello-world-golang
   

2. Create an Artifact Registry repository for your user:

   gcloud beta artifacts repositories create $USER --repository-format=Docker \
       --location=us-central1
   gcloud beta artifacts repositories add-iam-policy-binding \
       --member serviceAccount:tf-sa-dev-us-west1@${PROJECT_ID}.iam.gserviceaccount.com \
       --role roles/artifactregistry.reader \
       --location us-central1 $USER
   

3. Set up Docker authentication to your Artifact Registry repository:

   gcloud beta auth configure-docker us-central1-docker.pkg.dev
   

4. Create your own namespace on the development cluster to serve as your development sandbox:

   kubectl create namespace $USER
   kubens $USER
   

5. Create a service account for your application:

   kubectl create serviceaccount hello-world-golang-ksa
   

6. Configure Skaffold to use your Artifact Registry repository:

   skaffold config set default-repo us-central1-docker.pkg.dev/$PROJECT_ID/$USER
   

7. Start the Skaffold development loop:

   skaffold dev --port-forward
   

8. On the Cloud Shell toolbar, click Web Preview, and then click Preview on port 8080:

   

   The output is the following:

   Hello World!
   

   Keep this Cloud Shell session open.

Update the application

1. On the Cloud Shell toolbar, click Open a new tab to activate a new session.

2. Create a feature branch for your work:

   cd hello-world-golang
   git checkout -b my-feature
   

3. Update the application to output a different message:

   sed -i "s/Hello World/My new feature/g" main.go
   

   Note: Skaffold monitors your working directory for changes and is automatically rebuilding and deploying your application to the development cluster.

   To see your changes, refresh the web preview.

4. In your original Cloud Shell session, press CTRL+C to exit Skaffold.

5. Commit and push your changes into the feature branch:

   git add .
   git commit -m "Changed the message"
   git push -u origin my-feature
   

6. Close the other Cloud Shell session.

Deploy your change to the staging cluster

After you successfully push your changes into the feature branch of the application code repository, you can deploy them to the staging cluster. To deploy to the staging cluster, merge your changes into the main branch of the application repository. This action triggers a CI process to test the code, render the Kubernetes manifests, and push the rendered manifests into the staging branch of the application's configuration repository. When the CI process pushes the manifests into the application's configuration repository, a CD job starts, which deploys the manifests to the staging cluster.

To deploy your changes to the staging cluster, do the following:

1. In a web browser, go to GitLab, and log in using the URL and username and password from the reference architecture.
2. Click Groups, and then select Your Groups.
3. Click hello-world-golang.

4. Click hello-world-golang to go to the application code repository.

   To see your changes in the repository, select your feature branch from the Branches list:

   

5. In the adjacent pane, click Merge Requests, and then click Create merge request to create a merge request.

6. Select the Delete source branch when merge request is accepted option, and then click Submit merge request.

7. Click Merge.

8. In the menu, click CI/CD to view the execution of the CI pipeline.

9. Click Running to get more details.

   Note: When the pipeline is completed, the updated manifests have been pushed into the hello-world-golang-env repository. That push triggered a CD job to deploy your code changes to the staging cluster.

10. Click Groups, and then select Your Groups.

11. Click hello-world-golang.

12. Click hello-world-golang-env to view the application configuration repository.

13. In the menu, click CI/CD to view the execution of the CD pipeline.

    Note: When the pipeline is completed, the code changes have been deployed to the staging cluster.

View the changes on the staging cluster

1. In Cloud Shell, get credentials to the staging cluster:

   gcloud container clusters --region us-west2 get-credentials staging-us-west2
   

2. Rename your context:

   kubectx staging=gke_${PROJECT_ID}_us-west2_staging-us-west2
   

3. Switch to your staging cluster context:

   kubectx staging
   

4. Create a port forward:

   kubectl port-forward svc/hello-world-golang-app -n hello-world-golang 8080:8080
   

5. On the Cloud Shell toolbar, click Web Preview, and then click Preview on port 8080.

   The output is the following:

   My new feature!
   

6. In Cloud Shell, press CTRL+C to end the port forward.

Promote to the production clusters

After you verify your changes in the staging cluster, you are ready to promote the changes to the production clusters. The hello-world-golang-env repository contains a branch for each application environment. Updates to the application configuration automatically trigger the CD pipelines, which deploy the application to the environments associated with the branch—for example, updates to the staging branch trigger a deployment to the staging environment. Storing the application configuration in Git and automating the deployment of the applications form the foundation of the GitOps process. Storing the manifests in Git improves the auditability of configuration changes and deployments.

To trigger the deployment to production, you merge the application configuration from the staging branch into the main branch of the hello-world-golang-env repository.

1. In GitLab, click Groups, and then select Your Groups.
2. Click hello-world-golang.
3. Click hello-world-golang-env.
4. In the adjacent pane, click Merge Requests, and then click Create merge request to create a merge request.
5. Unselect Delete source branch when merge request is accepted.
6. Click Submit merge request.
7. Click Merge.
8. In the menu, click CI/CD to view the pipeline execution.
9. After the deployment to the prod-us-central1 cluster completes, click Play to approve the rollout to the prod-us-east1 cluster.

View the changes on a production cluster

1. In Cloud Shell, get credentials to the prod-us-central1 cluster:

   gcloud container clusters get-credentials prod-us-central1
   

2. Rename your context:

   kubectx prod-central=gke_${PROJECT_ID}_us-central1_prod-us-central1
   

3. Switch to your prod-central cluster context:

   kubectx prod-central
   

4. Create a port forward:

   kubectl port-forward svc/hello-world-golang-app -n hello-world-golang 8080:8080
   

5. On the Cloud Shell toolbar, click Web Preview, and then click Preview on port 8080.

   The output is the following:

   My new feature!
   

6. In Cloud Shell, press CTRL+C to end the port forward.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

• Learn about best practices for setting up identity federation.
• Learn about policy management with Anthos Config Management and GitLab.
• Read Kubernetes and the challenges of continuous software deployment.
• Learn about hybrid and multi-cloud monitoring and logging patterns.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.