Integrating microservices with Pub/Sub and GKE

In this tutorial, you deploy a photo sharing app to Google Kubernetes Engine (GKE), where you learn how to use Pub/Sub to invoke long-running processes asynchronously. You also learn how to use Pub/Sub notifications for Cloud Storage to add side jobs without modifying the app's application code. The app is containerized by Cloud Build and stored in Container Registry. The app uses Cloud Vision to detect inappropriate images.

When you design a web app based on a microservices architecture, you decide how to split your app features into microservices, and how to call those microservices from your app. For time-consuming services, you might want to use asynchronous service calls. This tutorial is intended for developers and architects who want to implement microservices in an asynchronous manner using modern technologies, including GKE and Pub/Sub.

The State of DevOps reports identified capabilities that drive software delivery performance. This tutorial will help you with the Architecture capability.

Architectural pattern for a thumbnail-generation app

The following diagram illustrates an example scenario where an app generates thumbnail images.

In the preceding diagram, the app receives image files from clients and then automatically generates thumbnails. Currently, this app is implemented with virtual machine (VM) instances on Compute Engine and with backend file storage on Cloud Storage. Cloud Load Balancing distributes requests to multiple VMs.

To reduce the operational overhead to maintain VMs, you migrate this system to a new architecture that doesn't use VMs.

The new architecture, illustrated in the following diagram, consists of several managed services.

In this new architecture, the client sends an image to the app directly to Cloud Storage. Then, Pub/Sub notifications put a message in the Pub/Sub message queue. The message calls a microservice that runs on GKE. The microservice retrieves the image from Cloud Storage, generates a thumbnail, and uploads the thumbnail to Cloud Storage.

This new architecture has the following advantages:

• Independent scalability: In the original architecture, the app running on Compute Engine provides two core features. One receives files, and the other generates a thumbnail from the original image. Receiving uploaded files consumes network bandwidth, while thumbnail generation is a CPU-intensive task. The Compute Engine instances might run out of CPU resources to generate images and still have enough network resources to receive files. In the new architecture, these tasks are shared by Cloud Storage and GKE, making the tasks independently scalable.
• Efficient resource usage: In the original architecture, scaling out Compute Engine instances consumes more resources to run operating systems. With GKE, you can efficiently use server resources that run multiple containers on just a few servers (bin packing). You can scale containers out and in quickly, so the new architecture can handle short burst of high load and scale in quickly when it's finished.
• Easy to add new functionality: In the original architecture, if you want to add functionality, you have to deploy it on the same Compute Engine instances. In the new architecture, you can develop an app, such as a mail sender, to notify you when a new thumbnail is generated. Pub/Sub can connect to the thumbnail generation and mail-sender apps in an asynchronous manner without modifying the original code on the Compute Engine instances.
• Reduced coupling: In the old architecture, a common problem is temporal coupling. If a mail relay server is down, then the app tries to send a notification but it fails. Those processes are tightly coupled and a client might not get a successful response from the application. In the new architecture, the client gets a successful response because generating a thumbnail and sending a notification are loosely coupled.

This new architecture has the following disadvantages:

• Extra effort to modernize the app: Containerizing an app takes time and effort. The new architecture uses more services and requires a different approach to observability, which includes changes to monitoring the app, the deployment process, and resource management.
• Handle thumbnails duplication on app side: Pub/Sub guarantees at-least-once message delivery, which means that duplicate messages might be sent.

About the photo album app

The following diagram shows the initial design of the photo album app.

The preceding diagram illustrates how the thumbnail is generated:

1. A client uploads an image to the app.
2. The app stores the image in Cloud Storage.
3. A request is generated for the thumbnail.
4. The thumbnail generator generates the thumbnail.
5. The successful response is sent to the photo album app.
6. The successful response is sent to the client and you can find the thumbnail in Cloud Storage.

After deploying the app, you are faced with the following problems:

• The app takes a long time to generate the thumbnail. As a result, clients are experiencing many timeouts.
• The service management team wants to detect inappropriate uploaded content. But the application team doesn't have enough resources to implement the feature at this point.

The following diagram extracts the thumbnail generation as a separate service in an asynchronous manner.

You use Pub/Sub to send service requests to the thumbnail generation service. This new architecture makes the service call asynchronous so that a thumbnail is created in the background after the app sends the response back to a client. This design also allows the thumbnail generation service to scale so that multiple jobs can run in parallel.

Objectives

• Deploy an example photo album app on GKE.
• Make asynchronous service calls from the app.
• Use Pub/Sub notifications for Cloud Storage to trigger the app when a new file is uploaded to the Cloud Storage bucket.
• Use Pub/Sub to perform more tasks without modifying the app.

Costs

This tutorial uses the following billable components of Google Cloud:

• Cloud Storage
• Cloud SQL
• Pub/Sub
• Compute Engine instances used by GKE
• Cloud Load Balancing
• Cloud Build
• Container Registry
• Vision

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

4. Enable the GKE, Cloud SQL, Cloud Build, and Cloud Vision APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

7. Enable the GKE, Cloud SQL, Cloud Build, and Cloud Vision APIs.

   Enable the APIs

8. In the console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Setting up the environment

In this section, you assign default settings for values that are used throughout the document. If you close your Cloud Shell session, you lose these environment settings.

1. In Cloud Shell, set your default Cloud project, replacing project-id with your Google Cloud project ID:

   gcloud config set project project-id
   

2. Set your default Compute Engine region, replacing region with a region that is close to you. For more information, see Regions and zones.

   gcloud config set compute/region region
   export REGION=region
   

3. Set your default Compute Engine zone, replacing zone with a zone that is close to you:

   gcloud config set compute/zone zone
   export ZONE=zone
   

4. Download the tutorial files and set your current directory:

   git clone https://github.com/GoogleCloudPlatform/gke-photoalbum-example
   cd gke-photoalbum-example
   

Creating a Cloud Storage bucket and uploading the default thumbnail image

1. In Cloud Shell, create a Cloud Storage bucket to store the original images and thumbnails:

   export PROJECT_ID=$(gcloud config get-value project)
   gsutil mb -c regional -l ${REGION} gs://${PROJECT_ID}-photostore
   

2. Upload the default thumbnail file:

   gsutil cp ./application/photoalbum/images/default.png \
       gs://${PROJECT_ID}-photostore/thumbnails/default.png
   

   • The uploaded images are stored in the following format: gs://project-id-photostore/filename where filename represents the name of the image file that is uploaded.
   • The generated thumbnails are stored in the following format: gs://project-id-photostore/thumbnails/filename.
   • The original image and the corresponding thumbnail have the same filename value, but the thumbnail is stored in the thumbnails bucket.

   • While the thumbnail is being created, the default.png placeholder thumbnail image is displayed in the photo album app.

     

3. Make the thumbnail file public:

   gsutil acl ch -u AllUsers:R \
       gs://${PROJECT_ID}-photostore/thumbnails/default.png
   

Creating a Cloud SQL instance and a MySQL database

1. In Cloud Shell, create the Cloud SQL instance:

   gcloud sql instances create photoalbum-db --region=${REGION} \
       --database-version=MYSQL_5_7
   

2. Retrieve the connection name:

   gcloud sql instances describe photoalbum-db \
       --format="value(connectionName)"
   

   Make a note of the name because you'll use it later in the tutorial.

3. Set the password for the root@% MySQL user:

   gcloud sql users set-password root --host=% --instance=photoalbum-db \
       --password=password
   

   Replace password with a secure password for the root@%user.

4. Connect to the Cloud SQL instance:

   gcloud sql connect photoalbum-db --user=root --quiet
   

   When prompted, enter the password that you set up in the preceding step.

5. Create a database called photo_db, where the user is appuser with a password of pas4appuser:

   create database photo_db;
   grant all privileges on photo_db.* to appuser@"%" \
       identified by 'pas4appuser' with grant option;
   

6. Confirm the result and exit from MySQL:

   show databases;
   select user from mysql.user;
   exit
   

   In the output, confirm that the photo_db database and the appuser user are created:

   MySQL [(none)]> show databases;
   +--------------------+
   | Database           |
   +--------------------+
   | information_schema |
   | mysql              |
   | performance_schema |
   | photo_db           |
   | sys                |
   +--------------------+
   5 rows in set (0.16 sec)
   
   MySQL [(none)]> select user from mysql.user;
   +-----------+
   | user      |
   +-----------+
   | appuser   |
   | root      |
   | mysql.sys |
   +-----------+
   3 rows in set (0.16 sec)
   
   MySQL [(none)]> exit
   Bye
   

Creating a Pub/Sub topic and a subscription

1. In Cloud Shell, create a Pub/Sub topic called thumbnail-service:

   gcloud pubsub topics create thumbnail-service
   

   The photo album app sends requests to the thumbnail generation service by publishing a message on the thumbnail-service topic.

2. Create a Pub/Sub subscription called thumbnail-workers:

   gcloud pubsub subscriptions create --topic thumbnail-service thumbnail-workers
   

   The thumbnail generation service receives requests from the thumbnail-workers subscription.

Creating a GKE cluster

1. In Cloud Shell, create a GKE cluster with permission to call APIs:

   gcloud container clusters create "photoalbum-cluster" \
       --scopes "https://www.googleapis.com/auth/cloud-platform" \
       --num-nodes "5"
   

2. Get access credentials configured so that you can manage the cluster using the kubectl command in later steps:

   gcloud container clusters get-credentials photoalbum-cluster
   

3. Show the list of nodes:

   kubectl get nodes
   

   In the output, confirm that there are five nodes with the STATUS of Ready:

   NAME                                                STATUS    ROLES     AGE       VERSION
   gke-photoalbum-cluster-default-pool-0912a91a-24vt   Ready     <none>    6m        v1.9.7-gke.6
   gke-photoalbum-cluster-default-pool-0912a91a-5h1n   Ready     <none>    6m        v1.9.7-gke.6
   gke-photoalbum-cluster-default-pool-0912a91a-gdm9   Ready     <none>    6m        v1.9.7-gke.6
   gke-photoalbum-cluster-default-pool-0912a91a-swv6   Ready     <none>    6m        v1.9.7-gke.6
   gke-photoalbum-cluster-default-pool-0912a91a-thv8   Ready     <none>    6m        v1.9.7-gke.6
   

Building images for the app

1. In a text editor, open the application/photoalbum/src/auth_decorator.py file and update the username and password:

   USERNAME = 'username'
   PASSWORD = 'passw0rd'
   

2. In Cloud Shell, build an image for the photo album app by using the Cloud Build service:

   gcloud builds submit ./application/photoalbum -t \
       gcr.io/${PROJECT_ID}/photoalbum-app
   

3. Build an image for the thumbnail-worker thumbnail generation service by using the Cloud Build service:

   gcloud builds submit ./application/thumbnail -t \
       gcr.io/${PROJECT_ID}/thumbnail-worker
   

Deploying the photo album app

1. In Cloud Shell, update the Kubernetes Deployment manifests for the photo album and the thumbnail generator with values from your environment:

   connection_name=$(gcloud sql instances describe photoalbum-db \
       --format "value(connectionName)")
   
   digest_photoalbum=$(gcloud container images describe \
       gcr.io/${PROJECT_ID}/photoalbum-app:latest --format \
       "value(image_summary.digest)")
   
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_photoalbum}/" \
       config/photoalbum-deployment.yaml
   
   digest_thumbnail=$(gcloud container images describe \
       gcr.io/${PROJECT_ID}/thumbnail-worker:latest --format \
       "value(image_summary.digest)")
   
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_thumbnail}/" \
           config/thumbnail-deployment.yaml
   

2. Create deployment resources to launch the photo album app and the thumbnail generation service:

   kubectl create -f config/photoalbum-deployment.yaml
   kubectl create -f config/thumbnail-deployment.yaml
   

3. Create a service resource to assign an external IP address to the app:

   kubectl create -f config/photoalbum-service.yaml
   

4. Check the results for the Pods:

   kubectl get pods
   

   In the output, confirm that there are three pods for each photoalbum-app and thumbail-worker with a STATUS of Running:

   NAME                                READY     STATUS    RESTARTS   AGE
   photoalbum-app-555f7cbdb7-cp8nw     2/2       Running   0          2m
   photoalbum-app-555f7cbdb7-ftlc6     2/2       Running   0          2m
   photoalbum-app-555f7cbdb7-xsr4b     2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-728k5   2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-hqxqr   2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-xnxhc   2/2       Running   0          2m
   

   Note: thumbnail-worker subscribes thumbnail generation requests from the thumbnail-workers subscription. For more details, see how the callback function is used in the source code.

5. Check the results for the Services:

   kubectl get services
   

   In the output, confirm that there is an external IP address in the EXTERNAL-IP column for photoalbum-service. It might take a few minutes until they are all set and running.

   NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
   kubernetes           ClusterIP      10.23.240.1     <none>            443/TCP        20m
   photoalbum-service   LoadBalancer   10.23.253.241   146.148.111.115   80:32657/TCP   2m
   

   Make a note of the external IP address because it's used later in this tutorial. In this example, it is 146.148.111.115.

Testing the photo album app

1. To access the deployed app in a web browser, go to the following URL and enter the username and password that you previously set up:

   http://external-ip
   

   Replace external-ip with the IP address that you copied in the previous step.

2. To upload an image file of your choice, click Upload. The thumbnail placeholder appears on the screen.

   

   In the background, the thumbnail generation service creates a thumbnail of the uploaded image. To see the generated thumbnail, click Refresh. The Cloud Vision API adds image labels that it detects.

   

   To see the original image, click the thumbnail.

Adding an inappropriate image detection feature

The following image illustrates how you can use Pub/Sub notifications for Cloud Storage to trigger a service that detects inappropriate content. This feature blurs the image when a new file with inappropriate content is stored in the Cloud Storage bucket.

In the preceding image, the service uses the Safe Search Detection feature from Vision API to detect inappropriate content within an image.

Create a Pub/Sub topic, subscription, and notification

1. In Cloud Shell, create a Pub/Sub topic called safeimage-service:

   gcloud pubsub topics create safeimage-service
   

2. Create a Pub/Sub subscription called safeimage-workers:

   gcloud pubsub subscriptions create --topic safeimage-service \
       safeimage-workers
   

3. Configure a Pub/Sub notification so that a message is sent to the safeimage-service topic when a new file is uploaded to the Cloud Storage bucket:

   gsutil notification create -t safeimage-service -f json \
       gs://${PROJECT_ID}-photostore
   

Build and deploy the worker image

1. In Cloud Shell, build a container image for the safeimage-workers subscription by using Cloud Build:

   gcloud builds submit ./application/safeimage \
       -t gcr.io/${PROJECT_ID}/safeimage-worker
   

2. Update the Kubernetes Deployment manifests for the safe-image service with your Cloud project ID, Cloud SQL connection name, and container image digests:

   digest_safeimage=$(gcloud container images describe \
       gcr.io/${PROJECT_ID}/safeimage-worker:latest --format \
       "value(image_summary.digest)")
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_safeimage}/" \
       config/safeimage-deployment.yaml
   

Create a deployment resource

1. Create a deployment resource called safeimage-deployment to deploy the safeimage-service topic:

   kubectl create -f config/safeimage-deployment.yaml
   

2. Check the results:

   kubectl get pods
   

   In the output, confirm that there are three pods of safeimage-worker with the STATUS of Running.

   NAME                                READY     STATUS    RESTARTS   AGE
   photoalbum-app-555f7cbdb7-cp8nw     2/2       Running   0          30m
   photoalbum-app-555f7cbdb7-ftlc6     2/2       Running   0          30m
   photoalbum-app-555f7cbdb7-xsr4b     2/2       Running   8          30m
   safeimage-worker-7dc8c84f54-6sqzs   1/1       Running   0          2m
   safeimage-worker-7dc8c84f54-9bskw   1/1       Running   0          2m
   safeimage-worker-7dc8c84f54-b7gtp   1/1       Running   0          2m
   thumbnail-worker-86bd95cd68-9wrpv   2/2       Running   0          30m
   thumbnail-worker-86bd95cd68-kbhsn   2/2       Running   2          30m
   thumbnail-worker-86bd95cd68-n4rj7   2/2       Running   0          30m
   

   Note: safeimage-worker subscribes inappropriate image detection requests from the safeimage-workers subscription. For more details, see how the callback function is used in the source code.

Test the inappropriate image detection feature

In this section, you upload a test image to verify that the Safe Search Detection feature blurs out an inappropriate image. The test image is a picture of a girl dressed up as a zombie (licensed under a CC0 license from Pixaby).

1. Download the test image.
2. To upload the image, go to http://external-ip, and then click Upload.

3. Click Refresh. The app displays a blurred thumbnail.

   

   To see that the uploaded image is also blurred, click the thumbnail.

Clean up

The easiest way to eliminate billing is to delete the Cloud project you created for the tutorial. Alternatively, you can delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the individual resources

Instead of deleting the project, you can delete the resources you created in this tutorial.

1. Delete resources from GKE:

   kubectl delete -f config/safeimage-deployment.yaml
   kubectl delete -f config/photoalbum-service.yaml
   kubectl delete -f config/thumbnail-deployment.yaml
   kubectl delete -f config/photoalbum-deployment.yaml
   

2. Delete the cluster from GKE:

   gcloud container clusters delete photoalbum-cluster --quiet
   

3. Delete images from Container Registry:

   gcloud container images delete safeimage-worker
   gcloud container images delete thumbnail-worker
   gcloud container images delete photoalbum-app
   

4. Delete subscriptions and topics from Pub/Sub:

   gcloud pubsub subscriptions delete safeimage-workers
   gcloud pubsub topics delete safeimage-service
   gcloud pubsub subscriptions delete thumbnail-workers
   gcloud pubsub topics delete thumbnail-service
   

5. Delete the Cloud SQL instance:

   gcloud sql instances delete photoalbum-db --quiet
   

6. Delete the Cloud Storage bucket:

   gsutil rm -r gs://${PROJECT_ID}-photostore
   gsutil rm -r gs://${PROJECT_ID}_cloudbuild
   

7. Delete the files:

   cd ..
   rm -rf gke-photoalbum-example
   

What's next

• Learn about the Architecture for using Pub/Sub for long-running tasks.
• Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.
• Read our resources about DevOps, and learn more about the Architecture capability related to this tutorial.
• Take the DevOps quick check to understand where you stand in comparison with the rest of the industry.