Deploy image processing using microservices and asynchronous messaging

Last reviewed 2023-07-17 UTC

This document describes how to implement the reference architecture that's described in Integrate microservices with Pub/Sub and GKE. The architecture is designed to handle long-running processes by using containers and asynchronous messaging.

The document uses an example photo-sharing application that generates photo thumbnails. You deploy the application using Google Kubernetes Engine (GKE) and use Pub/Sub to invoke long-running processes asynchronously. You also use Pub/Sub notifications for Cloud Storage to add side jobs without modifying the application's code.

The application is containerized by Cloud Build and stored in Artifact Registry. It uses Cloud Vision to detect inappropriate images.

Architecture

The following diagram illustrates the design of the example photo album application that implements the reference architecture.

Figure 1. Architecture for image processing that's based on using containers and asynchronous messaging.

The preceding diagram illustrates how the thumbnail is generated:

1. A client uploads an image to the application.
2. The application stores the image in Cloud Storage.
3. A request is generated for the thumbnail.
4. The thumbnail generator generates the thumbnail.
5. The successful response is sent to the photo album application.
6. The successful response is sent to the client and you can find the thumbnail in Cloud Storage.

The following diagram shows how the application implements thumbnail generation as a separate service in an asynchronous manner.

Figure 2. Architecture of the thumbnail extraction process.

You use Pub/Sub to send service requests to the thumbnail generation service. This new architecture makes the service call asynchronous so that a thumbnail is created in the background after the application sends the response back to a client. This design also allows the thumbnail generation service to scale so that multiple jobs can run in parallel.

Objectives

• Deploy an example photo album application on GKE.
• Make asynchronous service calls from the application.
• Use Pub/Sub notifications for Cloud Storage to trigger the application when a new file is uploaded to the Cloud Storage bucket.
• Use Pub/Sub to perform more tasks without modifying the application.

Costs

In this document, you use the following billable components of Google Cloud:

• Cloud Storage
• Cloud SQL
• Pub/Sub
• Compute Engine instances used by GKE
• Cluster-management fee for GKE
• Cloud Load Balancing
• Cloud Build
• Artifact Registry
• Vision

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish building the example application, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the GKE, Cloud SQL, Cloud Build, Artifact Registry, and Cloud Vision APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the GKE, Cloud SQL, Cloud Build, Artifact Registry, and Cloud Vision APIs.

   Enable the APIs

8. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Set up the environment

In this section, you assign default settings for values that are used throughout the document. If you close your Cloud Shell session, you lose these environment settings.

1. In Cloud Shell, set your default Google Cloud project:

   gcloud config set project PROJECT_ID
   

   Replace PROJECT_ID with your Google Cloud project ID.

2. Set your default Compute Engine region:

   gcloud config set compute/region REGION
   export REGION=REGION
   

   Replace REGION with a region that is close to you. For more information, see Regions and zones.

3. Set your default Compute Engine zone:

   gcloud config set compute/zone ZONE
   export ZONE=ZONE
   

   Replace ZONE with a zone that is close to you.

4. Download the example application files and set your current directory:

   git clone https://github.com/GoogleCloudPlatform/gke-photoalbum-example
   cd gke-photoalbum-example
   

Create a Cloud Storage bucket and upload the default thumbnail image

1. In Cloud Shell, create a Cloud Storage bucket to store the original images and thumbnails:

   export PROJECT_ID=$(gcloud config get-value project)
   gsutil mb -c regional -l ${REGION} gs://${PROJECT_ID}-photostore
   

2. Upload the default thumbnail file:

   gsutil cp ./application/photoalbum/images/default.png \
       gs://${PROJECT_ID}-photostore/thumbnails/default.png
   

   • The uploaded images are stored in the following format: gs://PROJECT_ID-photostore/FILENAME, where FILENAME represents the name of the image file that is uploaded.
   • The generated thumbnails are stored in the following format: gs://PROJECT_ID-photostore/thumbnails/FILENAME.
   • The original image and the corresponding thumbnail have the same FILENAME value, but the thumbnail is stored in the thumbnails bucket.

   • While the thumbnail is being created, the following default.png placeholder thumbnail image is displayed in the photo album application.

     

3. Make the thumbnail file public:

   gsutil acl ch -u AllUsers:R \
       gs://${PROJECT_ID}-photostore/thumbnails/default.png
   

Create a Cloud SQL instance and a MySQL database

1. In Cloud Shell, create the Cloud SQL instance:

   gcloud sql instances create photoalbum-db --region=${REGION} \
       --database-version=MYSQL_8_0
   

2. Retrieve the connection name:

   gcloud sql instances describe photoalbum-db \
       --format="value(connectionName)"
   

   Make a note of the name because you'll use it later.

3. Set the password for the root@% MySQL user:

   gcloud sql users set-password root --host=% --instance=photoalbum-db \
       --password=PASSWORD
   

   Replace PASSWORD with a secure password for the root@%user.

4. Connect to the Cloud SQL instance:

   gcloud sql connect photoalbum-db --user=root --quiet
   

   When you're prompted, enter the password that you set up in the preceding step.

5. Create a database called photo_db, where the user is appuser and the password is pas4appuser:

   create database photo_db;
   create user 'appuser'@'%' identified by 'pas4appuser';
   grant all on photo_db.* to 'appuser'@'%' with grant option;
   flush privileges;
   

6. Confirm the result and exit from MySQL:

   show databases;
   select user from mysql.user;
   exit
   

   In the output, confirm that the photo_db database and the appuser user are created:

   mysql> show databases;
   +--------------------+
   | Database           |
   +--------------------+
   | information_schema |
   | mysql              |
   | performance_schema |
   | photo_db           |
   | sys                |
   +--------------------+
   5 rows in set (0.16 sec)
   
   mysql> \t
   Outfile disabled.
   mysql> select user from mysql.user;
   +-------------------+
   | user              |
   +-------------------+
   | appuser           |
   | cloudsqlreplica   |
   | cloudsqlsuperuser |
   | root              |
   | cloudsqlexport    |
   | cloudsqlimport    |
   | cloudsqloneshot   |
   | root              |
   | cloudsqlexport    |
   | cloudsqlimport    |
   | cloudsqloneshot   |
   | root              |
   | cloudsqlapplier   |
   | cloudsqlimport    |
   | mysql.infoschema  |
   | mysql.session     |
   | mysql.sys         |
   | root              |
   +-------------------+
   18 rows in set (0.16 sec)
   
   mysql> exit
   Bye
   

Create a Pub/Sub topic and a subscription

1. In Cloud Shell, create a Pub/Sub topic called thumbnail-service:

   gcloud pubsub topics create thumbnail-service
   

   The photo album application sends requests to the thumbnail generation service by publishing a message on the thumbnail-service topic.

2. Create a Pub/Sub subscription called thumbnail-workers:

   gcloud pubsub subscriptions create --topic thumbnail-service thumbnail-workers
   

   The thumbnail generation service receives requests from the thumbnail-workers subscription.

Create a GKE cluster

1. In Cloud Shell, create a GKE cluster that has permission to call APIs:

   gcloud container clusters create "photoalbum-cluster" \
       --scopes "https://www.googleapis.com/auth/cloud-platform" \
       --num-nodes "5"
   

2. Get access credentials configured so that you can manage the cluster using the kubectl command in later steps:

   gcloud container clusters get-credentials photoalbum-cluster
   

3. Show the list of nodes:

   kubectl get nodes
   

   In the output, confirm that there are five nodes whose STATUS value is Ready:

   NAME                                                STATUS   ROLES    AGE     VERSION
   gke-photoalbum-cluster-default-pool-d637570a-2pfh   Ready    <none>   2m55s   v1.24.10-gke.2300
   gke-photoalbum-cluster-default-pool-d637570a-3rm4   Ready    <none>   2m55s   v1.24.10-gke.2300
   gke-photoalbum-cluster-default-pool-d637570a-f7l4   Ready    <none>   2m55s   v1.24.10-gke.2300
   gke-photoalbum-cluster-default-pool-d637570a-qb2z   Ready    <none>   2m53s   v1.24.10-gke.2300
   gke-photoalbum-cluster-default-pool-d637570a-rvnp   Ready    <none>   2m54s   v1.24.10-gke.2300
   

Create the Artifact Registry repository

• In Cloud Shell, create a repository to store container images:

  gcloud artifacts repositories create photoalbum-repo \
      --repository-format=docker \
      --location=us-central1 \
      --description="Docker repository"
  

Build images for the application

1. In a text editor, open the application/photoalbum/src/auth_decorator.py file and update the username and password:

   USERNAME = 'username'
   PASSWORD = 'passw0rd'
   

2. In Cloud Shell, build an image for the photo album application by using the Cloud Build service:

   gcloud builds submit ./application/photoalbum -t \
       us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/photoalbum-app
   

3. Build an image for the thumbnail-worker thumbnail generation service by using the Cloud Build service:

   gcloud builds submit ./application/thumbnail -t \
       us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/thumbnail-worker
   

Deploy the photo album application

1. In Cloud Shell, update the Kubernetes Deployment manifests for the photo album and the thumbnail generator with values from your environment:

   connection_name=$(gcloud sql instances describe photoalbum-db \
       --format "value(connectionName)")
   
   digest_photoalbum=$(gcloud container images describe \
       us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/photoalbum-app:latest --format \
       "value(image_summary.digest)")
   
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_photoalbum}/" \
       config/photoalbum-deployment.yaml
   
   digest_thumbnail=$(gcloud container images describe \
       us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/thumbnail-worker:latest --format \
       "value(image_summary.digest)")
   
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_thumbnail}/" \
           config/thumbnail-deployment.yaml
   

2. Create deployment resources to launch the photo album application and the thumbnail generation service:

   kubectl create -f config/photoalbum-deployment.yaml
   kubectl create -f config/thumbnail-deployment.yaml
   

3. Create a service resource to assign an external IP address to the application:

   kubectl create -f config/photoalbum-service.yaml
   

4. Check the results for the pods:

   kubectl get pods
   

   In the output, confirm that there are three pods for each photoalbum-app and thumbail-worker pod and that their STATUS value is Running:

   NAME                                READY     STATUS    RESTARTS   AGE
   photoalbum-app-555f7cbdb7-cp8nw     2/2       Running   0          2m
   photoalbum-app-555f7cbdb7-ftlc6     2/2       Running   0          2m
   photoalbum-app-555f7cbdb7-xsr4b     2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-728k5   2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-hqxqr   2/2       Running   0          2m
   thumbnail-worker-86bd95cd68-xnxhc   2/2       Running   0          2m
   

   The thumbnail-worker pods subscribe thumbnail generation requests from the thumbnail-workers subscription. For more information, see how the callback function is used in the source code.

5. Check the results for the services:

   kubectl get services
   

   In the output, confirm that there is an external IP address in the EXTERNAL-IP column for the photoalbum-service service. It might take a few minutes until the services are all running.

   NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
   kubernetes           ClusterIP      10.23.240.1     <none>            443/TCP        20m
   photoalbum-service   LoadBalancer   10.23.253.241   146.148.111.115   80:32657/TCP   2m
   

   Make a note of the external IP address because it's used later. In this example, it's 146.148.111.115.

Test the photo album application

1. To access the deployed application in a web browser, go to the following URL and enter the username and password that you previously set up:

   http://EXTERNAL_IP
   

   Replace EXTERNAL_IP with the IP address that you copied in the previous step.

2. To upload an image file, click Upload.

   

   The thumbnail placeholder appears on the screen.

   In the background, the thumbnail generation service creates a thumbnail of the uploaded image. To see the generated thumbnail, click Refresh. The Cloud Vision API adds image labels that it detects.

   

   To see the original image, click the thumbnail.

Add an inappropriate-image detection feature

The following image illustrates how you can use Pub/Sub notifications for Cloud Storage to trigger a service that detects inappropriate content. This feature blurs the image when a new file with inappropriate content is stored in the Cloud Storage bucket.

In the preceding image, the service uses the Safe Search Detection feature from Vision API to detect inappropriate content within an image.

The photo application triggers the thumbnail generator and the image checker asynchronously. As a result, there is no guarantee that they will be executed in a specific order. If the thumbnail generation occurs before the image is blurred, you might see an inappropriate thumbnail for a short while. However, the image checker eventually blurs both the inappropriate image and the inappropriate thumbnail.

Create a Pub/Sub topic, subscription, and notification

1. In Cloud Shell, create a Pub/Sub topic called safeimage-service:

   gcloud pubsub topics create safeimage-service
   

2. Create a Pub/Sub subscription called safeimage-workers:

   gcloud pubsub subscriptions create --topic safeimage-service \
       safeimage-workers
   

3. Configure a Pub/Sub notification so that a message is sent to the safeimage-service topic when a new file is uploaded to the Cloud Storage bucket:

   gsutil notification create -t safeimage-service -f json \
       gs://${PROJECT_ID}-photostore
   

Build and deploy the worker image

1. In Cloud Shell, build a container image for the safeimage-workers subscription by using Cloud Build:

   gcloud builds submit ./application/safeimage \
       -t us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/safeimage-worker
   

2. Update the Kubernetes Deployment manifests for the safe-image service with your Google Cloud project ID, Cloud SQL connection name, and container image digests:

   digest_safeimage=$(gcloud container images describe \
       us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/safeimage-worker:latest --format \
       "value(image_summary.digest)")
   sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
       -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
       -e "s/\[DIGEST\]/${digest_safeimage}/" \
       config/safeimage-deployment.yaml
   

Create a deployment resource

1. Create a deployment resource called safeimage-deployment to deploy the safeimage-service topic:

   kubectl create -f config/safeimage-deployment.yaml
   

2. Check the results:

   kubectl get pods
   

   In the output, confirm that there are three safeimage-worker pods whose STATUS value is Running.

   NAME                                READY     STATUS    RESTARTS   AGE
   photoalbum-app-555f7cbdb7-cp8nw     2/2       Running   0          30m
   photoalbum-app-555f7cbdb7-ftlc6     2/2       Running   0          30m
   photoalbum-app-555f7cbdb7-xsr4b     2/2       Running   8          30m
   safeimage-worker-7dc8c84f54-6sqzs   1/1       Running   0          2m
   safeimage-worker-7dc8c84f54-9bskw   1/1       Running   0          2m
   safeimage-worker-7dc8c84f54-b7gtp   1/1       Running   0          2m
   thumbnail-worker-86bd95cd68-9wrpv   2/2       Running   0          30m
   thumbnail-worker-86bd95cd68-kbhsn   2/2       Running   2          30m
   thumbnail-worker-86bd95cd68-n4rj7   2/2       Running   0          30m
   

   The safeimage-worker pods subscribe ro inappropriate-image detection requests from the safeimage-workers subscription. For more information, see how the callback function is used in the source code.

Test the inappropriate-image detection feature

In this section, you upload a test image to verify that the Safe Search Detection feature blurs out an inappropriate image. The test image is a picture of a girl who's dressed up as a zombie (licensed under a CC0 license from Pixaby).

1. Download the test image.
2. To upload the image, go to http://EXTERNAL_IP, and then click Upload.

3. Click Refresh. The application displays a blurred thumbnail.

   

   To see that the uploaded image is also blurred, click the thumbnail.

Clean up

If you don't want to keep the Google Cloud resources that you created for the example application, you can remove them so you won't be billed for them in the future. You can either delete the project entirely or delete cluster resources and then delete the cluster.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the individual resources

Instead of deleting the project, you can delete the individual resources that you created.

1. Delete resources from GKE:

   kubectl delete -f config/safeimage-deployment.yaml
   kubectl delete -f config/photoalbum-service.yaml
   kubectl delete -f config/thumbnail-deployment.yaml
   kubectl delete -f config/photoalbum-deployment.yaml
   

2. Delete the cluster from GKE:

   gcloud container clusters delete photoalbum-cluster --quiet
   

3. Delete the repository from Artifact Registry:

   gcloud artifacts repositories delete photoalbum-repo --location us-central1 --quiet
   

4. Delete subscriptions and topics from Pub/Sub:

   gcloud pubsub subscriptions delete safeimage-workers
   gcloud pubsub topics delete safeimage-service
   gcloud pubsub subscriptions delete thumbnail-workers
   gcloud pubsub topics delete thumbnail-service
   

5. Delete the Cloud SQL instance:

   gcloud sql instances delete photoalbum-db --quiet
   

6. Delete the Cloud Storage bucket:

   gsutil rm -r gs://${PROJECT_ID}-photostore
   gsutil rm -r gs://${PROJECT_ID}_cloudbuild
   

7. Delete the files:

   cd ..
   rm -rf gke-photoalbum-example
   

What's next

• Read about DevOps, and learn more about the Architecture capability that's related to this reference architecture.
• Take the DevOps quick check to understand where you stand in comparison with the rest of the industry.
• For more reference architectures, diagrams, and best practices, explore the Cloud Architecture Center.