Using Pub/Sub with Cloud Run tutorial

This tutorial shows how to write, deploy, and call a Cloud Run service from a Pub/Sub push subscription.

Objectives

  • Write, build, and deploy a service to Cloud Run
  • Call the service by publishing a message to a Pub/Sub topic.

Costs

This tutorial uses billable components of Google Cloud, including:

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. Enable the Cloud Run Admin API
  5. Install and initialize the Cloud SDK.
  6. Update components:
    gcloud components update

Setting up gcloud defaults

To configure gcloud with defaults for your Cloud Run service:

  1. Set your default project:

    gcloud config set project PROJECT_ID

    Replace PROJECT_ID with the name of the project you created for this tutorial.

  2. Configure gcloud for your chosen region:

    gcloud config set run/region REGION

    Replace REGION with the supported Cloud Run region of your choice.

Cloud Run locations

Cloud Run is regional, which means the infrastructure that runs your Cloud Run services is located in a specific region and is managed by Google to be redundantly available across all the zones within that region.

Meeting your latency, availability, or durability requirements are primary factors for selecting the region where your Cloud Run services are run. You can generally select the region nearest to your users but you should consider the location of the other Google Cloud products that are used by your Cloud Run service. Using Google Cloud products together across multiple locations can affect your service's latency as well as cost.

Cloud Run is available in the following regions:

Subject to Tier 1 pricing

  • asia-east1 (Taiwan)
  • asia-northeast1 (Tokyo)
  • asia-northeast2 (Osaka)
  • europe-north1 (Finland) leaf icon Low CO2
  • europe-west1 (Belgium) leaf icon Low CO2
  • europe-west4 (Netherlands)
  • us-central1 (Iowa) leaf icon Low CO2
  • us-east1 (South Carolina)
  • us-east4 (Northern Virginia)
  • us-west1 (Oregon) leaf icon Low CO2

Subject to Tier 2 pricing

  • asia-east2 (Hong Kong)
  • asia-northeast3 (Seoul, South Korea)
  • asia-southeast1 (Singapore)
  • asia-southeast2 (Jakarta)
  • asia-south1 (Mumbai, India)
  • asia-south2 (Delhi, India)
  • australia-southeast1 (Sydney)
  • australia-southeast2 (Melbourne)
  • europe-central2 (Warsaw, Poland)
  • europe-west2 (London, UK)
  • europe-west3 (Frankfurt, Germany)
  • europe-west6 (Zurich, Switzerland) leaf icon Low CO2
  • northamerica-northeast1 (Montreal) leaf icon Low CO2
  • northamerica-northeast2 (Toronto)
  • southamerica-east1 (Sao Paulo, Brazil) leaf icon Low CO2
  • us-west2 (Los Angeles)
  • us-west3 (Las Vegas)
  • us-west4 (Salt Lake City)

If you already created a Cloud Run service, you can view the region in the Cloud Run dashboard in the Cloud Console.

Creating a Pub/Sub topic

The sample service is triggered by messages published to a Pub/Sub topic, so you'll need to create a topic in Pub/Sub.

To create a new Pub/Sub topic, use the command:

gcloud pubsub topics create myRunTopic

You can use myRunTopic or replace with a topic name unique within your Cloud project.

Retrieving the code sample

To retrieve the code sample for use:

  1. Clone the sample app repository to your local machine:

    Node.js

    git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git

    Alternatively, you can download the sample as a zip file and extract it.

    Python

    git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

    Alternatively, you can download the sample as a zip file and extract it.

    Go

    git clone https://github.com/GoogleCloudPlatform/golang-samples.git

    Alternatively, you can download the sample as a zip file and extract it.

    Java

    git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git

    Alternatively, you can download the sample as a zip file and extract it.

  2. Change to the directory that contains the Cloud Run sample code:

    Node.js

    cd nodejs-docs-samples/run/pubsub/

    Python

    cd python-docs-samples/run/pubsub/

    Go

    cd golang-samples/run/pubsub/

    Java

    cd java-docs-samples/run/pubsub/

Looking at the code

The code for this tutorial consists of the following:

  • A server that handles incoming requests.

    Node.js

    To keep the Node.js service easy to test, the server configuration is separate from the server startup.

    The Node.js web server is set up in app.js.

    const express = require('express');
    const app = express();
    
    // This middleware is available in Express v4.16.0 onwards
    app.use(express.json());
    The web server is started in index.js:
    const app = require('./app.js');
    const PORT = process.env.PORT || 8080;
    
    app.listen(PORT, () =>
      console.log(`nodejs-pubsub-tutorial listening on port ${PORT}`)
    );

    Python

    import base64
    import os
    
    from flask import Flask, request
    
    
    app = Flask(__name__)

    Go

    
    // Sample run-pubsub is a Cloud Run service which handles Pub/Sub messages.
    package main
    
    import (
    	"encoding/json"
    	"io/ioutil"
    	"log"
    	"net/http"
    	"os"
    )
    
    func main() {
    	http.HandleFunc("/", HelloPubSub)
    	// Determine port for HTTP service.
    	port := os.Getenv("PORT")
    	if port == "" {
    		port = "8080"
    		log.Printf("Defaulting to port %s", port)
    	}
    	// Start HTTP server.
    	log.Printf("Listening on port %s", port)
    	if err := http.ListenAndServe(":"+port, nil); err != nil {
    		log.Fatal(err)
    	}
    }
    

    Java

    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class PubSubApplication {
      public static void main(String[] args) {
        SpringApplication.run(PubSubApplication.class, args);
      }
    }

  • A handler that processes the Pub/Sub message and logs a greeting.

    Node.js

    app.post('/', (req, res) => {
      if (!req.body) {
        const msg = 'no Pub/Sub message received';
        console.error(`error: ${msg}`);
        res.status(400).send(`Bad Request: ${msg}`);
        return;
      }
      if (!req.body.message) {
        const msg = 'invalid Pub/Sub message format';
        console.error(`error: ${msg}`);
        res.status(400).send(`Bad Request: ${msg}`);
        return;
      }
    
      const pubSubMessage = req.body.message;
      const name = pubSubMessage.data
        ? Buffer.from(pubSubMessage.data, 'base64').toString().trim()
        : 'World';
    
      console.log(`Hello ${name}!`);
      res.status(204).send();
    });

    Python

    @app.route("/", methods=["POST"])
    def index():
        envelope = request.get_json()
        if not envelope:
            msg = "no Pub/Sub message received"
            print(f"error: {msg}")
            return f"Bad Request: {msg}", 400
    
        if not isinstance(envelope, dict) or "message" not in envelope:
            msg = "invalid Pub/Sub message format"
            print(f"error: {msg}")
            return f"Bad Request: {msg}", 400
    
        pubsub_message = envelope["message"]
    
        name = "World"
        if isinstance(pubsub_message, dict) and "data" in pubsub_message:
            name = base64.b64decode(pubsub_message["data"]).decode("utf-8").strip()
    
        print(f"Hello {name}!")
    
        return ("", 204)
    
    

    Go

    
    // PubSubMessage is the payload of a Pub/Sub event.
    // See the documentation for more details:
    // https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
    type PubSubMessage struct {
    	Message struct {
    		Data []byte `json:"data,omitempty"`
    		ID   string `json:"id"`
    	} `json:"message"`
    	Subscription string `json:"subscription"`
    }
    
    // HelloPubSub receives and processes a Pub/Sub push message.
    func HelloPubSub(w http.ResponseWriter, r *http.Request) {
    	var m PubSubMessage
    	body, err := ioutil.ReadAll(r.Body)
    	if err != nil {
    		log.Printf("ioutil.ReadAll: %v", err)
    		http.Error(w, "Bad Request", http.StatusBadRequest)
    		return
    	}
    	if err := json.Unmarshal(body, &m); err != nil {
    		log.Printf("json.Unmarshal: %v", err)
    		http.Error(w, "Bad Request", http.StatusBadRequest)
    		return
    	}
    
    	name := string(m.Message.Data)
    	if name == "" {
    		name = "World"
    	}
    	log.Printf("Hello %s!", name)
    }
    

    Java

    import java.util.Base64;
    import org.apache.commons.lang3.StringUtils;
    import org.springframework.http.HttpStatus;
    import org.springframework.http.ResponseEntity;
    import org.springframework.web.bind.annotation.RequestBody;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.bind.annotation.RequestMethod;
    import org.springframework.web.bind.annotation.RestController;
    
    // PubsubController consumes a Pub/Sub message.
    @RestController
    public class PubSubController {
      @RequestMapping(value = "/", method = RequestMethod.POST)
      public ResponseEntity receiveMessage(@RequestBody Body body) {
        // Get PubSub message from request body.
        Body.Message message = body.getMessage();
        if (message == null) {
          String msg = "Bad Request: invalid Pub/Sub message format";
          System.out.println(msg);
          return new ResponseEntity(msg, HttpStatus.BAD_REQUEST);
        }
    
        String data = message.getData();
        String target =
            !StringUtils.isEmpty(data) ? new String(Base64.getDecoder().decode(data)) : "World";
        String msg = "Hello " + target + "!";
    
        System.out.println(msg);
        return new ResponseEntity(msg, HttpStatus.OK);
      }
    }

    You must code the service to return an accurate HTTP response code. Success codes, such as HTTP 200 or 204, acknowledge complete processing of the Pub/Sub message. Error codes, such as HTTP 400 or 500, indicate the message will be retried, as described in Receiving messages using Push guide.

  • A Dockerfile that defines the operating environment for the service. The contents of the Dockerfile vary by language.

    Node.js

    
    # Use the official lightweight Node.js 12 image.
    # https://hub.docker.com/_/node
    FROM node:12-slim
    
    # Create and change to the app directory.
    WORKDIR /usr/src/app
    
    # Copy application dependency manifests to the container image.
    # A wildcard is used to ensure both package.json AND package-lock.json are copied.
    # Copying this separately prevents re-running npm install on every code change.
    COPY package*.json ./
    
    # Install dependencies.
    # If you add a package-lock.json speed your build by switching to 'npm ci'.
    # RUN npm ci --only=production
    RUN npm install --production
    
    # Copy local code to the container image.
    COPY . .
    
    # Run the web service on container startup.
    CMD [ "npm", "start" ]
    

    Python

    
    # Use the official Python image.
    # https://hub.docker.com/_/python
    FROM python:3.9
    
    # Allow statements and log messages to immediately appear in the Cloud Run logs
    ENV PYTHONUNBUFFERED True
    
    # Copy application dependency manifests to the container image.
    # Copying this separately prevents re-running pip install on every code change.
    COPY requirements.txt ./
    
    # Install production dependencies.
    RUN pip install -r requirements.txt
    
    # Copy local code to the container image.
    ENV APP_HOME /app
    WORKDIR $APP_HOME
    COPY . ./
    
    # Run the web service on container startup.
    # Use gunicorn webserver with one worker process and 8 threads.
    # For environments with multiple CPU cores, increase the number of workers
    # to be equal to the cores available.
    # Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.
    CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
    

    Go

    
    # Use the offical golang image to create a binary.
    # This is based on Debian and sets the GOPATH to /go.
    # https://hub.docker.com/_/golang
    FROM golang:1.16-buster as builder
    
    # Create and change to the app directory.
    WORKDIR /app
    
    # Retrieve application dependencies.
    # This allows the container build to reuse cached dependencies.
    # Expecting to copy go.mod and if present go.sum.
    COPY go.* ./
    RUN go mod download
    
    # Copy local code to the container image.
    COPY . ./
    
    # Build the binary.
    RUN go build -v -o server
    
    # Use the official Debian slim image for a lean production container.
    # https://hub.docker.com/_/debian
    # https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds
    FROM debian:buster-slim
    RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        ca-certificates && \
        rm -rf /var/lib/apt/lists/*
    
    # Copy the binary to the production image from the builder stage.
    COPY --from=builder /app/server /server
    
    # Run the web service on container startup.
    CMD ["/server"]
    

    Java

    This sample uses Jib to build Docker images using common Java tools. Jib optimizes container builds without the need for a Dockerfile or having Docker installed. Learn more about building Java containers with Jib.
    <plugin>
      <groupId>com.google.cloud.tools</groupId>
      <artifactId>jib-maven-plugin</artifactId>
      <version>3.1.4</version>
      <configuration>
        <to>
          <image>gcr.io/PROJECT_ID/pubsub</image>
        </to>
      </configuration>
    </plugin>
    

For details on how to authenticate the origin of Pub/Sub requests, read the section below on Integrating with Pub/Sub.

Shipping the code

Shipping code consists of three steps: building a container image with Cloud Build, uploading the container image to Container Registry, and deploying the container image to Cloud Run.

To ship your code:

  1. Build your container and publish on Container Registry:

    Node.js

    gcloud builds submit --tag gcr.io/PROJECT_ID/pubsub

    Where PROJECT_ID is your Cloud project ID, and pubsub is the name you want to give your service.

    Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Container Registry and can be re-used if desired.

    Python

    gcloud builds submit --tag gcr.io/PROJECT_ID/pubsub

    Where PROJECT_ID is your Cloud project ID, and pubsub is the name you want to give your service.

    Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Container Registry and can be re-used if desired.

    Go

    gcloud builds submit --tag gcr.io/PROJECT_ID/pubsub

    Where PROJECT_ID is your Cloud project ID, and pubsub is the name you want to give your service.

    Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Container Registry and can be re-used if desired.

    Java

    1. Use the gcloud credential helper to authorize Docker to push to your Container Registry.

      gcloud auth configure-docker

    2. Use the Jib Maven Plugin to build and push the container to Container Registry.

      mvn compile jib:build -Dimage=gcr.io/PROJECT_ID/pubsub

    Where PROJECT_ID is your Cloud project ID, and pubsub is the name you want to give your service.

    Upon success, you should see a BUILD SUCCESS message. The image is stored in Container Registry and can be re-used if desired.

  2. Run the following command to deploy your app:

    gcloud run deploy pubsub-tutorial --image gcr.io/PROJECT_ID/pubsub  --no-allow-unauthenticated

    Replace PROJECT_ID with your Cloud project ID. pubsub is the container name and pubsub-tutorial is the name of the service. Notice that the container image is deployed to the service and region that you configured previously under Setting up gcloud

    The --no-allow-unauthenticated flag restricts unauthenticated access to the service. By keeping the service private you can rely on Cloud Run's automatic Pub/Sub integration to authenticate requests. See Integrating with Pub/Sub for more details on how this is configured. For more details about authentication that is based on Identity and Access Management (IAM), see Managing access using IAM.

    Wait until the deployment is complete: this can take about half a minute. On success, the command line displays the service URL. This URL is used to configure a Pub/Sub subscription.

  3. If you want to deploy a code update to the service, repeat the previous steps. Each deployment to a service creates a new revision and automatically starts serving traffic when ready.

Integrating with Pub/Sub

To integrate the service with Pub/Sub:

  1. Create or select a service account to represent the Pub/Sub subscription identity.

    gcloud iam service-accounts create cloud-run-pubsub-invoker \
         --display-name "Cloud Run Pub/Sub Invoker"

    You can use cloud-run-pubsub-invoker or replace with a name unique within your Cloud project.

  2. Create a Pub/Sub subscription with the service account:

    1. Give the invoker service account permission to invoke your pubsub-tutorial service:

      gcloud run services add-iam-policy-binding pubsub-tutorial \
         --member=serviceAccount:cloud-run-pubsub-invoker@PROJECT_ID.iam.gserviceaccount.com \
         --role=roles/run.invoker

      It can take several minutes for the IAM changes to propagate. In the meantime, you might see HTTP 403 errors in the service logs.

    2. Create a Pub/Sub subscription with the service account:

      gcloud pubsub subscriptions create myRunSubscription --topic myRunTopic \
         --push-endpoint=SERVICE-URL/ \
         --push-auth-service-account=cloud-run-pubsub-invoker@PROJECT_ID.iam.gserviceaccount.com

      Replace

      • myRunTopic with the topic you previously created.
      • SERVICE-URL with the HTTPS URL provided on deploying the service. This URL works even if you have also added a domain mapping.
      • PROJECT_ID with your Cloud project ID.

      The --push-auth-service-account flag activates the Pub/Sub push functionality for Authentication and authorization.

      Your Cloud Run service domain is automatically registered for use with Pub/Sub subscriptions.

      For Cloud Run only, there is a built-in authentication check that the token is valid and an authorization check that the service account has permission to invoke the Cloud Run service.

Your service is now fully integrated with Pub/Sub.

Trying it out

To test the end-to-end solution:

  1. Send a Pub/Sub message to the topic:

    gcloud pubsub topics publish myRunTopic --message "Runner"

    You can also publish messages programmatically instead of using the command-line as shown in this tutorial. For more information, see Publishing messages.

  2. Navigate to the service logs:

    1. Navigate to the Google Cloud Console
    2. Click the pubsub-tutorial service.
    3. Select the Logs tab.

      Logs might take a few moments to appear. If you don't see them immediately, check again after a few moments.

  3. Look for the "Hello Runner!" message.

Clean up

To walk through a more in-depth use case of using Cloud Run with Pub/Sub, skip cleanup for now and continue with the Image Processing with Cloud Run tutorial.

If you created a new project for this tutorial, delete the project. If you used an existing project and wish to keep it without the changes added in this tutorial, delete resources created for the tutorial.

Deleting the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting tutorial resources

  1. Delete the Cloud Run service you deployed in this tutorial:

    gcloud run services delete SERVICE-NAME

    Where SERVICE-NAME is your chosen service name.

    You can also delete Cloud Run services from the Google Cloud Console.

  2. Remove the gcloud default region configuration you added during tutorial setup:

     gcloud config unset run/region
    
  3. Remove the project configuration:

     gcloud config unset project
    
  4. Delete other Google Cloud resources created in this tutorial:

What's next