Build a one-to-many Pub/Sub system

This tutorial walks you through setting up a set of applications that communicate by sending messages through Pub/Sub rather than synchronous RPCs. By decoupling applications, messaging:

• Makes applications more robust
• Might simplify development

For example, the caller (publisher) does not need the receiver (subscriber) to be up and available. The publisher sends a message to Pub/Sub. The publisher doesn't need to know which and how many subscriber applications need to receive the message. As a result, the service can be relied upon to deliver the message to one or more subscriber applications whenever they are available.

System overview

In this tutorial, you start a publisher application that sends a "Hello, World!" message to two subscribers using one-to-many communication, as illustrated in the following diagram:

The two subscriber applications use the same code, but you start them at different times. This process demonstrates how Pub/Sub enables asynchronous communication. To build this system, you complete the following steps:

1. Create an IAM service account that the applications use for authentication.
2. Set up IAM permissions.
3. Create a Pub/Sub topic and a subscription.
4. Start three independent applications: one publisher and two subscribers.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Pub/Sub API:

   gcloud services enable pubsub.googleapis.com

7. Create local authentication credentials for your user account:

   gcloud auth application-default login

8. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/pubsub.publisher, roles/pubsub.subscriber

   gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

   • Replace PROJECT_ID with your project ID.

   • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:[e-mail].

   • Replace ROLE with each individual role.
9. Install the Google Cloud CLI.

10. To initialize the gcloud CLI, run the following command:

    gcloud init

11. Create or select a Google Cloud project.

    Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

12. Make sure that billing is enabled for your Google Cloud project.

13. Enable the Pub/Sub API:

    gcloud services enable pubsub.googleapis.com

14. Create local authentication credentials for your user account:

    gcloud auth application-default login

15. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/pubsub.publisher, roles/pubsub.subscriber

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE

    • Replace PROJECT_ID with your project ID.

    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:[e-mail].

    • Replace ROLE with each individual role.

Install Python

This tutorial uses the Pub/Sub Client Libraries, which requires Python 3.7 or higher. Complete the instructions for installing Python.

Set up your Pub/Sub project

To manage message flow between publishing and subscribing applications, you create a topic and two different subscriptions.

Create a Pub/Sub topic

Create a topic with the ID hello_topic:

gcloud pubsub topics create hello_topic

Create Pub/Sub subscriptions

Create two subscriptions and attach them to your topic.

These subscriptions are a StreamingPull subscription, which is a type of pull subscription.

Subscription 1

Create a subscription with the ID sub_one and attach it to hello_topic.

gcloud pubsub subscriptions create sub_one --topic=hello_topic

Subscription 2

Create a subscription with the ID sub_two and attach it to hello_topic.

gcloud pubsub subscriptions create sub_two --topic=hello_topic

Build the one-to-many system

Download the publisher and subscriber code

1. Download the Pub/Sub Python files needed for this tutorial.

    git clone https://github.com/googleapis/python-pubsub.git

2. Close any open terminals before proceeding.

Set up three terminals

1. Start one terminal for each tutorial application (one publisher and two subscribers). For convenience, this tutorial calls these terminals:

   • publisher terminal
   • sub_one terminal
   • sub_two terminal
   Note: Make sure all three terminals are in the same directory relative to where you downloaded the Pub/Sub Python files in the previous step.

2. In the publisher terminal, create and activate a Python virtual environment named pyenv-qs.

   Bash

   python -m venv pyenv-qs
   source pyenv-qs/bin/activate

   PowerShell

   py -m venv pyenv-qs
   .\pyenv-qs\Scripts\activate

   In the sub_one and sub_two terminals, run the following command:

   Bash

   source pyenv-qs/bin/activate

   PowerShell

   .\pyenv-qs\Scripts\activate

   After you run the activate command, your command prompt includes the following value (pyenv-qs) $.

3. In the publisher terminal, install the Pub/Sub Python client library using pip:

   python -m pip install --upgrade google-cloud-pubsub

4. In all three terminals, set up an environment variable with your current project ID. This gcloud command determines your selected project ID and sets it as a variable:

   Bash

   export PROJECT=`gcloud config get-value project`

   PowerShell

   $env:PROJECT=$(gcloud config get-value project)

5. In all three terminals, change to the project path that contains the sample code.

   cd python-pubsub/samples/snippets/quickstart/

Start the apps and observe message flow

Start the Subscriber 1 application

In the sub_one terminal, start Subscriber 1:

Bash

python sub.py $PROJECT sub_one

PowerShell

py sub.py $env:PROJECT sub_one

Once started, this application opens a bidirectional streaming connection with the server. Pub/Sub delivers messages over the stream.

Start the Publisher application

In the publisher terminal, start the Publisher application:

Bash

python pub.py $PROJECT hello_topic

PowerShell

py pub.py $env:PROJECT hello_topic

After the publisher application starts, the Pub/Sub system does the following:

• The Publisher application sends a "Hello, World!" message to Pub/Sub while remaining unaware of any existing subscriptions. The server also assigns a message ID.

• The Subscriber 1 application receives the 'Hello World' message, prints it, and sends an acknowledgment to Pub/Sub.

• The Publisher application prints the acknowledgement. The acknowledgment tells Pub/Sub that the message has been processed successfully and does not need to be re-sent to this or any other sub_one subscriber.

Pub/Sub removes the message from sub_one.

Start the Subscriber 2 application

In the sub_two terminal, start Subscriber 2:

Bash

python sub.py $PROJECT sub_two

PowerShell

py sub.py $env:PROJECT sub_two

This subscriber receives messages delivered to the sub_two subscription. Subscriber 2 reuses the sub.py script. The difference is that Subscriber 2 isn't started until after the Publisher has sent the message to the topic and subscriptions. If Publisher were calling Subscriber 2 directly, the publishing application would either have to wait until Subscriber 2 comes up or it would have to time out. Pub/Sub manages this process by effectively saving the message for Subscriber 2.

You are now ready to develop with Pub/Sub!

How did it go?

It worked!

Great! What did you like about the quickstart? What could we have done better? Let us know!.

I got stuck.

We're sorry to hear that. Let us know what went wrong. We'll want to fix it.

Additional resources and links are available on the Pub/Sub support page.

Clean up

1. Stop all running applications.
2. Delete the sample code directory from your local environment.

3. Delete the topic.

   gcloud pubsub topics delete hello_topic

4. Delete the subscriptions.

   gcloud pubsub subscriptions delete sub_one

   gcloud pubsub subscriptions delete sub_two

5. Shut down the tutorial project in the IAM & admin section of the Google Cloud console.

6. Optional: Revoke the authentication credentials that you created, and delete the local credential file.

   gcloud auth application-default revoke

7. Optional: Revoke credentials from the gcloud CLI.

   gcloud auth revoke

What's next

Here are some things that you can try:

• Examine the tutorial's pub.py and sub.py code and browse other Pub/Sub samples on GitHub. As an exercise, create a version of pub.py that publishes the local time every second.

• Learn to batch messages.

• Using Push subscriptions, receive messages that trigger App Engine endpoints or Cloud Functions.

• Retrieve previously acknowledged messages using replay. By default, Pub/Sub removes acknowledged messages from subscriptions. In this tutorial, for instance, you wouldn't be able to rerun sub.py to receive the "Hello, World!" message again. The replay feature lets you set up subscriptions so that you can receive messages after they have been acknowledged.

• Get started with client libraries in other languages.