Introduction
This tutorial walks you through setting up a simple set of applications that communicate by sending messages through Pub/Sub rather than synchronous RPCs. By decoupling applications, messaging:
- Makes applications more robust
- Might simplify development
For example, the caller (publisher) does not need the receiver (subscriber) to be up and available. It simply sends a message to Pub/Sub. Nor does the publisher need to know which and how many subscriber applications need to receive the message. As a result, the service can be relied upon to deliver the message to one or more subscriber applications whenever they are available.
Requirements:
- A Google Account
- Up to an hour of time to complete
System overview
In this tutorial, you start a publisher application that sends a "Hello, World!" message to two subscribers using one-to-many communication, as illustrated below:
The two subscriber applications use the same code, but you start them at different times. This process demonstrates how Pub/Sub enables asynchronous communication. To build this system, you:
- Create the Pub/Sub topic and subscriptions needed.
- Create a service account that the applications use for authentication.
- Set up IAM permissions.
- Start three independent applications: one publisher and two subscribers.
Tutorial setup
Set up your Google Cloud project and Pub/Sub topic and subscriptions
Log into Google Cloud Console.
Go to the Google Cloud Console
If you're new to Cloud, click Activate and follow the prompts to set up your Cloud account.
Select an existing project or create a new one. A default project is created for you the first time you use Google Cloud.
In the Home section of the Cloud Console, make a note of the Project ID. You use this value to set your current Cloud Storage project during the Cloud SDK initialization process. You also pass this ID to the Python script as you start the publisher and subscriber applications.
Go to the Pub/Sub section of the Google Cloud Console.
Follow the prompt to enable the API.
Click Create a topic. Publishing applications send messages to topics. Use hello_topic as the Name.
In the Topic details page, click Create Subscription:
Name the subscription sub_one. Do not change any of the default settings. You are creating a StreamingPull subscription, which is a type of pull subscription.
Use the same procedure to create another subscription attached to hello_topic, named sub_two.
You can click the topic name in the Topics view to see the new subscriptions or you can change to the Subscriptions view.
At this point, your Pub/Sub environment is ready to manage message flow between the publishing and subscribing applications.
Create service account credentials
Go to the Service accounts section of the console.
Select your project and click Create Service Account.
Enter a Service account name, such as pubsub-tutorial.
Click Create.
For the tutorial, the service account needs publishing and subscribing permissions. Use the Select a role dropdown to add the Pub/Sub Publisher role.
Click Add another role and add Pub/Sub Subscriber.
Click Continue. You do not need to grant users access to this service account.
Click Create Key. The key is used by the client library to access the Pub/Sub API.
Select JSON and click Create.
The key is sent to your Downloads folder. For the purposes of this tutorial, you can leave it there.
Rename the key file to ~/Downloads/key.json.
Install the Cloud SDK
Follow the instructions for installing and initializing the Cloud SDK.
While initializing the Cloud SDK, select the option to enter a project ID and enter the ID of the project that you created or chose in the setup section.
You can return to this tutorial after you install and initialize the Cloud SDK. You do not need to install other components or download the Cloud Client Libraries.
After you have installed the Cloud SDK, you can use the
gcloud
command-line tool perform Pub/Sub operations in Compute Engine.Start a new terminal before using these gcloud commands:
gcloud pubsub topics list gcloud pubsub subscriptions list
You can also use the
gcloud config set project PROJECT_ID
to change the project from the one you set up during initialization.
Get Python and set up a virtual environment
Complete the instructions for installing Python 3 on macOS.
(Optional) Set up a virtual environment.
This tutorial provides a usage example, so you don't need to follow the example shown in the virtual environment setup section. You can return to this tutorial after installing the virtual environment.
Check out the publisher and subscriber code
Create a project folder to contain the Pub/Sub Python files needed for this tutorial. Then change to it and download the code:
git clone https://github.com/googleapis/python-pubsub.git
Close any open terminals before proceeding.
Set up three terminals
Start one terminal for each tutorial application (one publisher and two subscribers). In each of the terminals, perform all of the operations in this section. For convenience, we refer to these terminals as:
- publisher terminal
- sub_one terminal
- sub_two terminal
Create a Python virtual environment and activate it.
In the first terminal, run this command to create and activate a virtual environment named
pyenv-qs
:Bash
python -m venv pyenv-qs source pyenv-qs/bin/activate
PowerShell
py -m venv pyenv-qs .\pyenv-qs\Scripts\activate
In the other two terminals, run the following command:
Bash
source pyenv-qs/bin/activate
PowerShell
.\pyenv-qs\Scripts\activate
After you run the activate command, your command prompt should include
(pyenv-qs) $
.You can also point your virtual environment to a different Python version.
Install the Pub/Sub Python client library using
pip
:pip install --upgrade google-cloud-pubsub
Associate the JSON key with the service account. You assigned the key Pub/Sub roles when you created the service account credentials. The Pub/Sub client libraries access the environment variable
GOOGLE_APPLICATION_CREDENTIALS
and are granted the roles and permissions associated with the service account.Bash
export GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/key.json
PowerShell
$env:GOOGLE_APPLICATION_CREDENTIALS="$HOME\Downloads\key.json"
Set up an environment variable with your current project ID. This gcloud command determines your currently selected project ID and sets it as a variable:
Bash
export PROJECT=
gcloud config get-value project
PowerShell
$env:PROJECT=$(gcloud config get-value project)
To verify that your current GCP is correctly registered as this variable:
Bash
echo $PROJECT
PowerShell
$env:PROJECT
Change to your project folder, then navigate to the tutorial sample folder:
cd python-pubsub/samples/snippets/quickstart/
Start the apps and observe message flow
Start the Subscriber 1 application
In the sub_one terminal, start Subscriber 1:
Bash
python sub.py $PROJECT sub_one
PowerShell
py sub.py $env:PROJECT sub_one
Once started, this application opens a bidirectional streaming connection with the server. Pub/Sub delivers messages over the stream.
Start the Publisher application
In the publisher terminal, start the Publisher application:
Bash
python pub.py $PROJECT hello_topic
PowerShell
py pub.py $env:PROJECT hello_topic
After the publisher application starts, the Pub/Sub system does the following:
The Publisher application sends a "Hello, World!" message to Pub/Sub while remaining unaware of any existing subscriptions. The server also assigns a message ID.
The Subscriber 1 application receives the 'Hello World' message, prints it, and sends an acknowledgment to Pub/Sub.
The Publisher application prints the acknowledgement. The acknowledgment tells Pub/Sub that the message has been processed successfully and does not need to be re-sent to this or any other sub_one subscriber.
Pub/Sub removes the message from sub_one.
Start the Subscriber 2 application
In the sub_two terminal, start Subscriber 2:
Bash
python sub.py $PROJECT sub_two
PowerShell
py sub.py $env:PROJECT sub_two
This subscriber receives messages delivered to the sub_two subscription.
Subscriber 2 reuses the sub.py
script. The difference is that Subscriber
2 isn't started until after the Publisher has sent the message to the topic
and subscriptions. If Publisher were calling Subscriber 2 directly, the
publishing application would either have to wait until Subscriber 2 comes up
or it would have to time out. Pub/Sub manages this process by
effectively saving the message for Subscriber 2.
You are now ready to develop with Pub/Sub!
How did it go?
Additional resources and links are available on the Pub/Sub support page.
Clean up
Stop all running applications.
Delete the
~/pubsub-quickstart
directory from your local environment.Shut down the tutorial project in the IAM & admin section of the Google Cloud Console.
Remove the service account credentials:
rm ~/Downloads/key.json
What's next
Here are some things that you can try:
Examine the tutorial's
pub.py
andsub.py
code and browse other Pub/Sub samples on github. As an exercise, create a version ofpub.py
that publishes the local time every second.Learn to batch messages.
Using Push subscriptions, receive messages that trigger App Engine endpoints or Cloud Functions.
Retrieve previously acknowledged messages using replay. By default, Pub/Sub removes acknowledged messages from subscriptions. In this tutorial, for instance, you would not be able to rerun
sub.py
to receive the "Hello, World!" message again. The replay feature allows you to set up subscriptions so that you can receive messages after they have been acknowledged.