Migrating pull queues to Pub/Sub (Python)

This page explains how to migrate pull queue code from Task Queues to Pub/Sub. Pub/Sub is now the preferred way of performing pull queue work in App Engine.

If your app uses both pull queues and push queues, use this guide to migrate your pull queues to Pub/Sub before migrating your push queues to the new push queue service Cloud Tasks. Migrating your pull queues after migrating push queues to Cloud Tasks is not recommended because the required use of the queue.yaml file is likely to cause unexpected behavior with Cloud Tasks.

Features not currently available in Pub/Sub

The following Task Queues features are not currently available in Pub/Sub:

Batching by tag
Automatic deduplication

Pricing and quotas

Migrating your pull queues to Pub/Sub might affect the pricing and quotas for your app.

Pricing

Pub/Sub has its own pricing. As with Task Queues, sending requests to your App Engine app with Pub/Sub can cause your app to incur costs.

Quotas

The Pub/Sub quotas are different from the quotas for Task Queues. Like with Task Queues, sending requests to your App Engine app from Pub/Sub might impact your App Engine request quotas.

Before migrating

If you have not done so already, set up your Python development environment to use a Python version that is compatible with Google Cloud, and install testing tools for creating isolated Python environments.

The following sections discuss the setup steps before migrating your pull queues to Pub/Sub.

Enabling the Pub/Sub API

To enable the Pub/Sub API, click Enable on the Pub/Sub API in the API Library. If you see a Manage button instead of an Enable button, you have previously enabled the Pub/Sub API for your project and do not need to do so again.

Authenticating your app to the Pub/Sub API

You must authenticate your app to the Pub/Sub API. This section discusses authentication for two different use cases.

To develop or test your app locally, we recommend using a service account. For instructions on setting up a service account and connecting it to your app, read Obtaining and providing service account credentials manually.

To deploy your app on App Engine, you do not need to provide any new authentication. The Application Default Credentials (ADC) infer authentication details for App Engine apps.

Downloading the Google Cloud CLI

Download and install the Google Cloud CLI to use the gcloud CLI with the Pub/Sub API if you have not installed it previously. Run the following command from your terminal if you already have the Google Cloud CLI installed.

gcloud components update

Importing the Cloud Client Libraries

Follow the steps below to use the Pub/Sub Python client library with your existing App Engine app:

Update the app.yaml file. Follow the instructions for your version of Python:
Python 2
For Python 2 apps, add the latest versions of the grpcio library.

The following is an example app.yaml file:
```
runtime: python27
threadsafe: yes
api_version: 1

libraries:
- name: grpcio
  version: latest
```
Python 3
For Python 3 apps, specify the runtime element in your app.yaml file with a supported Python 3 version. For example:
```
runtime: python310 # or another support version
```
The Python 3 runtime installs libraries automatically, so you do not need to specify built-in libraries from the previous Python 2 runtime. If your Python 3 app is using other legacy bundled services when migrating, you can continue to specify the necessary built-in libraries. Otherwise, you can delete the unnecessary lines in your app.yaml file.
Update the requirements.txt file. Follow the instructions for your version of Python:
Python 2
Add the Cloud Client Libraries for Pub/Sub to your list of dependencies in the requirements.txt file.
```
google-cloud-pubsub
```
Then run pip install -t lib -r requirements.txt to update the list of available libraries for your app.
Python 3
Add the Cloud Client Libraries for Pub/Sub to your list of dependencies in the requirements.txt file.
```
google-cloud-pubsub
```
App Engine automatically installs these dependencies during app deployment in the Python 3 runtime, so delete the lib folder if one exists.
For Python 2 apps, if your app is using built-in or copied libraries, you must specify those paths in the appengine_config.py file, located in the same folder as your app.yaml file:
```
import pkg_resources
from google.appengine.ext import vendor

# Set PATH to your libraries folder.
PATH = 'lib'
# Add libraries installed in the PATH folder.
vendor.add(PATH)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(PATH)
```
The appengine_config.py file above assumes that the current working directory is where the lib folder is located. In some cases, such as unit tests, the current working directory can be different. To avoid errors, you can explicitly pass in the full path to the lib folder using:
```
import os
path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
```
Import the Pub/Sub Python client library in any files that use pull queues from the Task Queues API:
```
from google.cloud import pubsub
```

Pub/Sub and pull queues

Feature comparison

Pub/Sub sends work to workers through a publisher/subscriber relationship. A pull subscription in Pub/Sub is like a pull queue in Task Queues because the subscriber pulls the message from the topic. The table below lists the core feature for pull queues in Task Queues and the associated feature for pull subscriptions in Pub/Sub.

Task Queues feature	Pub/Sub feature
Queue	Topic
Task	Message
Worker	Subscriber

To learn more about the Pub/Sub architecture, read Cloud Pub/Sub: A Google-Scale Messaging Service.

Workflow comparison

Below is a comparison of a typical workflow for a pull queue in Task Queues and a pull subscription in Pub/Sub.

Task Queues workflow	Pub/Sub workflow
You create the pull queue	You create the topic and subscribe your subscriber (i.e. worker) to the topic
You create and enqueue the task	You create the message and publish it to the topic
The worker leases the task	The subscriber pulls the message from the topic
The worker processes the task	The subscriber processes the message
The worker deletes the task from the queue	The subscriber acknowledges the message
The lease expires	The topic deletes the message when all of its subscribers have acknowledged the message

Creating pull subscriptions in Pub/Sub

You can use a Pub/Sub pull subscription like a Task Queues pull queue. Subscriptions to a topic do not expire and can exist simultaneously for multiple workers. This means that a message can be processed by more than one worker, which is one of the primary use cases for Pub/Sub. To recreate your Task Queues pull queues as Pub/Sub pull subscriptions, create a topic for each worker and subscribe only the associated worker to the topic. This ensures that each message is processed by exactly one worker as in Task Queues. To learn about creating and managing pull subscriptions, read about managing topics and subscriptions.

Deleting pull queues

After you migrate your Task Queues pull queues to Pub/Sub pull subscriptions, delete them from Task Queues using your queue.yaml file. We recommend deleting each pull queue before migrating the next one. This prevents your app from duplicating the work it receives from the new Pub/Sub pull subscription while you migrate your other pull queues. Note that deleting your Task Queues pull queues one by one rather than in a single deployment might have a greater effect on your App Engine deployment quota.

After you delete all your pull queues from Task Queues, you can omit the queue.yaml file from future deployments of your app.

If your app only uses pull queues, remove any references to the Task Queues API in your code. If your app uses both pull queues and push queues, you can either remove the references to the Task Queues API that occur in files that only use pull queues, or wait until you have also migrated your push queues and then remove references to the Task Queues API from all files.

What's next

Pub/Sub documentation
Migrating push queues
For a hands-on tutorial, see the Migrate from App Engine Task Queue pull queues to Pub/Sub codelab.