This page explains how to migrate pull queue code from Task Queues to Cloud Pub/Sub. Cloud Pub/Sub is now the preferred way of performing pull queue work in App Engine.
If your app uses both pull queues and push queues, use this guide to migrate
your pull queues to Cloud Pub/Sub before migrating your push queues to
the new push queue service Cloud Tasks. Migrating your pull queues
after migrating push queues to Cloud Tasks is not recommended
because the required use of the
queue.yaml file is likely to cause unexpected
behavior with Cloud Tasks.
Features not currently available in Cloud Pub/Sub
The following Task Queues features are not currently available in Cloud Pub/Sub:
- Batching by tag
- Automatic deduplication
Pricing and quotas
Migrating your pull queues to Cloud Pub/Sub might affect the pricing and quotas for your app.
Cloud Pub/Sub has its own pricing. As with Task Queues, sending requests to your App Engine app with Cloud Pub/Sub can cause your app to incur costs.
This section discusses topics you need to cover before migrating your pull queues to Cloud Pub/Sub.
Enabling the Cloud Pub/Sub API
To enable the Cloud Pub/Sub API, click Enable on the Cloud Pub/Sub API in the API Library. If you see a Manage button instead of an Enable button, you have previously enabled the Cloud Pub/Sub API for your project and do not need to do so again.
Authenticating your app to the Cloud Pub/Sub API
You must authenticate your app to the Cloud Pub/Sub API. This section discusses authentication for two different use cases.
To develop or test your app locally, we recommend using a service account. For instructions on setting up a service account and connecting it to your app, read Obtaining and providing service account credentials manually.
To deploy your app on App Engine, you do not need to provide any new authentication. The Application Default Credentials (ADC) infer authentication details for App Engine apps.
Downloading the Cloud SDK
Download and install the
Cloud SDK to use the
gcloud tool with the Cloud Pub/Sub API if you have not
installed it previously. Run the following command from your terminal if you
already have the Cloud SDK installed.
gcloud components update
Importing the Python client libraryFollow the steps below to use the Cloud Pub/Sub Python client library with your App Engine app:
Create a directory to store your third-party libraries, such as
Copy the necessary libraries.
We recommend using a
piprequirements file rather than installing the libraries one by one from the command line. Create a
requirements.txtfile in the same folder as your
app.yamlfile if you do not already have a
requirements.txtfile. Add the following line:
pip(version 6 or later) with the
-t <directory>flag to copy the Cloud Pub/Sub library you specified in your
requirements.txtfile into the folder you created in the previous step. For example:
pip install -t lib -r requirements.txt
Specify the RPC library in the
librariessection of your
libraries: - name: grpcio version: 1.0.0
pkg_resourcesmodule to ensure that your app uses the right distribution of the Cloud Pub/Sub Python client library.
appengine_config.pyfile in the same folder as your
app.yamlfile if you do not already have one. Add the following to your
# appengine_config.py import pkg_resources from google.appengine.ext import vendor # Set path to your libraries folder. path = 'lib' # Add libraries installed in the path folder. vendor.add(path) # Add libraries to pkg_resources working set to find the distribution. pkg_resources.working_set.add_entry(path)
appengine_config.pyfile above assumes that the current working directory is where the
libfolder is located. In some cases, such as unit tests, the current working directory can be different. To avoid errors, you can explicitly pass in the full path to the
path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
Import the Cloud Pub/Sub Python client library in any files that use pull queues from the Task Queues API:
from google.cloud import pubsub_v1
Cloud Pub/Sub and pull queues
Cloud Pub/Sub sends work to workers through a publisher/subscriber relationship. A pull subscription in Cloud Pub/Sub is like a pull queue in Task Queues because the subscriber pulls the message from the topic. The table below lists the core feature for pull queues in Task Queues and the associated feature for pull subscriptions in Cloud Pub/Sub.
|Task Queues feature||Cloud Pub/Sub feature|
To learn more about the Cloud Pub/Sub architecture, read Cloud Pub/Sub: A Google-Scale Messaging Service.
Below is a comparison of a typical workflow for a pull queue in Task Queues and a pull subscription in Cloud Pub/Sub.
|Task Queues workflow||Cloud Pub/Sub workflow|
|You create the pull queue||You create the topic and subscribe your subscriber (i.e. worker) to the topic|
|You create and enqueue the task||You create the message and publish it to the topic|
|The worker leases the task||The subscriber pulls the message from the topic|
|The worker processes the task||The subscriber processes the message|
|The worker deletes the task from the queue||The subscriber acknowledges the message|
|The lease expires||The topic deletes the message when all of its subscribers have acknowledged the message|
Creating pull subscriptions in Cloud Pub/Sub
You can use a Cloud Pub/Sub pull subscription like a Task Queues pull queue. Subscriptions to a topic do not expire and can exist simultaneously for multiple workers. This means that a message can be processed by more than one worker, which is one of the primary use cases for Cloud Pub/Sub. To recreate your Task Queues pull queues as Cloud Pub/Sub pull subscriptions, create a topic for each worker and subscribe only the associated worker to the topic. This ensures that each message is processed by exactly one worker as in Task Queues. To learn about creating and managing pull subscriptions, read about managing topics and subscriptions.
Deleting pull queues
After you migrate your Task Queues pull queues to
Cloud Pub/Sub pull subscriptions, delete them from
Task Queues using your
queue.yaml file. We recommend deleting
each pull queue before migrating the next one. This prevents your app from
duplicating the work it receives from the new Cloud Pub/Sub pull
subscription while you migrate your other pull queues. Note that deleting your
Task Queues pull queues one by one rather than in a single
deployment might have a greater effect on your App Engine
After you delete all your pull queues from Task Queues, you can
queue.yaml file from future deployments of your app.
If your app only uses pull queues, remove any references to the Task Queues API in your code. If your app uses both pull queues and push queues, you can either remove the references to the Task Queues API that occur in files that only use pull queues, or wait until you have also migrated your push queues and then remove references to the Task Queues API from all files.