Using Pub/Sub with Ruby

Many apps need to do background processing outside of the context of a web request. In this sample, the Bookshelf app sends tasks to a separate background worker for execution. The worker gathers information from the Google Books API and updates the book information in the database. This sample demonstrates how to set up separate services in App Engine, how to run a worker process in the App Engine flexible environment, and how to deal with lifecycle events.

This page is part of a multipage tutorial. To start from the beginning and read the setup instructions, go to Ruby Bookshelf app.

Installing dependencies

Go to the getting-started-ruby/6-task-queueing directory, and enter the following command.

bundle install

Creating a Pub/Sub topic and subscription

The Bookshelf app uses Pub/Sub for a background processing queue of requests to get data from the Google Books API for a book added to the Bookshelf.

  1. Create a new Pub/Sub topic using the following Cloud SDK command where [YOUR_PUBSUB_TOPIC] represents your Pub/Sub topic name.

    gcloud pubsub topics create [YOUR_PUBSUB_TOPIC]
  2. Create a new Pub/Sub subscription for the topic created in the previous step. Replace [YOUR_PUBSUB_SUBSCRIPTION] with the name you want to give to this new Pub/Sub subcritpion.

    gcloud pubsub subscriptions create --topic [YOUR_PUBSUB_TOPIC] [YOUR_PUBSUB_SUBSCRIPTION]

Configuring settings

  1. Copy the example settings file.

    cp config/settings.example.yml config/settings.yml
  2. Open the settings.yml file for editing. Replace the [YOUR_PROJECT_ID] with your Google Cloud project ID.

    default: &default
      project_id: [YOUR_PROJECT_ID]
      gcs_bucket: [YOUR_BUCKET_NAME]
      pubsub_topic: [YOUR_PUBSUB_TOPIC]
      pubsub_subscription: [YOUR_PUBSUB_SUBSCRIPTION]
        client_id: [YOUR_CLIENT_ID]
        client_secret: [YOUR_CLIENT_SECRET]
  3. Set the other variables to the same values you used in the Authenticating users part of this tutorial.

    For example, suppose your web app's client ID is XYZCLIENTID and your client secret is XYZCLIENTSECRET. Also suppose your project name is my-project, and your Cloud Storage bucket name is my-bucket. Then the default section of your settings.yml file would look like this:

    default: &default
      project_id: my-project
      gcs_bucket: my-bucket
      pubsub_topic: your-pubsub-topic
      pubsub_subscription: your-pubsub-subscription
        client_id: XYZCLIENTID
        client_secret: XYZCLIENTSECRET
  4. Copy the example database configuration file.

    cp config/database.example.yml config/database.yml
  5. Configure the sample app to use the same database that you set up during the Using structured data part of this tutorial.

    Cloud SQL

    • Edit database.yml. Uncomment the lines in the Cloud SQL portion of the file.

       mysql_settings: &mysql_settings
         adapter: mysql2
         encoding: utf8
         pool: 5
         timeout: 5000
         username: [MYSQL_USER]
         password: [MYSQL_PASS]
         database: [MYSQL_DATABASE]
         socket: /cloudsql/[YOUR_INSTANCE_CONNECTION_NAME]
      • Replace [MYSQL_USER] and [MYSQL_PASS] with your Cloud SQL instance username and password that you created previously.

      • Replace [MYSQL_DATABASE] with the name of the database that you created previously.

      • Replace [YOUR_INSTANCE_CONNECTION_NAME] with the Instance Connection Name of your Cloud SQL instance.

    • Run migrations.

      bundle exec rake db:migrate


    • Edit database.yml. Uncomment the lines in the PostgreSQL portion of the file. Replace the your-postgresql-* placeholders with the values for your PostgreSQL instance and database. For example, suppose your IPv4 address is, your username is postgres, and your password is pword123. Also suppose your database name of bookshelf. Then the PostgreSQL portion of your database.yml file would look like this:

      # PostgreSQL Sample Database Configuration
      # ----------------------------------------
        adapter: postgresql
        encoding: unicode
        pool: 5
        username: postgres
        password: pword123
        database: bookshelf
    • Create the required database and tables.

      bundle exec rake db:create
      bundle exec rake db:migrate


    • Edit database.yml. Uncomment the one line in the Datastore portion of the file. Replace your-project-id with your Google Cloud project ID. For example, suppose your project ID is my-project: Then the Datastore portion of your database.yml file would look like this:

      # Google Cloud Datastore Sample Database Configuration
      # ----------------------------------------------------
      dataset_id: my-project
    • Run a rake task to copy the sample project files for Datastore.

      bundle exec rake backend:datastore

Running the app on your local machine

  1. Start the local web server and two worker processes.

    bundle exec foreman start --formation web=1,worker=2
  2. In your web browser, enter http://localhost:8080.

Now add some books to the Bookshelf. You can watch the workers update the book information in the background.

The Foreman RubyGem starts the Rails web server and runs two worker processes.

The worker establishes a Pub/Sub subscription to listen for events. After the subscription exists, events published to the topic are queued, even if there is no worker currently listening for events. When a worker comes online, Pub/Sub delivers any queued events.

When you're ready to move forward, press Ctrl+C to exit the local web server and worker processes.

Deploying the app to the App Engine flexible environment

  1. Compile the JavaScript assets for production.

    RAILS_ENV=production bundle exec rake assets:precompile
  2. Deploy the worker.

    gcloud app deploy worker.yaml
  3. Deploy the sample app.

    gcloud app deploy
  4. In your web browser, enter the following address.


If you update your app, you can deploy the updated version by entering the same command you used to deploy the app the first time. The new deployment creates a new version of your app and promotes it to the default version. The older versions of your app remain, as do their associated VM instances. Be aware that all of these app versions and VM instances are billable resources.

You can reduce costs by deleting the non-default versions of your app.

To delete an app version:

  1. In the Cloud Console, go to the Versions page for App Engine.

    Go to the Versions page

  2. Select the checkbox for the non-default app version you want to delete.
  3. Click Delete to delete the app version.

For complete information about cleaning up billable resources, see the Cleaning up section in the final step of this tutorial.

App structure

This diagram shows the app's components and how they fit together.

Auth sample structure

Understanding the code

This section walks you through the app's code and explains how it works.

Queue tasks

To gather information from the Google Books API for books added to the Bookshelf, the Book class adds a task to the queue.

after_create :lookup_book_details

def lookup_book_details
  if [author, description, published_on, image_url].any? {|attr| attr.blank? }
    LookupBookDetailsJob.perform_later self

The preceding code creates an Active Record callback and specifies that after a book is created and saved in the database, lookup_book_details is called. If the book is missing any information, it adds the job, look up the book's details, to the queue.

LookupBookDetailsJob is an Active Job job.

The code passes self, referencing the book, to LookupBookDetailsJob.perform_later. This adds a job to look up the book's details to the queue.

Pub/Sub Active Job backend

You can configure Active Job to use a custom backend. For example, use delayed_job or resque, to add tasks to the queue. The Bookshelf sample app has its own custom backend, which is specified in the Application class.

config.active_job.queue_adapter = :pub_sub_queue

An Active Job backend, which is also called an adapter, must provide an enqueue method. When a job is enqueued using perform_later, the job is passed to the enqueue method of the configured Active Job backend.

The sample app adds a job to the queue by creating a subscription to a Pub/Sub topic, and then publishing the ID of a book to the topic. Once the subscription exists, messages are queued even if there is no worker currently listening. When a worker comes online, Pub/Sub delivers any queued events.

require "google/cloud/pubsub"

module ActiveJob
  module QueueAdapters
    class PubSubQueueAdapter

      def self.pubsub
        @pubsub ||= begin
          project_id = Rails.application.config.x.settings["project_id"]
 project_id: project_id

      def self.pubsub_topic
        @pubsub_topic ||= Rails.application.config.x.settings["pubsub_topic"]

      def self.pubsub_subscription
        @pubsub_subscription ||= Rails.application.config.x.settings["pubsub_subscription"]

      def self.enqueue job "[PubSubQueueAdapter] enqueue job #{job.inspect}"

        book  = job.arguments.first

        topic = pubsub.topic pubsub_topic


The preceding code uses the google-cloud-pubsub RubyGem to interact with Pub/Sub. The Cloud client library is an idiomatic Ruby client for interacting with Google Cloud services.

gem "google-cloud-pubsub", "~> 0.30"

To process books added to a queue, a Pub/Sub subscription listens for messages published to the lookup_book_details_queue topic. This is covered in the worker section.

Books API

The sample app uses the Google API client RubyGem to look up book details from the Books API.

gem "google-api-client", "~> 0.19"

When a job runs, the LookupBookDetailsJob.perform method retrieves a list of books, based on a book title, from the Books API.

require "google/apis/books_v1"

class LookupBookDetailsJob < ActiveJob::Base
  queue_as :default

  def perform book "[BookService] Lookup details for book" +
                      "#{} #{book.title.inspect}"

    # Create Book API Client
    book_service =

    # Lookup a list of relevant books based on the provided book title.
    book_service.list_volumes(book.title, order_by: "relevance") do |results, error|
      # Error ocurred soft-failure
      if error
        Rails.logger.error "[BookService] #{error.inspect}"

      # Book was not found
      if "[BookService] #{book.title} was not found."

      # List of relevant books
      volumes = results.items

If a book volume result includes a title, author, and book cover image, then it is selected as the best match. Otherwise the first result is used.

# To provide the best results, find the first returned book that
# includes title and author information as well as a book cover image.
best_match = volumes.find {|volume|
  info = volume.volume_info
  info.title && info.authors && info.image_links.try(:thumbnail)

volume = best_match || volumes.first

If any relevant volume is found, the book details are updated and saved in the database.

if volume
  info   = volume.volume_info
  images = info.image_links

  publication_date = info.published_date
  publication_date = "#{$1}-01-01" if publication_date =~ /^(\d{4})$/
  publication_date = Date.parse publication_date       = info.authors.join(", ") unless
  book.published_on = publication_date unless book.published_on.present?
  book.description  = info.description unless book.description.present?
  book.image_url    = images.try(:thumbnail) unless book.image_url.

The worker

A worker process handles book lookup jobs. To run the worker, you can run the following command, as specified in Procfile.

bundle exec rake run_worker

The run_worker rake task calls PubSubQueueAdapter to start a worker.

desc "Run task queue worker"
task run_worker: :environment do

When the worker runs, it listens for messages on the Pub/Sub subscription to the lookup_book_details_queue topic defined in your config/settings.yml file. When a message is received, the associated book is retrieved from the database and the LookupBookDetailsJob runs immediately to update the book.

def self.run_worker! "Running worker to lookup book details"

  topic        = pubsub.topic pubsub_topic
  subscription = topic.subscription pubsub_subscription

  subscriber = subscription.listen do |message|
    message.acknowledge! "Book lookup request (#{})"

    book_id =
    book    = Book.find_by_id book_id

    LookupBookDetailsJob.perform_now book if book

  # Start background threads that will call block passed to listen.

  # Fade into a deep sleep as worker will run indefinitely

Running on Google Cloud

The worker is deployed as a separate module within the same app. App Engine apps can have multiple, independent services. This means that you can independently deploy, configure, scale, and update pieces of your app. The frontend is deployed to the default module, and the worker is deployed to the worker module.

Even though the worker doesn't serve any web requests to users, or even run a web app, we strongly recommend that you provide an HTTP health check when running in the App Engine flexible environment to ensure that the service is running and responsive. It is, however, possible to disable health checking.

To provide a health check, the worker starts two processes instead of one. The first process is worker and the second process is health_check, which runs a simple Rack app that responds to HTTP requests with a successful response for health checks.

# Respond to HTTP requests with non-500 error code
run lambda {|env| [200, {"Content-Type" => "text/plain"}, ["ok"]] }

The app uses Foreman to manage multiple processes. The processes are configured in Procfile.

web: bundle exec rackup -p 8080
worker: bundle exec rake run_worker
health_check: bundle exec rackup -p 8080

Foreman is now used as the entrypoint for the docker container. This is specified in the app.yaml and worker.yaml files.

entrypoint: bundle exec foreman start --formation "$FORMATION"

Notice that Procfile contains an entry for the web frontend to run the Bookshelf Rails app as well. Because the default (frontend) and worker services share the same codebase, the FORMATION environment variable controls which processes are started. The following diagram contrasts the single module deployment on the left with the multi-module deployment on the right.

Pub/Sub deployment

The environment variables are set by the app.yaml and worker.yaml files.

  FORMATION: web=1

The worker is a separate module, so it needs its own YAML configuration file.

  FORMATION: worker=5,health_check=1

This configuration is similar to the app.yaml file that is used for the frontend; the key differences are the module: worker setting, and the FORMATION environment variable, which configures Foreman to run five workers and the frontend for the health check instead of the Bookshelf web app.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project you want to delete and click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete non-default versions of your app

If you don't want to delete your project, you can reduce costs by deleting the non-default versions of your app.

To delete an app version:

  1. In the Cloud Console, go to the Versions page for App Engine.

    Go to the Versions page

  2. Select the checkbox for the non-default app version you want to delete.
  3. Click Delete to delete the app version.

Delete your Cloud SQL instance

To delete a Cloud SQL instance:

  1. In the Cloud Console, go to the SQL Instances page.

    Go to the SQL Instances page

  2. Click the name of the SQL instance you want to delete.
  3. Click Delete to delete the instance.

Delete your Cloud Storage bucket

To delete a Cloud Storage bucket:

  1. In the Cloud Console, go to the Cloud Storage Browser page.

    Go to the Cloud Storage Browser page

  2. Click the checkbox for the bucket you want to delete.
  3. To delete the bucket, click Delete .

What's next

Learn how to run the Ruby Bookshelf sample app on Compute Engine.

Try out other Google Cloud Platform features for yourself. Have a look at our tutorials.

Σας βοήθησε αυτή η σελίδα; Πείτε μας τη γνώμη σας:

Αποστολή σχολίων σχετικά με…

Αυτή η σελίδα