Edit on GitHub
Report issue
Page history

Ruby on Rails background processing on App Engine with ActiveJob and Sidekiq

Author(s): @chingor13 @mohayat ,   Published: 2017-06-08

Jeff Ching | Software Engineer | Google

Contributed by Google employees.

This tutorial shows how to create and configure a Ruby on Rails application to run background processing jobs on the App Engine flexible environment using ActiveJob and Sidekiq.

Objectives

  • Create a background processing job.
  • Deploy your application to the App Engine flexible environment.
  • Verify that background jobs are running.

Before you begin

You'll need the following:

  • A Google Cloud project. You can use an existing project or create a new project.
  • Ruby 2.2.2+ installed.
  • A Rails 4.2+ application. Follow the "Getting started with Rails" guide to get started.
  • Cloud SDK installed.
  • A Redis instance running in your project. Follow this guide to set up Redis on Compute Engine. This tutorial assumes the Redis instance is running in the default network so that the App Engine services can access it without restriction. Save or copy the password that you set for the Redis instance, which is used later in this tutorial.

Costs

This tutorial uses billable components of Google Cloud including App Engine flexible environment.

Use the pricing calculator to generate a cost estimate based on your projected usage. Google Cloud users might be eligible for a free trial.

Creating your background job

Starting with version 4.2, Ruby on Rails includes an abstraction layer around background job processing called ActiveJob. This abstraction allows you to write the queuing and execution logic independently of queue and job runner implementations.

Ruby on Rails provides command-line tools for generating templated skeletons for things such as database migrations, controllers, and even background jobs.

You create a job named HelloJob that accepts a name argument and prints "Hello #{name}" to standard output.

  1. Use the Rails generator feature to create the HelloJob job:

    bin/rails generate job Hello
    

    Rails creates stub files from templates:

    invoke  test_unit
    create    test/jobs/hello_job_test.rb
    create  app/jobs/hello_job.rb
    create  app/jobs/application_job.rb
    
  2. Edit app/jobs/hello_job.rb with the following:

    class HelloJob < ApplicationJob
      queue_as :default
    
      def perform(name)
        # Do something later
        puts "Hello, #{name}"
      end
    end
    

Create a test URL to queue the job

You create a controller named HelloController that provides an action called say that queues the HelloJob job to run in the background.

  1. Use the Rails generator feature to create HelloController:

    bin/rails generate controller Hello
    

    Rails creates stub files from templates:

    create  app/controllers/hello_controller.rb
    invoke  erb
    create    app/views/hello
    invoke  test_unit
    create    test/controllers/hello_controller_test.rb
    invoke  helper
    create    app/helpers/hello_helper.rb
    invoke    test_unit
    invoke  assets
    invoke    coffee
    create      app/assets/javascripts/hello.coffee
    invoke    scss
    create      app/assets/stylesheets/hello.scss
    
  2. Add a say action to HelloController by adding the following to your app/controllers/hello_controller.rb file:

    class HelloController < ApplicationController
      def say
        HelloJob.perform_later(params[:name])
        render plain: 'OK'
      end
    end
    

    This action will queue our HelloJob with the provided request parameter name.

  3. Create a route to this action by adding the following to your config/routes.rb file:

    get '/hello/:name', to: 'hello#say'
    

    When you make an HTTP GET request to /hello/Jeff, the HelloController will handle the request using the say action with parameter :name as "Jeff".

Configuring your background worker to use Sidekiq

ActiveJob can be configured with various different background job runners. This tutorial uses Sidekiq, which requires a Redis instance to manage the job queue.

  1. Add sidekiq gem to your Gemfile:

    bundle add sidekiq
    
  2. Configure ActiveJob to use Sidekiq as its queue adapter. In config/application.rb:

    class Application < Rails::Application
      # ...
      config.active_job.queue_adapter = :sidekiq
    end
    

Deploying to App Engine flexible environment

For Sidekiq, the Redis connection configuration can be provided as an environment variable at run time. You need to obtain the internal address and password of your Redis instance. In the Cloud Console, go to the VM instances page and find the internal IP address of your Compute Engine instance with Redis installed. This IP address and the password that you saved are be provided through environment variables at deployment time to configure Sidekiq.

Option A: Shared worker and web application

For this option, the App Engine service runs both the web server and a worker process through a process manager called foreman. If you choose this method, App Engine scales your web and worker instances together.

  1. Add foreman gem to your Gemfile:

    bundle add foreman
    
  2. Create a Procfile at the root of your application:

    web: bundle exec rails server -p 8080
    worker: bundle exec sidekiq
    
  3. Create an app.yaml for deploying the application to Google App Engine:

    runtime: ruby
    env: flex
    
    entrypoint: bundle exec foreman start
    
    env_variables:
      REDIS_PROVIDER: REDIS_URL
      REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
      REDIS_PASSWORD: [PASSWORD]
      SECRET_KEY_BASE: [SECRET_KEY]
    

    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace [SECRET_KEY] with a secret key for Rails sessions.

Sidekiq initializer file

If your application uses Rails secret credentials, then you can omit the env_variable field in your app.yaml file, and instead supply the variables with the credentials.enc.yml file in an initializer.

  1. Open the credentials.enc.yml file for editing:

    rails credentials:edit
    

    This takes your master.key, decodes the file, and opens it in your default text editor for you to edit.

  2. Add the following entries:

    redis_url: redis://[REDIS_IP_ADDRESS]:6379
    redis_password: [PASSWORD]
    
  3. Save the file.

    It is re-encoded with your two new entries.

  4. Create a new file called sidekiq.rb in config/initializers/, with the following content:

    Sidekiq.configure_server do |config|
      if Rails.env.production?
        config.redis = {
          url: Rails.application.credentials.redis_url,
          password: Rails.application.credentials.redis_password
        }
      else
        config.redis = { url: 'redis://localhost:6379/1' }
      end
    end
    
    Sidekiq.configure_client do |config|
      if Rails.env.production?
        config.redis = {
          url: Rails.application.credentials.redis_url,
          password: Rails.application.credentials.redis_password
        }
      else
        config.redis = { url: 'redis://localhost:6379/1' }
      end
    end
    

    This ensures that Redis will seek the localhost port that Redis is using when you're testing locally in a development environment.

  5. Deploy to App Engine:

    gcloud app deploy app.yaml
    

Option B: Separate worker and web application

For this option, you create two App Engine services: one runs the web server and one runs worker processes. Both services use the same application code. This configuration allows you to scale background worker instances independently of your web instances at the cost of potentially using more resources. To pass the App Engine health checks and keep your background worker instance alive, you use the sidekiq_alive gem to enable the Sidekiq server to respond to each liveness and readiness request with a 200 HTTP status code.

  1. Add sidekiq_alive to your Gemfile:

    bundle add sidekiq_alive
    
  2. Create a sidekiq_alive.rb initializer. In config/initializers:

    SidekiqAlive.setup do |config|
      # ==> Server port
      # Port to bind the server.
      config.port = 8080
    
      # ==> Server path
      # HTTP path to respond to.
      config.path = '/health_check'
    
      # ==> Rack server
      # Web server used to serve an HTTP response.
      config.server = 'puma'
    end
    
  3. Create an app.yaml file for deploying the web service to App Engine:

    runtime: ruby
    env: flex
    
    entrypoint: bundle exec rails server -p 8080
    
    env_variables:
      REDIS_PROVIDER: REDIS_URL
      REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
      REDIS_PASSWORD: [PASSWORD]
      SECRET_KEY_BASE: [SECRET_KEY]
    

    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace [SECRET_KEY] with a secret key for Rails sessions.

  4. Create a worker.yaml file for deploying the worker service to App Engine:

    runtime: ruby
    env: flex
    service: worker
    
    entrypoint: bundle exec sidekiq
    
    env_variables:
      REDIS_PROVIDER: REDIS_URL
      REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
      REDIS_PASSWORD: [PASSWORD]
      SECRET_KEY_BASE: [SECRET_KEY]
    
    liveness_check: 
      path: '/health_check'
    
    readiness_check:
      path: '/health_check'
    
    # Optional scaling configuration
    manual_scaling:
      instances: 1
    

    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace the [SECRET_KEY] with a secret key for Rails sessions.

    The path attribute for both the liveness_check and readiness_check sections has been set to the value of config.path in your sidekiq_alive.rb initializer.

    As mentioned above, you can configure scaling for the worker service independent of the default (web) service. In the manual_scaling section, you have configured the worker service to start with one worker instance. For more information on scaling options, see scaling configuration options in app.yaml. If you choose an automatic_scaling option, be aware that scaling for the background processing is based off of CPU utilization, not queue size.

  5. Deploy both services to App Engine:

    gcloud app deploy app.yaml worker.yaml
    

Verify that background queuing works

  1. In the Cloud Console, go to the App Engine services page. Locate the service that is running your background workers (if option A, it should be the default service, if option B, it should be the worker service). Click Tools > Logs for that service.

  2. In a separate window, navigate to your deployed Rails application:

    https://[YOUR_PROJECT_ID].appspot.com/hello/Jeff
    

    Replace [YOUR_PROJECT_ID] with your Google Cloud project ID.

  3. Navigate back to the Logs dashboard. In the Filter by label or text search field, add "Hello, Jeff" and you should see a logging statement like the following if using foreman:

    13:13:52.000 worker.1 | Hello, Jeff
    

    Or, if using a second service:

    13:13:52.000 Hello, Jeff
    

Congratulations! You have successfully set up background job processing on App Engine with Sidekiq.

Cleaning up

After you've finished this tutorial, you can clean up the resources you created on Google Cloud so you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.

Deleting the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

  1. In the Cloud Console, go to the Projects page.
  2. Click the trash can icon to the right of the project name.

Warning: Deleting a project has the following consequences:

If you used an existing project, you'll also delete any other work you've done in the project. You can't reuse the project ID of a deleted project. If you created a custom project ID that you plan to use in the future, you should delete the resources inside the project instead. This ensures that URLs that use the project ID, such as an appspot.com URL, remain available.

Deleting App Engine services

To delete an App Engine service:

  1. In the Cloud Console, go to the App Engine Services page.
  2. Click the checkbox next to the service you wish to delete.
  3. Click Delete at the top of the page to delete the service.

If you are trying to delete the default service, you cannot. Instead, do the following:

  1. Click on the number of versions which will navigate you to App Engine Versions page.
  2. Select all the versions you wish to disable and click Stop at the top of the page. This will free all of the Google Compute Engine resources used for this App Engine service.

Next steps

Submit a tutorial

Share step-by-step guides

Submit a tutorial

Request a tutorial

Ask for community help

Submit a request

View tutorials

Search Google Cloud tutorials

View tutorials

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.