Creating a render farm in GCP using OpenCue

Robot dance party

This tutorial shows how to deploy the OpenCue render management system on Linux virtual machines (VMs) using Google Cloud Platform (GCP). Allow 3–4 hours to complete the entire tutorial.

Render management tools are used in visual effects and animation production for breaking down complex jobs into individual tasks and spreading these processes across a broad range of compute resources. These tools are used to manage jobs, such as 3D renders, 2D compositing, transcoding of data from one format to another, and checking assets in and out of databases. The tools are often used to link job stages and dependencies. At larger facilities, a render management system can support thousands of simultaneous jobs and provides the interface for allocating resources to each stage of a job, managing logs, troubleshooting tools, and managing resources.

Google has collaborated with Sony Pictures Imageworks to release OpenCue, a production-proven, open source render management system that works on Linux and macOS. OpenCue has been used to build on-premises render farms and for hybrid deployments at a very large scale utilizing resources from multiple cloud providers.

This tutorial is intended for IT administrators and pipeline technical directors who are familiar with the following tools and concepts. Don't worry if you aren't an expert at everything in this list; we indicate further resources where applicable.

  • Operating principles of distributed render management.
  • Git and GitHub.
  • Docker for creating and running containers.
  • Bash scripting for editing scripts and running resources.
  • Python configuration and script editing.
  • GCP resource management basic tools and concepts, including Cloud SQL, Compute Engine, Cloud Storage, and Container Registry.

Because render management tools cover various use cases and orchestrate a broad range of resources, there are many ways to configure a render farm. This tutorial focuses on one configuration and gets you up and running with OpenCue as quickly as possible. Creating a more optimal architecture for your render farm might require modifying steps, profiling performance for your workflow, or swapping types of GCP resources. For more information on options for setting up and managing a render farm on GCP, see Building a hybrid render farm. And for more information on helping secure your data, see Securing rendering workloads.

Objectives

  • Install and configure OpenCue modules using Docker and Python.
  • Create an OpenCue database and populate it with schema and example data.
  • Create Cloud Storage resources and populate them with the tutorial's animation scene files.
  • Configure and create Docker images locally, store them in Container Registry, and start Compute Engine resources using containers.
  • Create instance templates and managed instance groups to enable efficient scaling of compute resources.
  • Configure a workstation to run OpenCue's interface on-premises.
  • Render an animation sequence and explore job management using OpenCue's interface.

Costs

You can use the pricing calculator to generate a cost estimate based on your projected usage.

This tutorial covers every step of building out a functioning render farm using tight constraints on machine numbers and sizing. To fit within the GCP Free Tier pricing guidelines, in this tutorial you render an animation with low resolution and sampling rates on a few VMs with two vCPUs.

The cost of the resources that make up this tutorial, if used for a total of 4 hours, totals $3.27.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Google Cloud Platform project.

    Go to the Manage resources page

  3. Make sure that billing is enabled for your Google Cloud Platform project.

    Learn how to enable billing

  4. Enable the Container Registry and Compute Engine APIs.

    Enable the APIs

  5. Make sure you have the following installed on your local machine:
    • Docker This tutorial is tested with Docker Engine version 18.09.0.
    • Python This tutorial is tested with Version 2.7.10.
    • Pip This tutorial is tested with Version 19.0.
    • Virtualenv This tutorial is tested with Version 16.2.0.
    • gcsfuse This tutorial is tested with Version 0.23.0.
    • The latest Cloud SDK This tutorial requires that you install the Cloud SDK in order to use the gcloud and gsutil command-line tools. You run the gcloud and gsutil tools in a terminal on your local workstation.

Overview of OpenCue

OpenCue's interface uses various modules and commands and several unique concepts and terms. The following tools and techniques are used for deploying a basic hybrid render farm. For information on more advanced topics and different deployment options, see the OpenCue documentation.

  • Cuebot - The scheduling tool of OpenCue. Cuebot handles job distribution, manages rendering resources, and communicates with the database. Cuebot can be accessed using Python and Bash through a command-line interface (CLI), but a visual interface is also included as a separate module and is described later. In this example, you run Cuebot on a GCP VM. You can also run it locally for on-premises deployments. Or for larger/hybrid deployments, you can use multiple Cuebot instances for high availability.
  • Database - Cuebot interacts with a PostgreSQL database server to store the state of the application. Even in a deployment with multiple Cuebots, only one database is shared across the instances. For this tutorial, we use a Cloud SQL instance for ease of deployment.
  • CueGUI - The visual interface for interacting with Cuebot. CueGUI is used to monitor and manage settings for jobs and render hosts. In this tutorial, you run Cuebot in the cloud and CueGUI on your local machine.
  • Render hosts - Individual render resources used for processing individual tasks. These hosts can be on-premises, in the cloud, or a combination of both. In this tutorial, you use a managed instance group of VMs in GCP as render hosts.
  • RQD - Render Queue Daemon that runs on individual render hosts and interacts with Cuebot to send and receive instructions about commands that need to be run.
  • CueSubmit - GUI for configuring and starting jobs. You can run this Python program as a standalone GUI or as a plug-in in third-party software. OpenCue includes a Maya plug-in (tested with version 2017 and later) and Nuke (tested with version 11). The OpenCue repository includes code that you can modify to build your own plug-ins for other third-party tools or to customize your pipeline.
  • PyCue - A collection of Python libraries used for interacting with Cuebot. You can use these libraries to build more tools, or to communicate with Cuebot on the command line rather than through a GUI.
  • PyOutline - Python library for building OpenCue job submissions. PyOutline allows for building a job specification and defining dependencies. In this tutorial, you don't interact directly with this module, because it gets called behind the scenes by CueSubmit.
  • CueAdmin - A command line tool that uses the PyCue python API to administer OpenCue.

OpenCue program architecture

The following architecture diagram illustrates how the OpenCue modules connect and communicate.

OpenCue architecture diagram

GCP resources used in this tutorial

The following architectural diagram illustrates the GCP resources required to complete this tutorial.

GCP resources required to complete this tutorial

Downloading the OpenCue repository

The official repository for OpenCue is hosted on GitHub.

  • Navigate to the workspace and directory you will use for this tutorial and download the OpenCue repository:

    cd ~/
    git clone https://github.com/imageworks/OpenCue.git
    

Creating a Cloud Storage bucket and populating it with tutorial files

This tutorial uses a Cloud Storage bucket to store all animation scene files and other input file dependencies, as well as final rendered output and render-host log files.

In a later step, you install and use Cloud Storage FUSE (gcsfuse). This tool is an open source FUSE adapter that you can use to mount Cloud Storage buckets as file systems on your Linux render hosts, and on your local machine. gcsfuse provides an inexpensive and effective storage option for this workflow. However, you can drastically improve the performance of some jobs by matching individual jobs to a storage profile that is optimized to their specific needs. GCP has many storage options, but covering the pros and cons for each workflow is beyond the scope of this document. For more information on storage options, see Building a hybrid render farm, or speak with a GCP representative to tailor a storage option to your needs.

Create a Cloud Storage bucket

Before you can create a bucket, ensure that you are in the correct project:

  • Create a Cloud Storage bucket [YOUR_BUCKET_NAME] for storing the tutorial data.

    gcloud config set project [YOUR_PROJECT_ID]
        gsutil mb -l us-central1 gs://[YOUR_BUCKET_NAME]
    

    The gsutil command references creating resources in the us-central1 region.

Synchronize tutorial data to your bucket

The Cloud Storage bucket contains about 950 MB of data for this tutorial. Use gsutil rsync to synchronize the data from the source to a destination bucket without having to download this data to your local machine. Note the -r flag, which copies recursive subdirectories, and the -m flag, which synchronizes in parallel and is useful when dealing with multiple files.

  1. Make an environment variable for your Cloud Storage bucket name:

    export YOUR_BUCKET=[YOUR_BUCKET_NAME]
    
  2. Synchronize the example data to your bucket:

    gsutil -m rsync -r gs://cloud-solutions-group-opencue gs://$YOUR_BUCKET
    

Upload the example database schema and data

Upload the database schema and example demo data to populate your database table:

  1. Copy the V1__Initial_schema.sql file from the local OpenCue directory to the Cloud Storage tutorial bucket:

    gsutil cp ~/OpenCue/cuebot/src/main/resources/conf/ddl/postgres/migrations/V1__Initial_schema.sql gs://$YOUR_BUCKET/database/
    
  2. Copy the demo_data.sql file from the local OpenCue directory to the Cloud Storage tutorial bucket:

    gsutil cp ~/OpenCue/cuebot/src/main/resources/conf/ddl/postgres/demo_data.sql gs://$YOUR_BUCKET/database/
    

Deploying the database

OpenCue's Cuebot module stores state data for the program in a PostgreSQL database. For this tutorial, you first reserve an external IP for the VM that acts as Cuebot. Then you provision and configure a Cloud SQL instance, set up network connectivity by whitelisting the Cuebot IP address, and finally populate the database with a schema and demo data.

Reserve a static external IP address for Cuebot

Before you create the database or VM for Cuebot, reserve a static external IP for the Cuebot VM to be used for network communication between these two resources:

  1. Go to the Reserve a static address page in the GCP Console.

    GO TO THE RESERVE A STATIC ADDRESS PAGE

  2. Choose a name for the new address, for example, cuebotstaticip.

  3. For Network Service Tier, specify Premium.

  4. For IP version, specify IPv4.

  5. For Type, choose Regional.

  6. Select the region us-central1.

  7. Leave Attached to set to None.

  8. Click Reserve to reserve the IP.

After the resource is created, you are redirected to the External IP addresses page. You reference this address as [RESERVED_STATIC_IP_ADDRESS].

Create a Cloud SQL instance

There are many options for creating a PostgreSQL database. This tutorial uses Cloud SQL, which offers ease of setup and a fully managed database on GCP. This instance uses SSD for fast reads and writes but has small initial storage capacity, only 1 vCPU, and 3.75 GB of RAM. When scaling OpenCue for larger deployments, this resource is one of the first you want to grow, and Cloud SQL can scale along with your deployment.

  1. Go to the Cloud SQL Instances page in the GCP Console.

    GO TO THE CLOUD SQL INSTANCES PAGE

  2. Select your project and click Continue.

  3. Click Create Instance.

  4. Click PostgreSQL.

  5. Enter an Instance ID, for example, opencuedatabase.

  6. Enter a password for the default postgres user. When creating Cuebot, you reference this password as [DATABASEPASSWORD].

  7. Set region to us-central1 and zone to us-central1-a.

  8. Configure connectivity options:

    1. Open Show Configuration options and select Set connectivity.
    2. Select Public IP and +Add Network.
    3. In Network, enter your [RESERVED_STATIC_IP_ADDRESS].
    4. Click Create.

    You are returned to the instances list; your new instance is dimmed while it initializes and starts.

Create a Cloud SQL database

  1. In the Instance details page of your Cloud SQL resource, select Databases.
  2. Select Create database.
  3. Set Database name to cuebot.
  4. Click Create.
  5. Import the database schema and demo data:

    1. Select Import.
    2. In the Cloud Storage file line, click Browse.
    3. Navigate to the [YOUR_BUCKET_NAME] storage bucket that you uploaded, and import the schema for your database by selecting the following file:

      gs://[YOUR_BUCKET_NAME]/database/V1__Initial_schema.sql
      
    4. Set Format of import to SQL.

    5. For Database, select cuebot.

    6. Select Show advanced options.

    7. In User, select the user postgres.

    8. Click Import.

    When the import is complete, you are redirected to the Cloud SQL Instance details page.

    Repeat the preceding steps to import the demo data from the following file:

    gs://[YOUR_BUCKET_NAME]/database/demo_data.sql
    

When both imports are complete, you are redirected to the Cloud SQL Instance details page. Take note of the Cloud SQL IP address, referred to later as [CLOUD_SQL_IP_ADDRESS], because you need it when configuring Cuebot in a subsequent step.

Preparing a Cuebot Docker image for deployment

In this tutorial, Cuebot is deployed as a Docker Container on a single Compute Engine VM. You start by building a Cuebot Docker image locally from the OpenCue repository and pushing it to Container Registry. Container Registry is a private Docker image repository on GCP that you use to store the Docker images in this tutorial. For more information on this workflow, see the Compute Engine documentation.

Create the image

  1. Ensure that you are at the root level of the OpenCue repository:

    cd ~/OpenCue
    
  2. Create a variable called TAG_NAME and another for PROJECT_NAME, and then use Docker to build an image with this tag for tracking. By including gcr.io/[PROJECT_NAME] in the build step, you enable tagging for later synchronization to Container Registry.

    export TAG_NAME=demo
    export PROJECT_NAME=[PROJECT_NAME]
    docker build -t gcr.io/$PROJECT_NAME/opencue/cuebot:$TAG_NAME -f cuebot/Dockerfile .
    

Sync the image to Container Registry

Authorize the gcloud tool to allow Docker to upload images to Container Registry. Then synchronize your Cuebot Docker image to Container Registry.

  1. Run the following command to authorize the gcloud tool and Docker synchronization:

    gcloud auth configure-docker
    
  2. At the following prompt, type y and then press Enter.

    Do you want to continue (Y/n)?
  3. Push the Cuebot Docker image from your local repository to Container Registry:

    docker push gcr.io/$PROJECT_NAME/opencue/cuebot:$TAG_NAME
    

This upload takes a few moments as the synchronization completes.

Configuring and provisioning the Cuebot VM

Use the console to provision a VM using a Container Optimized OS, configure the container settings, create a connection to the database, and attach the reserved external static IP address.

  1. Go to the VM instances page.

    GO TO THE VM INSTANCES PAGE

  2. Click the Create instance button to create a new instance.

  3. Name your instance cuebot.

  4. Set region to us-central1 and zone to us-central1-a.

  5. In Machine type, select Customize and configure a machine type of 1 vCPU 6.5 GB.

  6. Configure Container settings:

    1. In the Container section, select Deploy container image.
    2. Update [PROJECT_NAME] and specify the following Container image.

      gcr.io/[PROJECT_NAME]/opencue/cuebot:demo

    3. Expand Advanced container options.

    4. Under Command arguments, add three arguments by clicking + Add argument three times.

    5. Enter the following command arguments, replacing [CLOUD_SQL_IP_ADDRESS], and [DATABASEPASSWORD] with your values.

      --datasource.cueDataSource.url=jdbc:postgresql://[CLOUD_SQL_IP_ADDRESS]/cuebot
      --datasource.cueDataSource.username=postgres
      --datasource.cueDataSource.password=[DATABASEPASSWORD]
      

      Where:

      Command argument Explanation and input required
      --datasource.cueDataSource.url=jdbc:postgresql://[CLOUD_SQL_IP_ADDRESS]/cuebot Enables communication from Cuebot to the database. Input is [CLOUD_SQL_IP_ADDRESS]/[DATABASE_NAME]. You used cuebot for the database name in a previous step, so it is entered here.
      --datasource.cueDataSource.username=postgres Cuebot requires the Database username when connecting. During setup, you used the default username, postgres, so it is entered here. For production, you should follow security best practices and create a new user with specific privileges.
      --datasource.cueDataSource.password=[DATABASEPASSWORD] Cuebot requires the Database password when connecting. Enter the [DATABASEPASSWORD] here.
  7. Configure network settings:

    1. Under Networking, select Network interfaces.
    2. Click External IP and select the reserved external IP address you previously created for Cuebot, cuebotstaticip.
    3. Click Done.
    4. Leave the rest of the settings at their default values.
    5. Click Create.

It takes a few moments for your VM to start. Make note of the internal IP address, which you need when creating the instance template. The address is referred to later as [CUEBOT_INTERNAL_IP_ADDRESS].

Creating render hosts

Your render hosts are made up of VMs running as a managed instance group (MIG), which enables controls for both location and scaling of your resources. The MIG uses an Instance Template to determine machine type, enable Service Account management, deploy a Container image and configure Container settings and environment variables.

The Container images are used to configure many of the dependencies and software packages required to run the tutorial render tasks. To configure the Container images for your project, use Docker to create a base render-host image and store it in a local repository. You can then customize this base by adding task-specific software and GCP settings from the tutorial repository before deploying it to Container Registry.

Create the base render-host image

  • In your local shell, create the base render-host image.

    cd ~/OpenCue
    export TAG_NAME=demo
    docker build -t opencue/rqd:$TAG_NAME -f rqd/Dockerfile .
    

Configure the Blender image

The Dockerfile and related scripts needed for creating the render-host Blender image are in the tutorial repository. In this section, you download the repository and update a few settings to make it work for your project.

  1. Navigate to the directory and workspace that you are using for this tutorial and download the tutorial repository.

    cd ~
    git clone https://github.com/GoogleCloudPlatform/python-opencue-demo.git
    
  2. When the render host is started, the rqd_startup.sh script in this repository calls gcsfuse to mount your tutorial bucket as a file system at the path /shots. Open the Dockerfile in your text editor and update the GCS_FUSE_BUCKET variable to point to your tutorial bucket:

    1. In a shell, copy the following file to the root of the repository.

      cp ~/python-opencue-demo/third_party/Dockerfile ~/python-opencue-demo/Dockerfile
      
    2. In a text editor of your choice, edit the file:

      ~/python-opencue-demo/Dockerfile
      
    3. Update [YOUR_BUCKET_NAME] with the name of your bucket in the following line:

      ENV GCS_FUSE_BUCKET [YOUR_BUCKET_NAME]
      
    4. Save, and then quit the text editor.

Build the image and push it to Container Registry

  1. Build the image:

    cd ~/python-opencue-demo
    docker build -t gcr.io/$PROJECT_NAME/opencue/rqd-blender:$TAG_NAME .
    
  2. When the build is complete, push the image to Container Registry:

    docker push gcr.io/$PROJECT_NAME/opencue/rqd-blender:$TAG_NAME
    

Deploying render hosts

Your render hosts must have read and write permissions when rendering. To grant them access, create a service account using the security principle of least privilege. Use this account for Identity and API access when provisioning your VMs.

Create a service account

  1. In the GCP Console, go to the Service Accounts page.

    GO TO THE SERVICE ACCOUNTS PAGE

  2. Click Select a project.

  3. Select your project and click Open.

  4. Click + Create Service Account .

  5. Enter a [SERVICE_ACCOUNT_NAME] for example, storageservice, and click Create.

  6. Select the role of Storage Object Admin to the service account, and click Continue.

  7. Click Done.

Create an instance template

Using the rqd-blender Docker image, create an instance template to determine VM sizing and settings:

  1. In the GCP Console, go to the Instance Templates page.

    GO TO THE INSTANCE TEMPLATES PAGE

  2. Click Create instance template.

  3. Give your instance template a name, for example, rqd-blender.

  4. In Machine type, select n1-highmem-2.

  5. Configure Container settings:

  6. In the Container section, select the Deploy a container image to this VM instance checkbox.

  7. Update [PROJECT_NAME] and specify the following for Container image:

    gcr.io/[PROJECT_NAME]/opencue/rqd-blender:demo
    
  8. Allow access to system devices to register your RQD-blender instances with Cuebot at start time:

  9. Click Advanced container options.

  10. Select Run as Privileged.

  11. Click + Add Variable under Environment variable and set NAME to CUEBOT_HOSTNAME and VALUE to [CUEBOT_INTERNAL_IP_ADDRESS], which refers to the Internal IP address of the Cuebot VM you provisioned earlier.

  12. Configure the service account:

    1. In the Service account section, select the [SERVICE_ACCOUNT_NAME] you created earlier.
    2. Leave the remaining settings at their defaults.
    3. Click Create.

Create a managed instance group

Using the instance template, create a Managed Instance group to determine VM location, number of instances, and autoscaling settings.

  1. In the GCP Console, go to the Instance Groups page.

    GO TO THE INSTANCE GROUPS PAGE

  2. Click Create Instance Group.

  3. Enter a name for the managed instance group, for example, rqd-blender-mig.

  4. Select Single zone and the region us-central1 and zone us-central1-a.

  5. Under Group type, select Managed instance group.

  6. Under Instance template, select your instance template, for example, rqd-blender.

  7. Set Autoscaling to Off.

  8. For Number of instances, specify 3.

  9. Click Create.

Configuring your local workstation

Cuebot is now running on GCP and ready to manage your render hosts and job queue. To enable the visual interface for submitting and managing tasks, configure your local workstation to run CueSubmit and CueGUI, and then connect from your local host to GCP.

Create a directory and set permissions for the bucket mount point

  • Create a mount point for your Cloud Storage bucket and set its permissions to read and write.

    sudo mkdir /shots
    sudo chmod 777 /shots
    sudo chown $USER /shots
    

    When mounted on your local workstation and render host, these paths resolve to something like /shots/robotdanceparty/sh_05/sh_05_lores.blend for your input animation scene file.

Mount your tutorial bucket

  • Instruct gcsfuse to mount your tutorial bucket on your local machine as a file system in the path /shots. The --implicit-dirs flag is used to help guide the mapping of objects to directories.

    gcsfuse --implicit-dirs $YOUR_BUCKET /shots
    

    Set up SSH forwarding

To help secure communication between your local workstation and GCP, this tutorial uses SSH forwarding. The following commands open a port to forward calls between the OpenCue interfaces and Cuebot.

  1. Open a new shell and reserve it for the SSH process.
  2. Update [YOUR_PROJECT_ID] and run the following commands:

    export CUEBOT_INSTANCE_NAME=cuebot
    export PROJECTID=[YOUR_PROJECT_ID]
    export ZONE=us-central1-a
    gcloud compute ssh $CUEBOT_INSTANCE_NAME --project $PROJECTID --zone $ZONE -- -N -L 8443:127.0.0.1:8443
    

Initialize a virtual environment

  1. Open a new shell window and reserve it for CueGUI and CueSubmit usage.
  2. To isolate your environment before installation, run virtualenv:

    cd ~/OpenCue
    virtualenv --python python2 venv
    

    Learn more about virtualenv.

  3. Source the virtual environment and install dependencies using pip.

    source venv/bin/activate
    pip install -r requirements.txt
    

    You can run cat requirements.txt to see the list of dependencies installed.

  4. Before you start CueGUI or CueSubmit, define variables required for deployment.

    export OPENCUE_ROOT=~/OpenCue
    export CUEBOT_HOSTS=localhost
    export PYTHONPATH=$OPENCUE_ROOT/pyoutline/src/:$OPENCUE_ROOT/pycue/:$OPENCUE_ROOT/cuesubmit/:$OPENCUE_ROOT/cuegui/
    
    Environment variable Explanation
    CUEBOT_HOSTS=localhost Reference to how PyCue is connecting to the Cuebot instance. It is defined as localhost when using SSH forwarding. If you are using a VPN, or Dedicated Interconnect, you enter the Cuebot IP address.
    PYTHONPATH=$OPENCUE_ROOT Defines the Python path to the OpenCue libraries.

Compile gRPC protos

OpenCue uses gRPC as its communication protocol to send messages between the server and clients. The gRPC services are defined using protocol buffers in a language neutral .proto file. To enable OpenCue's Python libraries to communicate using gRPC:

  • Compile the .proto definition files into Python readable files with the following commands:

    cd ~/OpenCue/proto
    python -m grpc_tools.protoc -I=. --python_out=$OPENCUE_ROOT/pycue/opencue/compiled_proto --grpc_python_out=$OPENCUE_ROOT/pycue/opencue/compiled_proto ./*.proto
    

Running the interfaces

In the next steps, you explore OpenCue by submitting a job using CueSubmit and explore settings for jobs and render hosts with CueGUI. Both of these tools use the same environment, so background each of these processes when running them in the virtual environment shell.

Run CueSubmit

  • In VM environment shell, run CueSubmit so you can submit a job:

    cd ~/OpenCue
    python cuesubmit/cuesubmit &
    

Submit a job

Using the CueSubmit interface, enter the following variables:

  1. Enter a Job Name, for example, yourjobname.
  2. Enter a Shot, for example, sh_05.
  3. Enter a Layer Name, for example, render.
  4. In Command to Run, enter the following.

    /usr/local/blender/blender -b -noaudio /shots/robotdanceparty/sh_05/sh_05_lores.blend -o /shots/renderimages/sh_05.lores.##### -F JPEG -f #IFRAME#
    
  5. In Frame Spec, enter 600-700.

  6. Set Job Type to Shell.

  7. Set Services to shell.

  8. Leave the rest at their defaults and click Submit.

    A window appears announcing "Submitted Job to OpenCue." It includes a job ID and job name to help you track the job as it renders.

    Command-line options for rendering:

    Command-line option Description
    /usr/local/blender/blender Path to blender executable.
    -b Flag renderer to run with no interface.
    -noaudio Flag renderer to load no audio libraries.
    /shots/robotdanceparty/sh_05/sh_05_lores.blend Path to scene file to render.
    -o /shots/renderimages/sh_05.lores.##### Define output path, name, and frame padding. ##### results in frame 650 being sh_05.00650.jpg on disc.
    -F JPEG Defines output file type as JPEG.
    -f #IFRAME# Defines the frame number at render time and is OpenCue specific. <Frame Spec> interacts with this variable, so enter it just as it is here.

    For more information on Blender command-line rendering, see the Blender documentation.

    Submitted job window

    Variable Explanation
    Job Name An arbitrary variable that you choose at the start. Most facilities follow naming conventions to enable tracking jobs consistently.
    User Name Filled in by OpenCue. Is registered as the user starting CueSubmit.
    Show OpenCue creates a default show named testing at the start. Most facilities call individual film projects or commercial shows in this context. Adding and editing shows is best done using the CueAdmin command-line tools.
    Shot One of a series of frames that play for an uninterrupted period. On a movie set, a shot is the time between "Action" and "Cut." Typically used for individual shots in a sequence or asset rotations during pre-production.
    Layer An individual command within a submission and a frame range. In this example, you are running a single layer, render, in your job.
    Command to Run The command the layer will run. Here you are running a Blender command-line render.
    Frame Spec The frames to process. You render frame 600-700 of the animation. Point to the Frame Spec box to see more options, such as frame stepping and arbitrary frame inputs.
    Job Type Used for defining the software package used for this layer in a job. The command you defined is interpreted as a shell script, so enter Shell.
    Services A list of requirements, such as minimum RAM or a user-defined resource. Your job does not require specific requirements other than shell, so select Shell.
    Submission Details The combined outcome of your settings.

Overview of CueGUI

CueGUI is the central interface for a collection of UI modules, each focused on different aspects of the render farm. Below is a list of common terms, explanations, and related modules you use to monitor your job and render hosts. For a further, visual overview of CueSubmit and CueGui, watch this walkthrough video.

Term Explanation Related CueGUI module
job An individual submission to OpenCue. Monitor Jobs
Attributes
layer An individual command within a submission and a frame range. In this example, you are running a single layer, render, in your job. Monitor Job Details
Attributes
render host An individual render resource used to process a task. Monitor Job Details
Monitor Hosts
LogView
Attributes
logs Render hosts write log files to disk that can be parsed for information, such as troubleshooting or performance metrics. Monitor Job Details
LogView
attributes A list of details about an OpenCue submission or resource. The Attribute window can load details about either a job, layer, or render host. Monitor Jobs
Monitor Job Details
Monitor Hosts
Attributes
tag A method of categorizing that can be added to a host or a layer to relate it to certain jobs. You add hosts tags for the resources available on each render host and the layer tags for the resources required to process that layer. Monitor Job Details
Monitor Hosts
Attributes
layer tag A tag which defines the requirements necessary to process a Layer. For example, a layer might require a GPU, so you would add a GPU tag to the Layer. Monitor Job Details
Monitor Hosts
Attributes
host tag A tag which defines a resource a host has available to it. For example, you can add a GPU resource to a host that has a GPU installed.
A host picks up a layer to render if these tags and a list of resource requirements are met.
Monitor Job Details
Monitor Hosts
Attributes

It is beyond the scope of this tutorial to cover the entirety of OpenCue's interface and concepts; see the documentation for more detailed information.

Run CueGUI

  • In the same virtual environment shell, open CueGUI:

    cd ~/OpenCue
    python cuegui/cuegui &
    

Open the Monitor Jobs window

  • When the CueCommander window opens, select Views/Plug-ins > Cuetopia > Monitor Jobs.

About the Monitor Jobs window

The Monitor Jobs window works at the job level, so it acts as a big picture view.

Monitor Jobs window in OpenCue

On the row containing [Your Job], look for the following pertinent information for this tutorial:

  • Job: Your job name is listed here.
  • State: This job is In Progress, because it has picked up and is rendering. When the job is complete, State changes to Finished. If the job had a problem, State changes to Failing and troubleshooting would be required. For more information on troubleshooting renders in OpenCue, see the documentation.
  • Done/Total: Provides an overview of the job's progress. In the image,100 frames have completed of a 101-frame job.
  • Running: Informs you how many frames are currently processing on render hosts. In this image, only 1 frame is processing.
  • Progress: Provides a visual overview of your job's progress.

Explore the Monitor Jobs window

  1. Monitor your job's progress in the Monitor Jobs window. Your job should be In Progress. If it is Failing, see the Troubleshooting section.
  2. Hover your mouse over the word Progress. A description window appears with useful tool information.
  3. Explore the interface by hovering your mouse over the other row headings and icons.
  4. Select [Your Job].

Open the Monitor Job Details window

  1. To open the Monitor Jobs Details window, select Views/Plug-ins > Cuetopia > Monitor Job Details.
  2. In the Monitor Jobs window, double-click [Your Job] to load its data in the Monitor Jobs Details window.

About the Monitor Job Details window

The Monitor Job Details window works at the layer level and contains detailed information about your job.

Monitor Job Details window in OpenCue

The Monitor Job Details module contains two windows:

  • Layer pane. In the window with [Your Job] in the title bar, you see the layer name.
  • Layer Info pane. In this window, you see information related to each frame of the selected layer.

Overview of the Layer pane:

  • Layer: Your job has one layer, render. If your job had multiple layers, they would be listed here and would be individually selectable by double clicking.
  • Avg: The average per frame render time of this layer.
  • Tags: Layer-level tags are listed here. Setting tags is not in the scope of this tutorial, but this is the location for accessing layer-level tags.
  • Depend: Layer dependencies are listed here. Managing dependencies is not in the scope of this tutorial, but this frame is the location for accessing layer-level dependencies.

Overview of the Layer Info pane:

  • Refresh button: Clicking this button reloads the interface with the latest information from the database.
  • Order: The order in which frames are rendered, listed here because frames can be rendered in arbitrary order. For example, you could render every 10th frame, then the rest of the frames, or in reverse order. This order is specified in CueSubmit using Frame Spec when you submit your job.
  • Frame: The frame number of the scene that will render.
  • Status: The status of the frame. In the image, these frames have Succeeded.
  • Host: The render host IP address that is assigned the frame.

Explore the Monitor Job Details window

  1. Click Refresh and examine your job's progress in the Monitor Job Details window.
  2. Hover your mouse over the word Status, a description window appears with useful tool information.
  3. Explore the interface by hovering your mouse over the other row headings and icons.

Open the LogView window

  • To open the LogView window, select Views/Plug-ins > Other > LogView.

About the LogView window

The LogView window provides a quick preview of logs for individual frames. This view can be helpful for gathering information for troubleshooting issues, right-sizing VMs for future render passes, or when optimizing renders.

Log View window in OpenCue

A log file starts with a header containing data specific to OpenCue and follows with output from the software processing the image. In this instance, you see OpenCue configuration followed by Blender log output.

Explore the LogView window

  1. In the Monitor Job Details window, select a frame labeled SUCCEEDED in the Layer Info pane to examine logs.
  2. Note the command being run.
  3. Note the logDestination.

You can learn more about the LogView window in the OpenCue documentation.

Open the Monitor Hosts window

  • To open the Monitor Hosts window, select Views/Plug-ins > CueCommander > Monitor Hosts.

About the Monitor Hosts window

The Monitor Hosts window provides information on individual Render Hosts and is an interface for adding or removing resources by using tags.

Monitor Hosts window in OpenCue

The following information is pertinent for this tutorial:

  • Auto-refresh box: Select this box to reload the Monitor Hosts window.
  • Refresh box: Click this button to reload the Monitor Hosts window.
  • IP address: The IP address allows easy tracking of VMs between OpenCue and GCP.
  • Load %: Load average on the host, a good indicator of machine usage.
  • Mem: Host memory allocation.
  • Tags: Registered resources on the host.

Explore the Monitor Hosts window

  1. Click Refresh, or select the Auto-refresh checkbox.
  2. Select a render host by clicking its IP Address.
  3. Right-click the render host to see more options, such as Delete Host or Add or Remove Tags.

    For more information on managing resources with tags, see the documentation.

  4. Double-click a host to check its attributes.

  5. Note the alloc, which is set to local.general.

  6. Note the tags, which are general and the host's IP address.

Open the Attributes window

  • Select Views/Plug-ins > Other > Attributes.

About the Attributes window

The Attributes window provides a list of details on either a render host, layer, or job. Below is an example of each.

Attributes window of OpenCue

Each window contains information pertinent to the host, layer, or job. The facility local tag in Job Attributes and the general tag in Layer Attributes are key indicators of the requirements of this submission. Frames of this job only pick up on render hosts with both of these tags assigned. You see these tags are assigned in Host Attributes by viewing the alloc and tags line.

Explore the Attributes window

  1. In the Monitor Hosts window, click a render host IP address to load its attributes.
  2. In the Attributes window, check the alloc and tags settings.
  3. In the Monitor Job Details window, click a Succeeded frame to load its attributes.
  4. In the Attributes window, explore the minMemory and tags settings.
  5. In the Monitor Jobs window, click a job.
  6. In the Attributes window, explore the facility and minCores settings.

For a more thorough explanation, see CueGUI Reference.

Checking rendered frames

There are multiple ways to check frames as they render, or to copy them locally after completing. It is beyond the scope of this tutorial to install animation preview tools but you can use the GCP Console to preview individual frames, or download them locally to view in your favorite image viewing application.

  1. In the GCP Console, view a list of objects in a bucket, or preview images in a browser:

    GO TO THE Cloud Storage PAGE

  2. Click [YOUR_BUCKET_NAME], and then click renderimages to see a list of frames.

  3. Click an image, for example, sh_05_lores.00650.jpg, for a browser window preview.

  4. Use your local shell to list the images in the output directory:

      ls /shots/renderimages
    

    You can also copy files locally, or open in the viewing tool of your choice.

  5. Use gsutil to list objects in a bucket:

    gsutil ls gs://[YOUR_BUCKET_NAME]/renderimages
    
  6. Use gsutil to copy all objects in your bucket to a local directory, [LOCAL_DIRECTORY].

    mkdir ~/[LOCAL_DIRECTORY]
    gsutil -m rsync -r gs://[YOUR_BUCKET_NAME]/renderimages ~/[LOCAL_DIRECTORY]
    

Watching the job complete

If your render is still processing, you can choose to complete it and explore OpenCue further, or stop the render by killing it in the Monitor Jobs window.

To kill your job in OpenCue:

  1. Open the Monitor Jobs window.
  2. Select [Your Job] and click the Kill Job button.
  3. A pop-up asks, "Are you sure you want to kill these jobs?"
  4. Click OK to kill the job.

(Optional) Exploring options for your render farm

To keep this tutorial within a limit of 8 total vCPUs, you have deployed a few modestly sized render hosts to your rendering task. Suppose your goal is to finish the job quickly, or to create an animation that's ready for the big screen. If so, you can deploy a larger number of resources with more CPUs by modifying your existing instance template to use an n1-highcpu-4 vCPU or higher and creating a managed instance group to control the total number of render hosts available to the job.

You have rendered the following low-resolution scene file, on n1-standard-2 VMs, in about 1 minute per frame.

  • Scene file:

    gs://[YOUR_BUCKET_NAME]/robotdanceparty/sh_05/sh_05_lores.blend
    
  • Resulting image:

    Low-resolution image of robot dance party

    Your tutorial bucket contains a higher-resolution scene file, which when rendered creates images similar to these in about 8 minutes using an n1-highcpu-4 VM.

    If you want to try out other options in OpenCue, upgrade your CueSubmit command to reference this higher-resolution file.

  • Scene file:

    gs://[YOUR_BUCKET_NAME]/robotdanceparty/sh_05/sh_05.blend
    
  • Resulting image:

    Slightly higher resolution of same robot image

(Optional) Requesting a change to quota

If you want to increase the number of render hosts or use larger VMs, you might need to increase the quota for your project.

  1. Go to the Quotas page.

    GO TO THE QUOTAS PAGE

  2. In the Quotas page, select the quotas you want to change.

  3. Click the Edit Quotas button at the top of the page.

  4. Check the box of the service you want to edit.

  5. Enter your name, email, and phone number and click Next.

  6. Enter your request to increase your quota and click Next.

  7. Submit your request.

Troubleshooting


Issue: Problems with Docker build process.

Solution: Try to build the Dockerfile with the --no-cache flag, forcing Docker to use fresh data for all steps. For more information on Docker build options, see the documentation.


Issue: Connect Failed error in shell when starting CueSubmit or CueGUI. Generally something similar to:

Encountered a server error. StatusCode.UNAVAILABLE : Connect Failed

Solution: Ensure your SSH tunnel is functioning, then restart CueSubmit or CueGUI.

  1. Run the Process Status command in a local shell to ensure the SSH tunnel is open:

    ps aux | grep ssh.
    

    A list of processes is displayed. One of them should have output similar to this, with your [USER_NAME], [PROJECT_NAME], and OS:

    [USER_NAME]     75335   0.0  0.3  4392464  54736 s002  T     3:03PM   0:00.84  /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -S /Users/[USER_NAME]/google-cloud-sdk/lib/gcloud.py compute ssh cuebot --project [PROJECT_NAME] --zone us-central1-a -- -N -L 8443:127.0.0.1:8443
    
  2. Ensure that [USER_NAME], [PROJECT_NAME], --zone, and name (cuebot) are correct for your deployment.

  3. Stop the SSH process and redo the steps in the "Set up SSH forwarding" section of this document to open a new SSH tunnel.


Issue: CueSubmit or CueGUI open but print errors in their launch shell.

Solution: Connect to your Cloud SQL instance and ensure it has been populated with schema and example data. If you don't see the expected outcome, import the files again in the section Create a Cloud SQL database (see the step, "Import the database schema and demo data").

  1. In the GCP Console, go to Cloud Shell, and run the following command to update your Cloud SQL instance. Replace [CLOUD_SQL_INSTANCE_ID] with the name of your instance.

    gcloud sql connect [CLOUD_SQL_INSTANCE_ID] --user=postgres
    
  2. When prompted, update [DATABASEPASSWORD] and press Enter:

    [DATABASEPASSWORD]
    
  3. Enter the following command and press Enter:

    \c cuebot
    
  4. Re-enter password when prompted and press Enter:

    [DATABASEPASSWORD]
    
  5. Enter the following command:

    \d+
    

    Expect output that looks like this:

    >
    List of relations
    Schema | Name     | Type  | Owner    | Size       | Description
    -------+----------+-------+----------+------------+-------------
    public | action   | table | postgres | 8192 bytes |
    public | alloc    | table | postgres | 8192 bytes |
    public | comments | table | postgres | 8192 bytes |
    public | config   | table | postgres | 8192 bytes |
    public | deed     | table | postgres | 0 bytes    |
    public | depend   | table | postgres | 0 bytes    |
    public | dept     | table | postgres | 8192 bytes |
    

Issue: Errors in CueSubmit or CueGUI shell, and either tool won't run.

Solution: Ensure that the connection between CueBot and your database is working by checking the following variables:

  1. Ensure that you entered [RESERVED_STATIC_IP_ADDRESS] properly in the "Network" step of the section Create a Cloud SQL instance.
  2. Ensure that you entered [CLOUD_SQL_IP_ADDRESS] and [DATABASEPASSWORD] correctly in the "Command arguments" step of the section Configuring and provisioning the Cuebot VM.
  3. Ensure that you correctly entered the reserved external IP address in the "Configure network settings" step of the section Configuring and provisioning the Cuebot VM.

Issue: A started job is listed in OpenCue but doesn't pick up on render hosts.

Solution: Ensure that the Job Facility and Job Layer tags match the assigned Render Host tags and that minimum resource requirements are met. For more information, see the OpenCue Render Troubleshooting and Deployment Troubleshooting documents.


Issue: Render hosts are not visible in CueGui.

Solution: Several issues might cause the hosts to not be visible, the most likely being the internal IP address for Cuebot being entered incorrectly when deploying the render-host instance template. For more information, see the Deployment Troubleshooting documentation.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...