This tutorial shows how to deploy the OpenCue render management system on Linux virtual machines (VMs) using Google Cloud. Allow 3–4 hours to complete the entire tutorial.
Render management tools are used in visual effects and animation production for breaking down complex jobs into individual tasks and spreading these processes across a broad range of compute resources. These tools are used to manage jobs, such as 3D renders, 2D compositing, transcoding of data from one format to another, and checking assets in and out of databases. The tools are often used to link job stages and dependencies. At larger facilities, a render management system can support thousands of simultaneous jobs and provides the interface for allocating resources to each stage of a job, managing logs, troubleshooting tools, and managing resources.
Google has collaborated with Sony Pictures Imageworks to release OpenCue, a production-proven, open source render management system that works on Linux and macOS. OpenCue has been used to build on-premises render farms and for hybrid deployments at a very large scale utilizing resources from multiple cloud providers.
This tutorial is intended for IT administrators and pipeline technical directors who are familiar with the following tools and concepts. Don't worry if you aren't an expert at everything in this list; we indicate further resources where applicable.
- Operating principles of distributed render management.
- Git and GitHub.
- Docker for creating and running containers.
- Bash scripting for editing scripts and running resources.
- Python configuration and script editing.
- Google Cloud resource management basic tools and concepts, including Cloud SQL, Compute Engine, Cloud Storage, and Container Registry.
Because render management tools cover various use cases and orchestrate a broad range of resources, there are many ways to configure a render farm. This tutorial focuses on one configuration and gets you up and running with OpenCue as quickly as possible. Creating a more optimal architecture for your render farm might require modifying steps, profiling performance for your workflow, or swapping types of Google Cloud resources. For more information on options for setting up and managing a render farm on Google Cloud, see Building a hybrid render farm. And for more information on helping secure your data, see Securing rendering workloads.
- Install and configure OpenCue modules using Docker and Python.
- Create an OpenCue database and populate it with schema and example data.
- Create Cloud Storage resources and populate them with the tutorial's animation scene files.
- Configure and create Docker images locally, store them in Container Registry, and start Compute Engine resources using containers.
- Create instance templates and managed instance groups to enable efficient scaling of compute resources.
- Configure a workstation to run OpenCue's interface on-premises.
- Render an animation sequence and explore job management using OpenCue's interface.
You can use the pricing calculator to generate a cost estimate based on your projected usage.
This tutorial covers every step of building out a functioning render farm using tight constraints on machine numbers and sizing. To fit within the Google Cloud Free Tier pricing guidelines, in this tutorial you render an animation with low resolution and sampling rates on a few VMs with two vCPUs.
The cost of the resources that make up this tutorial, if used for a total of 4 hours, totals $3.27.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.
- Enable the Container Registry and Compute Engine APIs.
- Make sure you have the following installed on your local machine:
- Docker This tutorial is tested with Docker Engine version 18.09.0.
- Python This tutorial is tested with Version 2.7.10.
- Pip This tutorial is tested with Version 19.0.
- Virtualenv This tutorial is tested with Version 16.2.0.
- gcsfuse This tutorial is tested with Version 0.23.0.
- The latest
This tutorial requires that you install the
in order to use the
gsutilcommand-line tools. You run the
gsutiltools in a terminal on your local workstation.
Overview of OpenCue
OpenCue's interface uses various modules and commands and several unique concepts and terms. The following tools and techniques are used for deploying a basic hybrid render farm. For information on more advanced topics and different deployment options, see the OpenCue documentation.
- Cuebot - The scheduling tool of OpenCue. Cuebot handles job distribution, manages rendering resources, and communicates with the database. Cuebot can be accessed using Python and Bash through a command-line interface (CLI), but a visual interface is also included as a separate module and is described later. In this example, you run Cuebot on a Google Cloud VM. You can also run it locally for on-premises deployments. Or for larger/hybrid deployments, you can use multiple Cuebot instances for high availability.
- Database - Cuebot interacts with a PostgreSQL database server to store the state of the application. Even in a deployment with multiple Cuebots, only one database is shared across the instances. For this tutorial, we use a Cloud SQL instance for ease of deployment.
- CueGUI - The visual interface for interacting with Cuebot. CueGUI is used to monitor and manage settings for jobs and render hosts. In this tutorial, you run Cuebot in the cloud and CueGUI on your local machine.
- Render hosts - Individual render resources used for processing individual tasks. These hosts can be on-premises, in the cloud, or a combination of both. In this tutorial, you use a managed instance group of VMs in Google Cloud as render hosts.
- RQD - Render Queue Daemon that runs on individual render hosts and interacts with Cuebot to send and receive instructions about commands that need to be run.
- CueSubmit - GUI for configuring and starting jobs. You can run this Python program as a standalone GUI or as a plug-in in third-party software. OpenCue includes a Maya plug-in (tested with version 2017 and later) and Nuke (tested with version 11). The OpenCue repository includes code that you can modify to build your own plug-ins for other third-party tools or to customize your pipeline.
- PyCue - A collection of Python libraries used for interacting with Cuebot. You can use these libraries to build more tools, or to communicate with Cuebot on the command line rather than through a GUI.
- PyOutline - Python library for building OpenCue job submissions. PyOutline allows for building a job specification and defining dependencies. In this tutorial, you don't interact directly with this module, because it gets called behind the scenes by CueSubmit.
- CueAdmin - A command line tool that uses the PyCue python API to administer OpenCue.
OpenCue program architecture
The following architecture diagram illustrates how the OpenCue modules connect and communicate.
Google Cloud resources used in this tutorial
The following architectural diagram illustrates the Google Cloud resources required to complete this tutorial.
Downloading the OpenCue repository
The official repository for OpenCue is hosted on GitHub.
Navigate to the workspace and directory you will use for this tutorial and download the OpenCue repository:
cd ~/ git clone --branch 0.3.6 https://github.com/AcademySoftwareFoundation/OpenCue
Creating a Cloud Storage bucket and populating it with tutorial files
This tutorial uses a Cloud Storage bucket to store all animation scene files and other input file dependencies, as well as final rendered output and render-host log files.
In a later step, you install and use
Cloud Storage FUSE
gcsfuse). This tool is an open source FUSE adapter that you can use to mount
Cloud Storage buckets as file systems on your Linux render hosts, and
on your local machine.
gcsfuse provides an inexpensive and effective storage
option for this workflow. However, you can drastically improve the performance
of some jobs by matching individual jobs to a storage profile that is optimized
to their specific needs. Google Cloud has many storage options, but
covering the pros and cons for each workflow is beyond the scope of this
document. For more information on storage options, see
Building a hybrid render farm,
or speak with a Google Cloud representative to tailor a storage option to
Create a Cloud Storage bucket
Before you can create a bucket, ensure that you are in the correct project:
Create a Cloud Storage bucket
[YOUR_BUCKET_NAME]for storing the tutorial data.
gcloud config set project [YOUR_PROJECT_ID] gsutil mb -l us-central1 gs://[YOUR_BUCKET_NAME]
gsutilcommand references creating resources in the
Synchronize tutorial data to your bucket
The Cloud Storage bucket contains about 950 MB of data for this
to synchronize the data from the source to a destination bucket without having
to download this data to your local machine. Note the
-r flag, which copies
recursive subdirectories, and the
-m flag, which synchronizes in parallel and
is useful when dealing with multiple files.
Make an environment variable for your Cloud Storage bucket name:
Synchronize the example data to your bucket:
gsutil -m rsync -r gs://cloud-solutions-group-opencue gs://$YOUR_BUCKET
Upload the example database schema and data
Upload the database schema and example demo data to populate your database table:
V3__increase_show_length.sqlfiles from the local OpenCue directory to the Cloud Storage tutorial bucket:
gsutil cp ~/OpenCue/cuebot/src/main/resources/conf/ddl/postgres/migrations/* gs://$YOUR_BUCKET/database/
demo_data.sqlfile from the local OpenCue directory to the Cloud Storage tutorial bucket:
gsutil cp ~/OpenCue/cuebot/src/main/resources/conf/ddl/postgres/demo_data.sql gs://$YOUR_BUCKET/database/
Deploying the database
OpenCue's Cuebot module stores state data for the program in a PostgreSQL database. For this tutorial, you first reserve an external IP for the VM that acts as Cuebot. Then you provision and configure a Cloud SQL instance, set up network connectivity by whitelisting the Cuebot IP address, and finally populate the database with a schema and demo data.
Reserve a static external IP address for Cuebot
Before you create the database or VM for Cuebot, reserve a static external IP for the Cuebot VM to be used for network communication between these two resources:
Go to the Reserve a static address page in the Cloud Console.
Choose a name for the new address, for example,
For Network Service Tier, specify Premium.
For IP version, specify
For Type, choose Regional.
Select the region us-central1.
Leave Attached to set to None.
Click Reserve to reserve the IP.
After the resource is created, you are redirected to the External IP
addresses page. You reference this address as
Create a Cloud SQL instance
There are many options for creating a PostgreSQL database. This tutorial uses Cloud SQL, which offers ease of setup and a fully managed database on Google Cloud. This instance uses SSD for fast reads and writes but has small initial storage capacity, only 1 vCPU, and 3.75 GB of RAM. When scaling OpenCue for larger deployments, this resource is one of the first you want to grow, and Cloud SQL can scale along with your deployment.
Go to the Cloud SQL Instances page in the Cloud Console.
Select your project and click Continue.
Click Create Instance.
Enter an Instance ID, for example,
Enter a password for the default postgres user. When creating Cuebot, you reference this password as
Set region to
us-central1and zone to
Configure connectivity options:
- Open Show Configuration options and select Connectivity.
- Select Public IP and +Add Network.
- In Network, enter your
- Click Create.
You are returned to the instances list; your new instance is dimmed while it initializes and starts.
Create a Cloud SQL database
- In the Instance details page of your Cloud SQL resource, select Databases.
- Select Create database.
- Set Database name to
- Click Create.
Import the database schema and demo data:
- Select Import.
- In the Cloud Storage file line, click Browse.
Navigate to the
[YOUR_BUCKET_NAME]storage bucket that you uploaded, and import the schema for your database by selecting the following file:
Set Format of import to SQL.
For Database, select cuebot.
Select Show advanced options.
In User, select the user postgres.
When the import is complete, you are redirected to the Cloud SQL Instance details page.
Repeat the preceding steps to import the demo data from the following files:
gs://[YOUR_BUCKET_NAME]/database/V2__Add_limit_table.sql gs://[YOUR_BUCKET_NAME]/database/V3__increase_show_length.sql gs://[YOUR_BUCKET_NAME]/database/demo_data.sql
When these imports are complete, you are redirected to the Cloud SQL
Instance details page. Take note of the Cloud SQL IP address,
referred to later as
[CLOUD_SQL_IP_ADDRESS], because you need it when
configuring Cuebot in a subsequent step.
Preparing a Cuebot Docker image for deployment
In this tutorial, Cuebot is deployed as a Docker Container on a single Compute Engine VM. You start by building a Cuebot Docker image locally from the OpenCue repository and pushing it to Container Registry. Container Registry is a private Docker image repository on Google Cloud that you use to store the Docker images in this tutorial. For more information on this workflow, see the Compute Engine documentation.
Create the image
Ensure that you are at the root level of the OpenCue repository:
Create a variable called
TAG_NAMEand another for
PROJECT_NAME, and then use Docker to build an image with this tag for tracking. By including
gcr.io/[PROJECT_NAME]in the build step, you enable tagging for later synchronization to Container Registry.
export TAG_NAME=demo export PROJECT_NAME=[PROJECT_NAME] docker build -t gcr.io/$PROJECT_NAME/opencue/cuebot:$TAG_NAME -f cuebot/Dockerfile .
Sync the image to Container Registry
gcloud tool to allow Docker to upload images to
Container Registry. Then synchronize your Cuebot Docker image to
Run the following command to authorize the
gcloudtool and Docker synchronization:
gcloud auth configure-docker
At the following prompt, type y and then press Enter.
Do you want to continue (Y/n)?
Push the Cuebot Docker image from your local repository to Container Registry:
docker push gcr.io/$PROJECT_NAME/opencue/cuebot:$TAG_NAME
This upload takes a few moments as the synchronization completes.
Configuring and provisioning the Cuebot VM
Use the console to provision a VM using a Container Optimized OS, configure the container settings, create a connection to the database, and attach the reserved external static IP address.
Go to the VM instances page.
Click the Create instance button to create a new instance.
Name your instance
Set region to
us-central1and zone to
In Machine type, select Customize and configure a machine type of 1 vCPU 6.5 GB.
Configure Container settings:
- In the Container section, select Deploy container image.
[PROJECT_NAME]and specify the following Container image.
Expand Advanced container options.
Under Command arguments, add three arguments by clicking + Add argument three times.
Enter the following command arguments, replacing
[DATABASEPASSWORD]with your values.
--datasource.cue-data-source.jdbc-url=jdbc:postgresql://[CLOUD_SQL_IP_ADDRESS]/cuebot --datasource.cue-data-source.username=postgres --datasource.cue-data-source.password=[DATABASEPASSWORD]
Command argument Explanation and input required
Enables communication from Cuebot to the database. Input is
[DATABASE_NAME]. You used cuebot for the database name in a previous step, so it is entered here.
Cuebot requires the Database username when connecting. During setup, you used the default username,
postgres, so it is entered here. For production, you should follow security best practices and create a new user with specific privileges.
Cuebot requires the Database password when connecting. Enter the
Configure network settings:
- Under Networking, select Network interfaces.
- Click External IP and select the reserved external IP
address you previously created for Cuebot,
- Click Done.
- Leave the rest of the settings at their default values.
- Click Create.
It takes a few moments for your VM to start. Make note of the internal IP
address, which you need when creating the instance template. The address is
referred to later as
Creating render hosts
Your render hosts are made up of VMs running as a managed instance group (MIG), which enables controls for both location and scaling of your resources. The MIG uses an Instance Template to determine machine type, enable Service Account management, deploy a Container image and configure Container settings and environment variables.
The Container images are used to configure many of the dependencies and software packages required to run the tutorial render tasks. To configure the Container images for your project, use Docker to create a base render-host image and store it in a local repository. You can then customize this base by adding task-specific software and Google Cloud settings from the tutorial repository before deploying it to Container Registry.
Create the base render-host image
In your local shell, create the base render-host image.
cd ~/OpenCue export TAG_NAME=demo docker build -t opencue/rqd:$TAG_NAME -f rqd/Dockerfile .
Configure the Blender image
The Dockerfile and related scripts needed for creating the render-host Blender image are in the tutorial repository. In this section, you download the repository and update a few settings to make it work for your project.
Navigate to the directory and workspace that you are using for this tutorial and download the tutorial repository.
cd ~ git clone https://github.com/GoogleCloudPlatform/python-opencue-demo.git
When the render host is started, the
rqd_startup.shscript in this repository calls
gcsfuseto mount your tutorial bucket as a file system at the path
/shots. Open the Dockerfile in your text editor and update the
GCS_FUSE_BUCKETvariable to point to your tutorial bucket:
In a shell, copy the following file to the root of the repository.
cp ~/python-opencue-demo/third_party/Dockerfile ~/python-opencue-demo/Dockerfile
In a text editor of your choice, edit the file:
[YOUR_BUCKET_NAME]with the name of your bucket in the following line:
ENV GCS_FUSE_BUCKET [YOUR_BUCKET_NAME]
Save, and then quit the text editor.
Build the image and push it to Container Registry
Build the image:
cd ~/python-opencue-demo docker build -t gcr.io/$PROJECT_NAME/opencue/rqd-blender:$TAG_NAME .
When the build is complete, push the image to Container Registry:
docker push gcr.io/$PROJECT_NAME/opencue/rqd-blender:$TAG_NAME
Deploying render hosts
Your render hosts must have read and write permissions when rendering. To grant them access, create a service account using the security principle of least privilege. Use this account for Identity and API access when provisioning your VMs.
Create a service account
In the Cloud Console, go to the Service Accounts page.
Click Select a project.
Select your project and click Open.
Click + Create Service Account .
storageservice, and click Create.
Select the role of Storage Object Admin to the service account, and click Continue.
Create an instance template
rqd-blender Docker image, create an instance template to determine
VM sizing and settings:
In the Cloud Console, go to the Instance Templates page.
Click Create instance template.
Give your instance template a name, for example,
In Machine type, select
Configure Container settings:
In the Container section, select the Deploy a container image to this VM instance checkbox.
[PROJECT_NAME]and specify the following for Container image:
Allow access to system devices to register your RQD-blender instances with Cuebot at start time:
- Click Advanced container options.
- Select Run as Privileged.
- Click + Add Variable under Environment variable and set NAME
CUEBOT_HOSTNAMEand VALUE to
[CUEBOT_INTERNAL_IP_ADDRESS], which refers to the Internal IP address of the Cuebot VM you provisioned earlier.
Configure the service account:
- In the Service account section, select the
[SERVICE_ACCOUNT_NAME]you created earlier.
- Leave the remaining settings at their defaults.
- Click Create.
- In the Service account section, select the
Create a managed instance group
Using the instance template, create a Managed Instance group to determine VM location, number of instances, and autoscaling settings.
In the Cloud Console, go to the Instance Groups page.
Click Create Instance Group.
Enter a name for the managed instance group, for example,
Select Single zone and the region
Under Group type, select Managed instance group.
Under Instance template, select your instance template, for example,
Set Autoscaling to Off.
For Number of instances, specify 3.
Configuring your local workstation
Cuebot is now running on Google Cloud and ready to manage your render hosts and job queue. To enable the visual interface for submitting and managing tasks, configure your local workstation to run CueSubmit and CueGUI, and then connect from your local host to Google Cloud.
Create a directory and set permissions for the bucket mount point
Create a mount point for your Cloud Storage bucket and set its permissions to read and write.
sudo mkdir /shots sudo chmod 777 /shots sudo chown $USER /shots
When mounted on your local workstation and render host, these paths resolve to something like
/shots/robotdanceparty/sh_05/sh_05_lores.blendfor your input animation scene file.
Mount your tutorial bucket
gcsfuseto mount your tutorial bucket on your local machine as a file system in the path
--implicit-dirsflag is used to help guide the mapping of objects to directories.
gcsfuse --implicit-dirs $YOUR_BUCKET /shots
Set up SSH forwarding
To help secure communication between your local workstation and Google Cloud, this tutorial uses SSH forwarding. The following commands open a port to forward calls between the OpenCue interfaces and Cuebot.
- Open a new shell and reserve it for the SSH process.
[YOUR_PROJECT_ID]and run the following commands:
export CUEBOT_INSTANCE_NAME=cuebot export PROJECTID=[YOUR_PROJECT_ID] export ZONE=us-central1-a gcloud compute ssh $CUEBOT_INSTANCE_NAME --project $PROJECTID --zone $ZONE -- -N -L 8443:127.0.0.1:8443
Initialize a virtual environment
- Open a new shell window and reserve it for CueGUI and CueSubmit usage.
To isolate your environment before installation, run
cd ~/OpenCue virtualenv --python python2 venv
Learn more about
Source the virtual environment and install dependencies using
source venv/bin/activate pip install -r requirements.txt
You can run
cat requirements.txtto see the list of dependencies installed.
Before you start CueGUI or CueSubmit, define variables required for deployment.
export OPENCUE_ROOT=~/OpenCue export CUEBOT_HOSTS=localhost export PYTHONPATH=$OPENCUE_ROOT/pyoutline/:$OPENCUE_ROOT/pycue/:$OPENCUE_ROOT/cuesubmit/:$OPENCUE_ROOT/cuegui/
Environment variable Explanation
Reference to how PyCue is connecting to the Cuebot instance. It is defined as localhost when using SSH forwarding. If you are using a VPN, or Dedicated Interconnect, you enter the Cuebot IP address.
Defines the Python path to the OpenCue libraries.
Compile gRPC protos
as its communication protocol to send messages between the server and clients.
The gRPC services are defined using
in a language neutral
.proto file. To enable OpenCue's Python libraries to
communicate using gRPC:
.protodefinition files into Python readable files with the following commands:
cd ~/OpenCue/proto python -m grpc_tools.protoc -I=. --python_out=$OPENCUE_ROOT/pycue/opencue/compiled_proto --grpc_python_out=$OPENCUE_ROOT/pycue/opencue/compiled_proto ./*.proto
Running the interfaces
In the next steps, you explore OpenCue by submitting a job using CueSubmit and explore settings for jobs and render hosts with CueGUI. Both of these tools use the same environment, so background each of these processes when running them in the virtual environment shell.
In VM environment shell, run CueSubmit so you can submit a job:
cd ~/OpenCue python cuesubmit/cuesubmit &
Submit a job
Using the CueSubmit interface, enter the following variables:
- Enter a Job Name, for example,
- Enter a Shot, for example,
- Enter a Layer Name, for example,
In Command to Run, enter the following.
/usr/local/blender/blender -b -noaudio /shots/robotdanceparty/sh_05/sh_05_lores.blend -o /shots/renderimages/sh_05.lores.##### -F JPEG -f #IFRAME#
In Frame Spec, enter
Set Job Type to
Set Services to
Leave the rest at their defaults and click Submit.
A window appears announcing "Submitted Job to OpenCue." It includes a job ID and job name to help you track the job as it renders.
Command-line options for rendering:
Command-line option Description
Path to blender executable.
Flag renderer to run with no interface.
Flag renderer to load no audio libraries.
Path to scene file to render.
Define output path, name, and frame padding.
#####results in frame 650 being
Defines output file type as JPEG.
Defines the frame number at render time and is OpenCue specific.
<Frame Spec>interacts with this variable, so enter it just as it is here.
For more information on Blender command-line rendering, see the Blender documentation.
Variable Explanation Job Name An arbitrary variable that you choose at the start. Most facilities follow naming conventions to enable tracking jobs consistently. User Name Filled in by OpenCue. Is registered as the user starting CueSubmit. Show OpenCue creates a default show named
testingat the start. Most facilities call individual film projects or commercial shows in this context. Adding and editing shows is best done using the CueAdmin command-line tools.
Shot One of a series of frames that play for an uninterrupted period. On a movie set, a shot is the time between "Action" and "Cut." Typically used for individual shots in a sequence or asset rotations during pre-production. Layer An individual command within a submission and a frame range. In this example, you are running a single layer, render, in your job. Command to Run The command the layer will run. Here you are running a Blender command-line render. Frame Spec The frames to process. You render frame 600-700 of the animation. Point to the Frame Spec box to see more options, such as frame stepping and arbitrary frame inputs. Job Type Used for defining the software package used for this layer in a job. The command you defined is interpreted as a shell script, so enter Shell. Services A list of requirements, such as minimum RAM or a user-defined resource. Your job does not require specific requirements other than shell, so select Shell. Submission Details The combined outcome of your settings.
Overview of CueGUI
CueGUI is the central interface for a collection of UI modules, each focused on different aspects of the render farm. Below is a list of common terms, explanations, and related modules you use to monitor your job and render hosts. For a further, visual overview of CueSubmit and CueGui, watch this walkthrough video.
|Term||Explanation||Related CueGUI module|
|job||An individual submission to OpenCue.||Monitor Jobs
|layer||An individual command within a submission and a frame range. In this example, you are running a single layer, render, in your job.||Monitor Job Details
|render host||An individual render resource used to process a task.||Monitor Job Details
|logs||Render hosts write log files to disk that can be parsed for information, such as troubleshooting or performance metrics.||Monitor Job Details
|attributes||A list of details about an OpenCue submission or resource. The Attribute window can load details about either a job, layer, or render host.||Monitor Jobs
Monitor Job Details
|tag||A method of categorizing that can be added to a host or a layer to relate it to certain jobs. You add hosts tags for the resources available on each render host and the layer tags for the resources required to process that layer.||Monitor Job Details
|layer tag||A tag which defines the requirements necessary to process a Layer. For example, a layer might require a GPU, so you would add a GPU tag to the Layer.||Monitor Job Details
|host tag||A tag which defines a resource a host has available to it. For example, you
can add a GPU resource to a host that has a GPU installed.
A host picks up a layer to render if these tags and a list of resource requirements are met.
|Monitor Job Details
It is beyond the scope of this tutorial to cover the entirety of OpenCue's interface and concepts; see the documentation for more detailed information.
In the same virtual environment shell, open CueGUI:
cd ~/OpenCue python cuegui/cuegui &
Open the Monitor Jobs window
- When the CueCommander window opens, select Views/Plug-ins > Cuetopia > Monitor Jobs.
About the Monitor Jobs window
The Monitor Jobs window works at the job level, so it acts as a big picture view.
On the row containing
[Your Job], look for the following pertinent
information for this tutorial:
- Job: Your job name is listed here.
- State: This job is In Progress, because it has picked up and is rendering. When the job is complete, State changes to Finished. If the job had a problem, State changes to Failing and troubleshooting would be required. For more information on troubleshooting renders in OpenCue, see the documentation.
- Done/Total: Provides an overview of the job's progress. In the image,100 frames have completed of a 101-frame job.
- Running: Informs you how many frames are currently processing on render hosts. In this image, only 1 frame is processing.
- Progress: Provides a visual overview of your job's progress.
Explore the Monitor Jobs window
- Monitor your job's progress in the Monitor Jobs window. Your job should be In Progress. If it is Failing, see the Troubleshooting section.
- Hover your mouse over the word Progress. A description window appears with useful tool information.
- Explore the interface by hovering your mouse over the other row headings and icons.
Open the Monitor Job Details window
- To open the Monitor Jobs Details window, select Views/Plug-ins > Cuetopia > Monitor Job Details.
- In the Monitor Jobs window, double-click
[Your Job]to load its data in the Monitor Jobs Details window.
About the Monitor Job Details window
The Monitor Job Details window works at the layer level and contains detailed information about your job.
The Monitor Job Details module contains two windows:
- Layer pane. In the window with
[Your Job]in the title bar, you see the layer name.
- Layer Info pane. In this window, you see information related to each frame of the selected layer.
Overview of the Layer pane:
- Layer: Your job has one layer, render. If your job had multiple layers, they would be listed here and would be individually selectable by double clicking.
- Avg: The average per frame render time of this layer.
- Tags: Layer-level tags are listed here. Setting tags is not in the scope of this tutorial, but this is the location for accessing layer-level tags.
- Depend: Layer dependencies are listed here. Managing dependencies is not in the scope of this tutorial, but this frame is the location for accessing layer-level dependencies.
Overview of the Layer Info pane:
- Refresh button: Clicking this button reloads the interface with the latest information from the database.
- Order: The order in which frames are rendered, listed here because frames can be rendered in arbitrary order. For example, you could render every 10th frame, then the rest of the frames, or in reverse order. This order is specified in CueSubmit using Frame Spec when you submit your job.
- Frame: The frame number of the scene that will render.
- Status: The status of the frame. In the image, these frames have Succeeded.
- Host: The render host IP address that is assigned the frame.
Explore the Monitor Job Details window
- Click Refresh and examine your job's progress in the Monitor Job Details window.
- Hover your mouse over the word Status, a description window appears with useful tool information.
- Explore the interface by hovering your mouse over the other row headings and icons.
Open the LogView window
- To open the LogView window, select Views/Plug-ins > Other > LogView.
About the LogView window
The LogView window provides a quick preview of logs for individual frames. This view can be helpful for gathering information for troubleshooting issues, right-sizing VMs for future render passes, or when optimizing renders.
A log file starts with a header containing data specific to OpenCue and follows with output from the software processing the image. In this instance, you see OpenCue configuration followed by Blender log output.
Explore the LogView window
- In the Monitor Job Details window, select a frame labeled SUCCEEDED in the Layer Info pane to examine logs.
- Note the command being run.
- Note the
You can learn more about the LogView window in the OpenCue documentation.
Open the Monitor Hosts window
- To open the Monitor Hosts window, select Views/Plug-ins > CueCommander > Monitor Hosts.
About the Monitor Hosts window
The Monitor Hosts window provides information on individual Render Hosts and is an interface for adding or removing resources by using tags.
The following information is pertinent for this tutorial:
- Auto-refresh box: Select this box to reload the Monitor Hosts window.
- Refresh box: Click this button to reload the Monitor Hosts window.
- IP address: The IP address allows easy tracking of VMs between OpenCue and Google Cloud.
- Load %: Load average on the host, a good indicator of machine usage.
- Mem: Host memory allocation.
- Tags: Registered resources on the host.
Explore the Monitor Hosts window
- Click Refresh, or select the Auto-refresh checkbox.
- Select a render host by clicking its IP Address.
Right-click the render host to see more options, such as Delete Host or Add or Remove Tags.
For more information on managing resources with tags, see the documentation.
Double-click a host to check its attributes.
Note the alloc, which is set to
Note the tags, which are
generaland the host's IP address.
Open the Attributes window
- Select Views/Plug-ins > Other > Attributes.
About the Attributes window
The Attributes window provides a list of details on either a render host, layer, or job. Below is an example of each.
Each window contains information pertinent to the host, layer, or job. The facility local tag in Job Attributes and the general tag in Layer Attributes are key indicators of the requirements of this submission. Frames of this job only pick up on render hosts with both of these tags assigned. You see these tags are assigned in Host Attributes by viewing the alloc and tags line.
Explore the Attributes window
- In the Monitor Hosts window, click a render host IP address to load its attributes.
- In the Attributes window, check the alloc and tags settings.
- In the Monitor Job Details window, click a Succeeded frame to load its attributes.
- In the Attributes window, explore the minMemory and tags settings.
- In the Monitor Jobs window, click a job.
- In the Attributes window, explore the facility and minCores settings.
For a more thorough explanation, see CueGUI Reference.
Checking rendered frames
There are multiple ways to check frames as they render, or to copy them locally after completing. It is beyond the scope of this tutorial to install animation preview tools but you can use the Cloud Console to preview individual frames, or download them locally to view in your favorite image viewing application.
In the Cloud Console, view a list of objects in a bucket, or preview images in a browser:
[YOUR_BUCKET_NAME], and then click
renderimagesto see a list of frames.
Click an image, for example,
sh_05_lores.00650.jpg, for a browser window preview.
Use your local shell to list the images in the output directory:
You can also copy files locally, or open in the viewing tool of your choice.
gsutilto list objects in a bucket:
gsutil ls gs://[YOUR_BUCKET_NAME]/renderimages
gsutilto copy all objects in your bucket to a local directory,
mkdir ~/[LOCAL_DIRECTORY] gsutil -m rsync -r gs://[YOUR_BUCKET_NAME]/renderimages ~/[LOCAL_DIRECTORY]
Watching the job complete
If your render is still processing, you can choose to complete it and explore OpenCue further, or stop the render by killing it in the Monitor Jobs window.
To kill your job in OpenCue:
- Open the Monitor Jobs window.
[Your Job]and click the Kill Job button.
- A pop-up asks, "Are you sure you want to kill these jobs?"
Click OK to kill the job.
(Optional) Exploring options for your render farm
To keep this tutorial within a limit of 8 total vCPUs, you have deployed a few
modestly sized render hosts to your rendering task. Suppose your goal is to
finish the job quickly, or to create an animation that's ready for the big
screen. If so, you can deploy a larger number of resources with more CPUs by
modifying your existing instance template to use an
n1-highcpu-4 vCPU or
higher and creating a managed instance group to control the total number of
render hosts available to the job.
You have rendered the following low-resolution scene file, on
VMs, in about 1 minute per frame.
Your tutorial bucket contains a higher-resolution scene file, which when rendered creates images similar to these in about 8 minutes using an
If you want to try out other options in OpenCue, upgrade your CueSubmit command to reference this higher-resolution file.
(Optional) Requesting a change to quota
If you want to increase the number of render hosts or use larger VMs, you might need to increase the quota for your project.
Go to the Quotas page.
In the Quotas page, select the quotas you want to change.
Click the Edit Quotas button at the top of the page.
Check the box of the service you want to edit.
Enter your name, email, and phone number and click Next.
Enter your request to increase your quota and click Next.
Submit your request.
Issue: Problems with Docker build process.
Solution: Try to build the Dockerfile with the
--no-cache flag, forcing
Docker to use fresh data for all steps. For more information on
Docker build options,
see the documentation.
Connect Failed error in shell when starting CueSubmit or CueGUI.
Generally something similar to:
Encountered a server error. StatusCode.UNAVAILABLE : Connect Failed
Solution: Ensure your SSH tunnel is functioning, then restart CueSubmit or CueGUI.
Run the Process Status command in a local shell to ensure the SSH tunnel is open:
ps aux | grep ssh.
A list of processes is displayed. One of them should have output similar to this, with your
[PROJECT_NAME], and OS:
[USER_NAME] 75335 0.0 0.3 4392464 54736 s002 T 3:03PM 0:00.84 /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -S /Users/[USER_NAME]/google-cloud-sdk/lib/gcloud.py compute ssh cuebot --project [PROJECT_NAME] --zone us-central1-a -- -N -L 8443:127.0.0.1:8443
--zone, and name (
cuebot) are correct for your deployment.
Stop the SSH process and redo the steps in the "Set up SSH forwarding" section of this document to open a new SSH tunnel.
Issue: CueSubmit or CueGUI open but print errors in their launch shell.
Solution: Connect to your Cloud SQL instance and ensure it has been populated with schema and example data. If you don't see the expected outcome, import the files again in the section Create a Cloud SQL database (see the step, "Import the database schema and demo data").
In the Cloud Console, go to Cloud Shell, and run the following command to update your Cloud SQL instance. Replace
[CLOUD_SQL_INSTANCE_ID]with the name of your instance.
gcloud sql connect [CLOUD_SQL_INSTANCE_ID] --user=postgres
When prompted, update
[DATABASEPASSWORD]and press Enter:
Enter the following command and press Enter:
Re-enter password when prompted and press Enter:
Enter the following command:
Expect output that looks like this:
> List of relations Schema | Name | Type | Owner | Size | Description -------+----------+-------+----------+------------+------------- public | action | table | postgres | 8192 bytes | public | alloc | table | postgres | 8192 bytes | public | comments | table | postgres | 8192 bytes | public | config | table | postgres | 8192 bytes | public | deed | table | postgres | 0 bytes | public | depend | table | postgres | 0 bytes | public | dept | table | postgres | 8192 bytes |
Issue: Errors in CueSubmit or CueGUI shell, and either tool won't run.
Solution: Ensure that the connection between CueBot and your database is working by checking the following variables:
- Ensure that you entered
[RESERVED_STATIC_IP_ADDRESS]properly in the "Network" step of the section Create a Cloud SQL instance.
- Ensure that you entered
[DATABASEPASSWORD]correctly in the "Command arguments" step of the section Configuring and provisioning the Cuebot VM.
- Ensure that you correctly entered the reserved external IP address in the "Configure network settings" step of the section Configuring and provisioning the Cuebot VM.
Issue: A started job is listed in OpenCue but doesn't pick up on render hosts.
Solution: Ensure that the Job Facility and Job Layer tags match the assigned Render Host tags and that minimum resource requirements are met. For more information, see the OpenCue Render Troubleshooting and Deployment Troubleshooting documents.
Issue: Render hosts are not visible in CueGui.
Solution: Several issues might cause the hosts to not be visible, the most likely being the internal IP address for Cuebot being entered incorrectly when deploying the render-host instance template. For more information, see the Deployment Troubleshooting documentation.
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
- In the Cloud Console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
- Learn more about options for Building a hybrid render farm.
- Learn more about Connecting to the cloud.
- Learn more about Securing rendering workloads.
- Learn more about using Container Registry to deploy Docker images on GCP.
- Learn more about Deploying containers on VMs and Configuring options to run your containers.
- Read the OpenCue documentation.
- To join the conversation, or for technical questions to the community, join the OpenCue users mailing list.
- Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.