Automating Image Rendering Using ZYNC

This tutorial shows you how to automate 3D rendering using ZYNC, a service used to deploy 3D renders on Google Cloud Platform (GCP). This tutorial is a companion to Automating IoT Machine Learning, and some of the steps outlined don't apply to other workflows.

Engineers, architects, visual effects artists, and researchers deal with massive amounts of geometric data, typically stored in formats such as OBJ, FBX, and Alembic. While these datasets have immense potential for visualization, simulation, or machine learning (ML), creating a visual library of these assets can be a complicated and time-consuming process. Given hundreds or thousands of individual models, it's challenging to ensure the kind of consistent look shown in Figure 1.

A collection of consistently rendered 3D assets.
Figure 1: A collection of consistently rendered 3D assets, courtesy of Andrew Averkin.

What if you could take each asset and render it using a simple, repeatable template? What if you could automate the process so you could render new assets simply by changing a text file?

By automating the rendering process across many assets, you can maintain a stable animation and lighting environment, enforce naming conventions, and consistently populate metadata. Your data will be well organized as soon as it's generated, a requirement for automated ML training or other applications requiring a structured and labeled set of assets.

In this tutorial, you learn how to 1) procedurally build 3D scenes using file templates, and 2) render them using the ZYNC Python API. The ZYNC render farm works with popular Digital Content Creation (DCC) packages, and can also be accessed with a Python API.

Rendering terminology

A render is a generic unit of work in the visual effects production pipeline. Rendering means computing a scene file to produce an image. Running a render refers to sending the scene file to a render farm, where a number of machines render one frame each, computing that frame's data and rendering that one frame.

Objectives

  • Procedurally generate images from a library of geometry data.
  • Submit large numbers of jobs to ZYNC for rendering on GCP.
  • Build a simple YAML script to perform repetitive tasks.

Costs

This tutorial uses ZYNC Render and is subject to the per-minute billing costs of the particular software, renderer, and machine type used. Synchronized assets and renders are stored on Cloud Storage. See the ZYNC pricing page for more information.

Use the ZYNC pricing calculator to generate a cost estimate based on your projected usage. The default configuration used here renders with Maya/Arnold on a preemptible instance of type zync-16vcpu-32gb. New GCP users might be eligible for a free trial.

Using this tutorial's default configuration, render resolution, and software choices, the rendering of each part costs approximately $4.00. All three parts will cost approximately $12.00 to render on ZYNC, as the following cost breakdown shows:

cost breakdown

Before you begin

This tutorial requires shell access from a computer that can run the ZYNC client application and has a graphical display. You can run this from your local workstation using Terminal on macOS, PowerShell on Windows, or Remote Desktop Protocol on a Compute Engine Windows Server instance.

Windows note: This tutorial requires Linux or a Linux-like environment and a command-line interface. On Windows, use tools such as Cygwin or PowerShell.

Setup

  1. Select or create a Google Cloud Platform project.

    Go to the Manage resources page

  2. Make sure that billing is enabled for your Google Cloud Platform project.

    Learn how to enable billing

  3. Enable the ZYNC and Cloud Storage API.

    Enable the API

  4. Install and initialize the Cloud SDK.
  5. Clone the git repository for this tutorial containing the project scripts:
    git clone https://github.com/GoogleCloudPlatform/automating-zync-renders

    The resulting directory, automating-zync-renders, is your working directory.

  6. Navigate to your working directory before continuing:
    cd automating-zync-renders
  7. To isolate your environment prior to installation, run virtualenv:
    virtualenv --python=python2.7 env
    source env/bin/activate

    Learn more about virtualenv.

  8. Ensure you have all required Python packages installed in your environment:
    pip install -r requirements.txt
  9. Register your free ZYNC account and reserve a ZYNC URL.
    1. Use the ID for the project created in the first step for the Google Project ID field. If you lost track of this number, you can locate the ID.
    2. Choose a unique ZYNC URL based on your name, company name, or some other unique identifier. You will be assigned a URL of [NAME].zync.io.
    3. After you are registered, your URL takes a few minutes to become available.
  10. Download the ZYNC Python API to your working directory. Take note of the directory path because you will use it later when you update the configuration file. To ensure compatiblity for this tutorial, the specific commit version is included here. Replace [NAME] with the unique URL you chose when you registered for your ZYNC account:
    git clone https://github.com/zync/zync-python
    cd zync-python
    git checkout 927ea857782a0e3245cff3470f749892e572c55c
    echo 'ZYNC_URL = "https://[NAME].zync.io"' > config.py
  11. Download, install, and launch the ZYNC client application on your local workstation.

If you navigate to the https://[NAME].zync.io URL you reserved as part of the ZYNC registration process, you can track ZYNC usage in the Breakdown by jobs panel under My Account > Usage.

Learn more about configuring and setting up your ZYNC project on ZYNC's documentation page.

Breaking down the production pipeline

You don't require access to 3D software to complete this tutorial, but if you don't have access, you won't be able to edit the assets or environment. This tutorial's repository includes files originally created in Autodesk Maya and can be found in the rigs directory.

If you want to modify the render environment or have other software requirements, you must modify the pipeline in order to accommodate differences between applications.

Figure 2 shows an overview of the render process.

The render production pipeline
Figure 2: The render production pipeline.

Understanding the components

Each asset is rendered in an identical environment. This environment consists of the combination of a number of templates, or rigs. Each of the template files is found in the working directory under the rigs directory, and is in mayaAscii format, Maya's native human-readable file format. The following rigs are included.

Template or rig Filename Description
Base template base_scene_template.ma An empty Maya ASCII that performs string substitution to reference the camera rig, the light rig, and each individual asset.
Camera rig cam_rig.ma A Maya ASCII file that contains an animated camera.
Light rig light_rig.ma A Maya ASCII file that contains a seamless environment background, and three lights. This file also contains render settings configured to perform an Arnold render.

Prepping the models

For this example, you use a library of 3D geometry assets ("parts") based on a collection of random mechanical parts commonly known as a kitbash. Worldspace placement of each model in 3D (orientation, scale, and location) must be consistent in order for the light and camera rigs to treat each part the same way. Refer to Figure 3 for a consistent render example of three assets, as well as their placement in worldspace.

The render production pipeline
Figure 3: Three assets, prepared for rendering. Top row: wireframe view. Bottom row: final rendered image. Each asset is centered at the origin with consistent lighting and scaling.

To prepare your models for rendering, you must ensure that each part:

  • Is in its own file.
  • Conforms to a consistent naming convention.
  • Is centered at the origin.
  • Is scaled relative to all other parts.

All the files included in this tutorial meet these requirements.

Authenticating on GCP

Create a service account key, which enables ZYNC on your local workstation to communicate with GCP.

  1. In the console, go to the Credentials page under APIs & Services > Credentials.
  2. Select Create Credentials > Service Account Key.

    create credentials

  3. Enter the following values. When you fill in Service account name, Service account ID is automatically populated, although you can edit it to another value of your choice.

    • Service account: New service account
    • Service account name: zync-api
    • Role: Storage Admin

      To set Role, click Select a role, scroll down the list to Storage, and select Storage Admin.

    • Service account ID: zync-api

    • Key type: JSON

    credential values

  4. Click Create. A JSON file is downloaded to your local workstation. Move the JSON file to the root of your working directory.

Updating the configuration file

Your pipeline runs a Python script reading a YAML configuration file located in your working directory. The configuration file contains paths to all files, including the authentication file and other ZYNC and project-specific settings.

  1. In your working directory, find the projectData.config-template template configuration file.

  2. Duplicate the configuration file in your working directory, and rename it projectData.config:

    cp projectData.config-template projectData.config
    

Understanding the configuration file

Some default values in the example configuration file need to be populated with your user data, local paths, and filenames. To run this tutorial, you must modify the following setting's default values: gcp_project, base_path, auth_file, and zync_lib_path.

The following list shows the variables in the order that they appear in the projectData.config-template configuration file:

gcp_project
Cloud Project ID you're using for this tutorial.
base_path
Absolute path to your working directory.
auth_file
Relative or absolute path to the authorization file generated in Authenticating on GCP.
zync_lib_path
Absolute path to the ZYNC Python API you installed during Setup.
scene_template
Relative path to base_scene_template.ma.
camera_rig
Relative path to cam_rig.ma.
light_rig
Relative path to light_rig.ma.
scene_dir
Relative path to the directory where assembled scene files are written. After these files are generated, they are uploaded to ZYNC for rendering.
image_dir
Relative path to the directory where the final, rendered images are written.
parts_dir
Relative path to the directory containing the individual geometry files.
parts
List of individual geometry file names (name only, no path) to iterate over.

Running a render

When your configuration file is populated correctly, launch your first render to ZYNC using the render.py script. From the command line, navigate to the project directory and launch a render:

cd automating-zync-renders
./render.py --config projectData.config

The first time you run the script, you might be prompted to authenticate your Google Account in a browser.

The render.py script generates output detailing the process described in Breaking down the production pipeline. Notice that the default logging level is set to 4 (INFO). You can override the default logging level to provide even more output using the --verbose flag. For more information, use the --help flag.

Monitoring your renders

After you run the script, verify your jobs are running by navigating to the https://[NAME].zync.io URL you reserved as part of the ZYNC registration process. It can take up to five minutes for ZYNC to deploy and allocate a render instance. Instances are listed in the Machines panel, as shown in Figure 4.

The ZYNC Console
Figure 4: The ZYNC Console.

After jobs have been submitted for rendering, the ZYNC client application uploads files from your local workstation to Cloud Storage. You can view upload progress on the Job Details panel, as well as in the ZYNC client application under the Log tab.

After running the render.py script, you will see three ZYNC render jobs. Each job is rendering frames 1–1000 for a different part, drawn from the list of Alembic files under parts in projectData.config.

ZYNC allows you to launch as many jobs as you want, and automatically queues jobs based on resource availability and job priority. In this tutorial, you submit jobs to ZYNC using the Python API. You can also submit jobs using other popular 3D applications. You can launch jobs all at once or one at a time.

You can learn more about ZYNC by reading its documentation.

Rendering your own geometry

To render your own geometry in this pipeline, you need access to a 3D application that can export to Alembic format. Follow these steps:

  1. Prepare your parts as described in Prepping the models.
  2. Export your models in Alembic format to the parts directory.
  3. Rename all exported files in the parts directory to follow this naming convention: part#.abc. The render.py script will accept a part number of any length. Numbers less than 1000 are padded to four digits.
  4. Replace the default list of parts with your own Alembic file names in projectData.config.
  5. Follow the instructions in Running a render.

Pushing final renders to Cloud Storage

When the renders are complete, the ZYNC client application automatically downloads them to your working directory under images/[PART_NAME].

During job submission, a Cloud Storage bucket is created with the name gs://[PROJECT-ID]-renders.


  1. After the entire frame range has finished rendering and downloading, navigate to the images directory and copy the image directory to the bucket using gsutil:

    cd images/
    
    gsutil -m cp -r [PART-NAME] gs://[PROJECT-ID]-renders/
    

    For example:

    gsutil -m cp -r part01 gs://cad-iot-ml-renders
    
  2. You can also use the rsync feature of gsutil to upload all directories in the images directory to your render bucket:

    gsutil -m rsync -r . gs://[PROJECT-ID]-renders
    

Troubleshooting

This tutorial depends on both local and cloud-based workflows, so some steps might not work as expected. This section covers some common challenges and how to solve them.

Your working directory

Much of this tutorial is based in the working directory you created in step 5 of Setup. While some of the included scripts handle both absolute and relative paths, files like projectData.config rely on relative paths to locate the files and rigs necessary for ZYNC Render deployment.

You must ensure:

  • After downloading, all contents in the working directory remain in the working directory.
  • You have permission to read and write to the working directory, as well as to create new directories in the working directory.

Ensure Python libraries are installed

The render.py script requires a number of Python libraries in order to run. These libraries are listed in requirements.txt and are installed during Step 8 of Setup. To ensure these libraries were installed correctly, open a Python shell and type the following:

from google.cloud import storage
import yaml

If no errors are returned, these libraries are present. If these libraries aren't present, you'll see error messages such as:

ImportError: No module named yaml
ImportError: cannot import name storage

Authentication errors in render.py

If you encounter authentication errors when running the render.py script, ensure the key created in Authenticating on GCP has a unique service account name. Reusing service account names can cause authentication to fail.

Errors submitting ZYNC jobs

If you experience a ZYNC connection error when running the render.py script, such as:

zync.ZyncConnectionError: ZYNC is down at URL: https://[NAME].zync.io

Check the following:

  • Verify that your ZYNC URL is up and running by navigating to the URL in a browser.
  • Check that the value of ZYNC_URL in config.py is exactly the same as your ZYNC URL, as explained in Step 9 of Setup.

Files not found

If you experience 'Files not found' errors during ZYNC submission, ensure all files in your working directory are readable.

Viewing render logs

In the ZYNC Console, each subjob ("chunk") is listed in the Tasks panel:

viewing render logs

Under the Log column, click the maya link to view the log output for the associated chunk. Render logs are the first place to look if your renders are failing.

Renders not downloading

ZYNC Render automatically attempts to download rendered images to your local workstation after completion of each chunk. If you put your computer to sleep, log out of your Google account, or quit the ZYNC client application, the render job continues and rendered frames will be stored on your ZYNC account's storage. When you reconnect, log back in, or launch the ZYNC client application, ZYNC recommences downloading frames.

In the event of a download job failure, you can force chunks to re-download by:

  1. Selecting the chunks to download by clicking the checkbox on the left column of the Tasks panel.
  2. Clicking the menu under Task on the upper right, and selecting Redownload Task(s).

    redownload tasks

Monitoring ZYNC activity

All communication between your local workstation and ZYNC Render goes through the ZYNC client application. You can view ZYNC activity logs by selecting the Log tab in the ZYNC client application:

ZYNC activity logs

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  1. In the GCP Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting ZYNC resources

Instances deployed by ZYNC are automatically deleted when tasks are complete, killed, or encounter an error. No action is required to delete these resources.

Deleting stored data

Synchronized assets and finished renders are stored on Cloud Storage and billed on a monthly basis. You are charged for storage until you delete your data. If you no longer need to keep a project's data on ZYNC, you can delete it from the Projects tab of the My Account page in your ZYNC web console.

Exiting your virtualenv

If you used virtualenv, exit this temporary environment by entering deactivate in your shell.

What's next

Czy ta strona była pomocna? Podziel się z nami swoją opinią:

Wyślij opinię na temat...