This tutorial shows you how to automate 3D rendering using ZYNC, a service used to deploy 3D renders on Google Cloud Platform (GCP). This tutorial is a companion to Automating IoT Machine Learning, and some of the steps outlined don't apply to other workflows.
Engineers, architects, visual effects artists, and researchers deal with massive amounts of geometric data, typically stored in formats such as OBJ, FBX, and Alembic. While these datasets have immense potential for visualization, simulation, or machine learning (ML), creating a visual library of these assets can be a complicated and time-consuming process. Given hundreds or thousands of individual models, it's challenging to ensure the kind of consistent look shown in Figure 1.
What if you could take each asset and render it using a simple, repeatable template? What if you could automate the process so you could render new assets simply by changing a text file?
By automating the rendering process across many assets, you can maintain a stable animation and lighting environment, enforce naming conventions, and consistently populate metadata. Your data will be well organized as soon as it's generated, a requirement for automated ML training or other applications requiring a structured and labeled set of assets.
In this tutorial, you learn how to 1) procedurally build 3D scenes using file templates, and 2) render them using the ZYNC Python API. The ZYNC render farm works with popular Digital Content Creation (DCC) packages, and can also be accessed with a Python API.
A render is a generic unit of work in the visual effects production pipeline. Rendering means computing a scene file to produce an image. Running a render refers to sending the scene file to a render farm, where a number of machines render one frame each, computing that frame's data and rendering that one frame.
- Procedurally generate images from a library of geometry data.
- Submit large numbers of jobs to ZYNC for rendering on GCP.
- Build a simple YAML script to perform repetitive tasks.
This tutorial uses ZYNC Render and is subject to the per-minute billing costs of the particular software, renderer, and machine type used. Synchronized assets and renders are stored on Cloud Storage. See the ZYNC pricing page for more information.
Use the ZYNC pricing calculator
to generate a cost estimate based on your projected usage. The default
configuration used here renders with Maya/Arnold on a preemptible instance of
zync-16vcpu-32gb. New GCP users might be eligible for a
Using this tutorial's default configuration, render resolution, and software choices, the rendering of each part costs approximately $4.00. All three parts will cost approximately $12.00 to render on ZYNC, as the following cost breakdown shows:
Before you begin
This tutorial requires shell access from a computer that can run the ZYNC client application and has a graphical display. You can run this from your local workstation using Terminal on macOS, PowerShell on Windows, or Remote Desktop Protocol on a Compute Engine Windows Server instance.
|Windows note: This tutorial requires Linux or a Linux-like environment and a command-line interface. On Windows, use tools such as Cygwin or PowerShell.|
Select or create a GCP project.
Make sure that billing is enabled for your Google Cloud Platform project. Learn how to enable billing.
- Enable the ZYNC and Cloud Storage APIs.
- Install and initialize the Cloud SDK.
- Clone the git repository for this tutorial containing the project scripts:
git clone https://github.com/GoogleCloudPlatform/automating-zync-renders
The resulting directory,
automating-zync-renders, is your working directory.
- Navigate to your working directory before continuing:
- To isolate your environment prior to installation, run
virtualenv --python=python2.7 env source env/bin/activate
Learn more about
- Ensure you have all required Python packages installed in your environment:
pip install -r requirements.txt
- Register your free ZYNC account and reserve a ZYNC URL.
- Use the ID for the project created in the first step for the Google Project ID field. If you lost track of this number, you can locate the ID.
- Choose a unique ZYNC URL based on your name, company name, or some other
unique identifier. You will be assigned a URL of
- After you are registered, your URL takes a few minutes to become available.
- Download the ZYNC Python API to your working directory. Take note of
the directory path because you will use it later when you
update the configuration file. To ensure compatiblity for this tutorial, the specific commit version is
included here. Replace
[NAME]with the unique URL you chose when you registered for your ZYNC account:
git clone https://github.com/zync/zync-python cd zync-python git checkout 927ea857782a0e3245cff3470f749892e572c55c echo 'ZYNC_URL = "https://[NAME].zync.io"' > config.py
- Download, install, and launch the ZYNC client application on your local workstation.
If you navigate to the
https://[NAME].zync.io URL you reserved as part of the
ZYNC registration process, you can track ZYNC usage in the Breakdown by jobs
panel under My Account > Usage.
Learn more about configuring and setting up your ZYNC project on ZYNC's documentation page.
Breaking down the production pipeline
You don't require access to 3D software to complete this tutorial, but
if you don't have access, you won't be able to edit the assets or
environment. This tutorial's repository includes files originally created in
Autodesk Maya and can be found in the
If you want to modify the render environment or have other software requirements, you must modify the pipeline in order to accommodate differences between applications.
Figure 2 shows an overview of the render process.
Understanding the components
Each asset is rendered in an identical environment. This environment consists of
the combination of a number of templates, or rigs. Each of the template files
is found in the working directory under the
rigs directory, and is in
mayaAscii format, Maya's native human-readable file format. The following rigs
|Template or rig||Filename||Description|
||An empty Maya ASCII that performs string substitution to reference the camera rig, the light rig, and each individual asset.|
||A Maya ASCII file that contains an animated camera.|
||A Maya ASCII file that contains a seamless environment background, and three lights. This file also contains render settings configured to perform an Arnold render.|
Prepping the models
For this example, you use a library of 3D geometry assets ("parts") based on a collection of random mechanical parts commonly known as a kitbash. Worldspace placement of each model in 3D (orientation, scale, and location) must be consistent in order for the light and camera rigs to treat each part the same way. Refer to Figure 3 for a consistent render example of three assets, as well as their placement in worldspace.
To prepare your models for rendering, you must ensure that each part:
- Is in its own file.
- Conforms to a consistent naming convention.
- Is centered at the origin.
- Is scaled relative to all other parts.
All the files included in this tutorial meet these requirements.
Authenticating on GCP
Create a service account key, which enables ZYNC on your local workstation to communicate with GCP.
- In the console, go to the Credentials page under APIs & Services > Credentials.
Select Create Credentials > Service Account Key.
Enter the following values. When you fill in Service account name, Service account ID is automatically populated, although you can edit it to another value of your choice.
- Service account:
New service account
- Service account name:
To set Role, click Select a role, scroll down the list to Storage, and select Storage Admin.
Service account ID:
- Service account:
Click Create. A JSON file is downloaded to your local workstation. Move the JSON file to the root of your working directory.
Updating the configuration file
Your pipeline runs a Python script reading a YAML configuration file located in your working directory. The configuration file contains paths to all files, including the authentication file and other ZYNC and project-specific settings.
In your working directory, find the
projectData.config-templatetemplate configuration file.
Duplicate the configuration file in your working directory, and rename it
cp projectData.config-template projectData.config
Understanding the configuration file
Some default values in the example configuration file need to be populated with
your user data, local paths, and filenames. To run this tutorial, you
must modify the following setting's default values:
The following list shows the variables in the order that they appear in the
projectData.config-template configuration file:
- Cloud Project ID you're using for this tutorial.
- Absolute path to your working directory.
- Relative or absolute path to the authorization file generated in Authenticating on GCP.
- Absolute path to the ZYNC Python API you installed during Setup.
- Relative path to
- Relative path to
- Relative path to
- Relative path to the directory where assembled scene files are written. After these files are generated, they are uploaded to ZYNC for rendering.
- Relative path to the directory where the final, rendered images are written.
- Relative path to the directory containing the individual geometry files.
- List of individual geometry file names (name only, no path) to iterate over.
Running a render
When your configuration file is populated correctly, launch your first render to
ZYNC using the
render.py script. From the command line, navigate to the
project directory and launch a render:
cd automating-zync-renders ./render.py --config projectData.config
The first time you run the script, you might be prompted to authenticate your Google Account in a browser.
render.py script generates output detailing the process described in
Breaking down the production pipeline.
Notice that the default logging level is set to 4 (
INFO). You can override the
default logging level to provide even more output using the
For more information, use the
Monitoring your renders
After you run the script, verify your jobs are running by navigating to the
https://[NAME].zync.io URL you reserved as part of the ZYNC registration
process. It can take up to five minutes for ZYNC to deploy and allocate a render
instance. Instances are listed in the Machines panel, as shown in Figure 4.
After jobs have been submitted for rendering, the ZYNC client application uploads files from your local workstation to Cloud Storage. You can view upload progress on the Job Details panel, as well as in the ZYNC client application under the Log tab.
After running the
render.py script, you will see three ZYNC render jobs. Each
job is rendering frames 1–1000 for a different part, drawn from the list of
Alembic files under
ZYNC allows you to launch as many jobs as you want, and automatically queues jobs based on resource availability and job priority. In this tutorial, you submit jobs to ZYNC using the Python API. You can also submit jobs using other popular 3D applications. You can launch jobs all at once or one at a time.
You can learn more about ZYNC by reading its documentation.
Rendering your own geometry
To render your own geometry in this pipeline, you need access to a 3D application that can export to Alembic format. Follow these steps:
- Prepare your parts as described in Prepping the models.
- Export your models in Alembic format to the
- Rename all exported files in the parts directory to follow this naming
render.pyscript will accept a part number of any length. Numbers less than 1000 are padded to four digits.
- Replace the default list of parts with your own Alembic file names in
- Follow the instructions in Running a render.
Pushing final renders to Cloud Storage
When the renders are complete, the ZYNC client application automatically
downloads them to your working directory under
During job submission, a Cloud Storage bucket is created with the name
After the entire frame range has finished rendering and downloading, navigate to the
imagesdirectory and copy the image directory to the bucket using
cd images/ gsutil -m cp -r [PART-NAME] gs://[PROJECT-ID]-renders/
gsutil -m cp -r part01 gs://cad-iot-ml-renders
You can also use the
gsutilto upload all directories in the
imagesdirectory to your render bucket:
gsutil -m rsync -r . gs://[PROJECT-ID]-renders
This tutorial depends on both local and cloud-based workflows, so some steps might not work as expected. This section covers some common challenges and how to solve them.
Your working directory
Much of this tutorial is based in the working directory you created in step 5
While some of the included scripts handle both absolute and relative paths,
projectData.config rely on relative paths to locate the files and
rigs necessary for ZYNC Render deployment.
You must ensure:
- After downloading, all contents in the working directory remain in the working directory.
- You have permission to read and write to the working directory, as well as to create new directories in the working directory.
Ensure Python libraries are installed
render.py script requires a number of Python libraries in order to run.
These libraries are listed in
requirements.txt and are installed during Step 8
To ensure these libraries were installed correctly, open a Python shell and
type the following:
from google.cloud import storage import yaml
If no errors are returned, these libraries are present. If these libraries aren't present, you'll see error messages such as:
ImportError: No module named yaml ImportError: cannot import name storage
Authentication errors in render.py
If you encounter authentication errors when running the
ensure the key created in Authenticating on GCP
has a unique service account name. Reusing service account names can cause
authentication to fail.
Errors submitting ZYNC jobs
If you experience a ZYNC connection error when running the
zync.ZyncConnectionError: ZYNC is down at URL: https://[NAME].zync.io
Check the following:
- Verify that your ZYNC URL is up and running by navigating to the URL in a browser.
- Check that the value of
config.pyis exactly the same as your ZYNC URL, as explained in Step 9 of Setup.
Files not found
If you experience
'Files not found' errors during ZYNC submission, ensure all
files in your working directory are readable.
Viewing render logs
In the ZYNC Console, each subjob ("chunk") is listed in the Tasks panel:
Under the Log column, click the maya link to view the log output for the associated chunk. Render logs are the first place to look if your renders are failing.
Renders not downloading
ZYNC Render automatically attempts to download rendered images to your local workstation after completion of each chunk. If you put your computer to sleep, log out of your Google account, or quit the ZYNC client application, the render job continues and rendered frames will be stored on your ZYNC account's storage. When you reconnect, log back in, or launch the ZYNC client application, ZYNC recommences downloading frames.
In the event of a download job failure, you can force chunks to re-download by:
- Selecting the chunks to download by clicking the checkbox on the left column of the Tasks panel.
Clicking the menu under Task on the upper right, and selecting Redownload Task(s).
Monitoring ZYNC activity
All communication between your local workstation and ZYNC Render goes through the ZYNC client application. You can view ZYNC activity logs by selecting the Log tab in the ZYNC client application:
To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:
- In the GCP Console, go to the Manage resources page.
- In the project list, select the project you want to delete and click Delete delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Deleting ZYNC resources
Instances deployed by ZYNC are automatically deleted when tasks are complete, killed, or encounter an error. No action is required to delete these resources.
Deleting stored data
Synchronized assets and finished renders are stored on Cloud Storage and billed on a monthly basis. You are charged for storage until you delete your data. If you no longer need to keep a project's data on ZYNC, you can delete it from the Projects tab of the My Account page in your ZYNC web console.
Exiting your virtualenv
If you used
virtualenv, exit this temporary environment by entering
deactivate in your shell.
- See how this tutorial's data is used in Automation of IoT Machine Learning.
- Read the ZYNC documentation as well as the FAQ.
- Learn more about Maya, or try it for free for 30 days.
- Learn more about the Arnold renderer.
- Learn more about Andrew Averkin's Hardsurface Kitbash Pack.
- Try out other Google Cloud Platform features for yourself. Have a look at our tutorials.