{ % include "_shared/apis/console/_local_variables.html" % }

Using Stackdriver as a Logging Server for a VFX Rendering Pipeline

This tutorial outlines how to use Google Stackdriver Logging instead of a logging server for application-specific logging. By default, Stackdriver Logging aggregates logs from the system and from many common applications. The tutorial also shows how to get logs into Stackdriver from custom workflows or applications that are not on the common applications list.

Short-lived workers are common in many compute workloads, such as in visual effects (VFX) render pipelines and build systems. This tutorial focuses on VFX workloads and uses the standalone VFX renderer V-Ray as an example. In a typical use case, VM instances are created on-demand by a queueing system, assigned a job (such as one or more frames to render), and terminated after the job is completed. You must capture the logs not only from the render process, but also from any pre- or post-render jobs the instance performs, such as file conversion, or copying frames rendered locally to common storage.

Objectives

  • Create an instance and install the Stackdriver agent on the instance.
  • Configure your custom application or workflow to send logs to the Stackdriver agent.
  • Use the Python client library to send logs directly to Stackdriver.
  • View, filter, and search logs in Stackdriver.
  • Export logs from Stackdriver to long-term accessible storage.

Costs

This tutorial's use of Google Stackdriver, Google BigQuery, and Google Cloud Storage falls within the free usage tiers.

Before you begin

  1. Cloud Platform Console プロジェクトを選択または作成します。

    [プロジェクト] ページに移動

  2. プロジェクトの課金を有効にします。

    課金の有効化

  3. Compute Engine API を有効にする。

    Enable the API

  4. Install the gcloud beta commands component:

    gcloud components install beta

  5. Set your default project so you don't have to supply the --project flag with each command:

    gcloud config set project PROJECT_ID

Creating a Compute Engine instance

The Stackdriver Logging Agent works on Google Compute Engine virtual machine (VM) instances and Amazon Elastic Compute Cloud (Amazon EC2) VM instances. For more information about the agent and the VM instances it supports, see Logging Agent in the product documentation.

For the purposes of this tutorial, you can create an instance with a default VM type. In production, however, you must decide how much computing power your application requires and choose a VM accordingly.

  1. Cloud Platform Console で、[VM インスタンス] ページに移動します。

    [VM インスタンス] ページに移動する

  2. [インスタンスを作成] ボタンをクリックします。
  3. [新しいインスタンスの作成] ページで、インスタンス向けに希望するプロパティを入力します。 詳細設定オプションでは、[管理、ディスク、ネットワーキング、SSH キー] を開きます。
  4. [作成] ボタンをクリックし、インスタンスを作成します。

It takes a few moments to create your new instance. In this tutorial, the VM instance is named sd-tutorial.

Configuring the Compute Engine instance

After you've created the VM instance, using an account that has superuser permissions, complete these steps:

  1. Use SSH to connect to the sd-tutorial instance.

    gcloud compute ssh sd-tutorial

  2. Install the Stackdriver Logging Agent. For detailed instructions, see Installing the Stackdriver Logging Agent.

  3. Download and install pip:

    sudo yum install python-pip

  4. Download and install the Stackdriver Logging Python Library:

    sudo pip install --upgrade google-cloud-logging

  5. Create the Stackdriver Logging Agent config file for V-Ray at /etc/google-fluentd/config.d/vray.conf with the following contents:

    <source>
  @type tail
  read_from_head true
  format /^(\[(?<time>.*?)\])?((?<severity> ?(debug|info|warning|error|critical)): )?(?<message>.*)$/
  time_format %Y/%b/%d|%H:%M:%S
    # Log file names must be of the format SH.SEQ.SHOT.ROLE.log.
    # For example: myfilm.fba.0050.render.log
  path /home/*/vray/logs/*.log
  pos_file /var/lib/google-fluentd/pos/vray.pos
  tag vray.*
</source>
<filter vray.**>
  @type record_transformer
  <record>
    # Parse the log file name and add additional key:value records
    # to aid in filtering and searching logs in Stackdriver.
    # Assumes you are following the convention in this tutorial.
    show ${tag_parts[-5]}
    seq ${tag_parts[-4]}
    shot ${tag_parts[-3]}
    role ${tag_parts[-2]}
    tag ${tag_suffix[1]} # Strip off the "vray." prefix.
  </record>
</filter>

    For more information on fluentd configuration, see Configuration File Syntax.

  6. Reload the Stackdriver configuration:

    sudo service google-fluentd reload

    In a production setting, you could next make this configured VM into a custom VM image that your pipeline could start on demand.

Special considerations for VFX

Each rendering software package generates its own log output. Although this tutorial uses the V-Ray standalone renderer, you can adapt the tutorial to other renderers or applications that output to stdout/stderr. This tutorial uses a generalized fluentd configuration and expects your queueing system to redirect the output of the renderer to a filename with a specific, search-friendly format. If you run multiple jobs on a single VM, you will need to ensure unique filenames.

Name your log files in Stackdriver

When you define the naming convention for your Stackdriver logs, follow the best practices for naming conventions in use at your studio. Stackdriver can search across resources, so using a consistent naming convention ensures that you can search for logs generated from different resources that are tagged with the same or similar data. In this tutorial, the queue manager populates the following values into environment variables before starting the render worker process, which are then used to tag the logs and generate a unique filename:

Field name Environment Variable Value
Show (Project) Name SHOW myfilm
Sequence Name SEQ fba
Shot Number SHOT 0050
Role ROLE render

This example assembles these values into a naming convention typical for visual effects workflows:

<SHOW>.<SEQ>.<SHOT>.<ROLE>.log

For example, the render logs for shot fba0050 would be tagged as follows:

myfilm.fba.0050.render.log

This tutorial expects the queue manager to set the log filename according to this convention, but modifying it to fit your studio's differing needs is straightforward.

Test the Stackdriver configuration manually

To check your configuration without using a renderer or a queueing manager, copy a sample log entry into a test log. From your home directory, enter the following commands:

mkdir -p vray/logs/
export SHOW='testshow' SEQ='testseq' SHOT='testshot' ROLE='testrole'
echo "debug: Test log line at `date` from ${HOSTNAME}" >> vray/logs/${SHOW}.${SEQ}.${SHOT}.${ROLE}.log

This line should appear shortly in the log viewer.

Verify log delivery

  1. In the Cloud Platform Console, go to the Logs Viewer page.

    Go to the Logs Viewer page

    Under the Logs menu, you should see an entry tagged with the log name you created—in this case, testshow.testseq.testshot.testrole.

  2. View this log to see your output:

    Stackdriver log viewer

    You can also read logs using the gcloud tool beta command, replacing [project-id] and [log-name] as applicable:

    # List all logs.
gcloud beta logging logs list

# Read the contents of a specific log.
gcloud beta logging read projects/[project-id]/logs/[log-name]

For more information about logging by using the gcloud command-line tool, see the documentation.

Log to Stackdriver from a render process

With the configuration set up correctly and verified, you can use a V-Ray standalone command line like the following to send logs to Stackdriver. In this example, the command-line flags start the V-Ray process with output optimized for redirection to a file. The queueing manager is expected to replace SCENE_FILE with the appropriate local file path. It should also populate the four environment variables (SHOW, SEQ, SHOT, and ROLE) that are used to generate the name of the log, as described in the Name your log files section.

vray \
   -display=0 \
   -showProgress=0 \
   -progressUseCR=0 \
   -progressUseColor=0 \
   -sceneFile SCENE_FILE > vray/logs/${SHOW}.${SEQ}.${SHOT}.${ROLE}.log 2>&1

Logging directly to the Stackdriver API

Most VFX pipelines use some sort of scripting language to perform programmatic tasks such as asset preparation, publishing, data transfer, rendering, or transcoding. You can log the output from these tasks to Stackdriver using a Client Library. This tutorial uses Python because it is so widely used in the VFX industry.

You can send logs to Stackdriver from both on-premises and cloud-based workstations. You don't install the Stackdriver logging agent to write logs this way because you communicate with Stackdriver through its Python API.

Write a log to Stackdriver using the Python library

To log to Stackdriver by using a Python script, you must first do the following:

  • Build the log metadata.
  • Provide the severity level.
  • Decide what type of resource to log to.

The script performs the following:

  • Checks to ensure the use of proper naming conventions.
  • Assembles the log data.
  • Writes the log at the Google project resource level.

If you are logging from an on-premises workstation or server, you must authenticate prior to logging to Stackdriver. If you are logging from a cloud instance, authentication is already completed.

logToStackdriver.py 
#!/usr/bin/python
#
# Copyright 2017 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from google.cloud import logging
from google.cloud.logging.resource import Resource
import getpass
import os
import socket


def write(text, severity='INFO', show=None, seq=None, shot=None, role=None, **kwargs):
    '''Wrapper method for assembling the payload to send to logger.log_text.'''

    # Extract and build LOG_ID from environment.
    # For example: 'myfilm.slb.0050.render'
    if not show:
        show = os.getenv('SHOW')
    if not seq:
        seq = os.getenv('SEQ')
    if not shot:
        shot = os.getenv('SHOT')
    if not role:
        role = os.getenv('ROLE')

    if not show or not seq or not shot or not role:
        raise Exception('One or more log name tokens are empty. Unable to log.')
    # end if

    # Assemble logger name.
    logger_name = '.'.join([
        show,
        seq,
        shot,
        role
    ])

    print '# Logging to %s...' % logger_name

    # Build logger object.
    logging_client = logging.Client()
    logger = logging_client.logger(logger_name)

    # Assemble the required log metadata.
    label_payload = {
        "artist" : getpass.getuser(),
        "hostname" : socket.gethostname(),
        "show" : show,
        "seq" : seq,
        "shot" : shot,
        "role" : role
    }

    # Add optional kwargs to payload.
    label_payload.update(kwargs)

    # Write log.
    logger.log_text(
        text,
        resource=Resource(type='project', labels={'project_id':show}),
        severity=severity,
        labels=label_payload
    )

# end write

You can import the module into any Python script in your pipeline and run it on either an on-premises artist workstation or a cloud instance:

import logToStackdriver as lts
lts.write( 'This is the text to log.', show='myfilm', seq='slb', shot='0050', role='render' )

By default, all logs write to the 'project' resource, which you can find in the Google Cloud Platform Console, under Stackdriver Logging > Logs > Google Project:

Stackdriver Logging > Logs > Google Project

Exporting logs

If you want to keep your logs beyond the Stackdriver retention period, you should export them.

For inexpensive, long-term storage, export your logs to Google Cloud Storage buckets. To perform big data analysis on them, export to a BigQuery dataset. In either case, you first create an object called a sink. The sink allows you to create a filter that selects the log entries you want to export and choose Cloud Storage or BigQuery as the destination. Creation of a sink immediately begins exporting the specified logs to the specified destination. You can export logs in the Logs Viewer, by using the Stackdriver API, or directly using the gcloud logging command-line tool.

Export to BigQuery

You can use SQL semantics to query logs stored in BigQuery. Many third-party analytics tools natively support these logs. Follow these steps:

  1. Create a BigQuery dataset.

  2. Create the sink with a filter to export logs to that table. Note that [PROJECT_ID] in the following command line refers to your GCP project.

    gcloud beta logging sinks create render-errors-bq  \
    bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET] \
    --log-filter "jsonPayload.SHOW=myfilm AND jsonPayload.SHOT=fba AND severity>=WARNING"

  3. You will receive a message with the service account name to add to your BigQuery dataset. You can use the Web UI, by clicking the dropdown next to a dataset name, and then clicking Share dataset.

The next log sent to Stackdriver that matches this filter will be sent to the dataset, with a slight delay. You can find more details in the How sinks work documentation.

Export to Cloud Storage

To save your logs to a file, export them to a Cloud Storage bucket. For Cloud Storage buckets, you can select a storage class with lower prices for less frequently accessed files, or take advantage of the free usage allowance. Cloud Storage provides easy access either by HTTP, or by direct integration with many other GCP products.

The following steps show how to export to Cloud Storage:

  1. Create a Cloud Storage bucket.

  2. In Stackdriver, create a sink with a filter to export those logs to Cloud Storage.

    gcloud beta logging sinks create render-errors-gcs  \
    storage.googleapis.com/my-gcs-bucket \
    --log-filter "jsonPayload.SHOW=myfilm AND jsonPayload.SHOT=fba AND severity>=WARNING"

The next log sent to Stackdriver that matches this filter will be sent to a file in the bucket, with a slight delay. You can find more details in the How sinks work documentation.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

  1. Cloud Platform Console で、[プロジェクト] ページに移動します。

    [プロジェクト] ページに移動する

  2. プロジェクト名の右側にある、ゴミ箱アイコンをクリックします。

Delete your Compute Engine instances

To delete a Compute Engine instance:

  1. Cloud Platform Console で、[VM インスタンス] ページに移動します。

    [VM インスタンス] ページに移動する

  2. 削除するインスタンスの隣にあるチェックボックスをオンにします。
  3. ページの上部にある、[削除] ボタンをクリックし、インスタンスを削除します。

Delete your Cloud Storage bucket

To delete a Cloud Storage bucket:

  1. Cloud Platform Console で、Cloud Storage ブラウザに移動します。

    Cloud Storage ブラウザに移動する

  2. 削除したいバケットの隣にあるチェックボックスをクリックします。
  3. ページの上部にある [削除] ボタンをクリックし、バケットを削除します。

What's next

外出先でもリソースをモニタリング

Google Cloud Console アプリを入手して、プロジェクトの管理にお役立てください。

後で アプリを使用

フィードバックを送信...

このページ
ドキュメントのフィードバック
Solutions
サービスのフィードバック