Tracking classes

The Vertex AI SDK for Python includes classes to help with visualization, measurements, and tracking. These classes can be grouped into three types:

  • Classes that use metadata to track resources in your machine learning (ML) workflow
  • Classes that are used for Vertex AI Experiments
  • Classes that are used for a Vertex AI TensorBoard

The following topics provide an overview of the classes related to tracking and monitoring an ML workflow in Vertex AI SDK for Python.

Metadata classes

You can use the Vertex AI SDK for Python to create Vertex ML Metadata to help you track and analyze the metadata in your ML workflow. For more information, see Introduction to Vertex ML Metadata.

Artifact

The Artifact class represents the metadata in an artifact in Vertex AI. An artifact is a discrete entity or piece of data that's produced by an ML workflow. Examples of an artifact are a dataset, a model, and an input file. For more information, see Track executions and artifacts.

When you create an Artifact resource, you need to specify its schema. Each type of an artifact has a unique schema. For example, the system.Dataset schema represents a dataset and the system.Metrics schema represents evaluation metrics. For more information, see How to use system schemas.

The following sample code shows how to create an Artifact resource that represents a model:

model_artifact = aiplatform.Artifact.create(
        schema_title="system.Model",
        display_name=PREPROCESSED_DATASET_NAME,
        uri=PREPROCESSED_DATASET_URI,

Execution

The Execution class represents the metadata in an execution in Vertex AI. An execution is a step in an ML workflow. Examples of an execution are data processing, training, and model evaluation. An execution can consume artifacts, such as a dataset, and produce an artifact, such as a model.

Use aiplatform.start_execution to create an Execution resource. After you create an Execution resource, use the same aiplatform.start_execution method with its resume parameter set to True to resume it.

The following sample code shows how to create an Execution resource:

with aiplatform.start_execution(schema_title='system.ContainerExecution',
                                display_name='trainer') as execution:
    execution.assign_input_artifacts([my_artifact])
    model = aiplatform.Artifact.create(uri='gs://my-uri', schema_title='system.Model')
    execution.assign_output_artifacts([model])

Vertex AI Experiments classes

You can use the Vertex AI SDK for Python to create and run Vertex AI Experiments. Use Vertex AI Experiments to track logged metrics and parameters to help you analyze and optimize your ML workflow. For more information, see Introduction to Vertex AI Experiments.

To learn more about how to use the Experiment and ExperimentRun classes, try one of the following tutorials:

Experiment

The Experiment class represents an experiment in Vertex AI. Use an experiment to analyze its experiment runs and pipeline runs with different configurations, such as multiple input artifacts and hyperparameters.

There are two ways to create an Experiment resource:

  1. The preferred way to create an Experimentis by specifying a name for your experiment as a parameter when you call aiplatform.init:

    # In a real world scenario it's likely you would specify more parameters
# when you call aiplatform.init. This sample shows only how to use the
# parameter used to create an Experiment.

# Specify a name for the experiment
EXPERIMENT_NAME = "your-experiment-name"

# Create the experiment
aiplatform.init(experiment=EXPERIMENT_NAME)

  2. You can also create an Experiment by calling aiplatform.Experiment.create. aiplatform.Experiment.create creates the Experiment resource but doesn't set it to a global environment. Because of this, you can't run the experiment with aiplatform.start_run. The following sample code shows how to use aiplatform.Experiment.create to create an experiment and then run the experiment:

    # Specify a name for the experiment
EXPERIMENT_NAME = "your-experiment-name"
EXPERIMENT_RUN_NAME = "your-run"

# Create the experiment
experiment = aiplatform.Experiment.create(experiment_name=EXPERIMENT_NAME)
experiment_run = aiplatform.ExperimentRun.create(EXPERIMENT_RUN_NAME, experiment=EXPERIMENT_NAME)

ExperimentRun

The ExperimentRun class represents a run of an experiment.

The following sample code shows how to create and start an experiment run, then use it to get information about your experiment. To delete the experiment run, get a reference to the ExperimentRun instance and call its delete method.

# Specify your project name, location, experiment name, and run name
PROJECT_NAME = "my-project"
LOCATION = "us-central1"
EXPERIMENT_NAME = "experiment-1"
RUN_NAME = "run-1"

# Create the experiment to run
aiplatform.init(experiment_name=EXPERIMENT_NAME,
                project=PROJECT_NAME,
                location=LOCATION)

# Create and run an ExperimentRun resource. Next, you can use it to get
# information about your experiment. For example, you can log parameters and
# metrics with specified key-value pairs.
with aiplatform.start_run(RUN_NAME):
     aiplatform.log_params({'learning_rate': 0.1, 'dropout_rate': 0.2})
     aiplatform.log_metrics({'accuracy': 0.9, 'recall': 0.8})

# Get a reference to the ExperimentRun resource, get the parameters logged to 
# the run, get the summary metrics logged to the run, then delete it.
with aiplatform.start_run(RUN_NAME, resume=True) as run:
     run.get_params()
     run.get_metrics()
     run.delete()

Vertex AI TensorBoard classes

The Vertex AI SDK for Python includes classes to work with a managed version of the open source Vertex AI TensorBoard. Vertex AI TensorBoard is a tool used to monitor measurements and visualizations during your ML workflow. For more information, see Get started with Vertex AI TensorBoard.

To learn more about using the Vertex AI SDK for Python to work with Vertex AI TensorBoard, try one of the following notebook tutorials:

Tensorboard

The Tensorboard class represents a managed resource that stores Vertex AI TensorBoard experiments. You need to create a Tensorboard instance before the experiments can be visualized. You can create more than one Tensorboard instance in a Google Cloud project.

The following sample code shows how to create a Tensorboard instance:

# Specify your project name, location, and the name of your Tensorboard
PROJECT_NAME = "my-project"
LOCATION = "us-central1"
TENSORBOARD_NAME = "my-tensorboard"

aiplatform.init(project=PROJECT_NAME, location=LOCATION)

tensorboard = aiplatform.Tensorboard.create(
    display_name=TENSORBOARD_NAME,
    project=PROJECT_NAME,
    location=LOCATION,
)

TensorboardExperiment

The TensorboardExperiment represents a group of TensorboardRun objects. A TensorboardRun instance represents the results of a training job run in a Tensorboard.

TensorboardRun

An instance of the TensorboardRun class maps to a training job run in a Tensorboard with a specified set of hyperparameters, a model definition, a dataset, and more.

TensorboardTimeSeries

The TensorboardTimeSeries class represents a series produced in training runs.

What's next