The Vertex AI SDK for Python includes classes to help with visualization, measurements, and tracking. These classes can be grouped into three types:
- Classes that use metadata to track resources in your machine learning (ML) workflow
- Classes that are used for Vertex AI Experiments
- Classes that are used for a Vertex AI TensorBoard
The following topics provide an overview of the classes related to tracking and monitoring an ML workflow in Vertex AI SDK for Python.
Metadata classes
You can use the Vertex AI SDK for Python to create Vertex ML Metadata to help you track and analyze the metadata in your ML workflow. For more information, see Introduction to Vertex ML Metadata.
Artifact
The Artifact
class represents the metadata in an
artifact in Vertex AI. An artifact is a discrete entity or piece of data
that's produced by an ML workflow. Examples of an artifact are a
dataset, a
model, and an
input file.
For more information, see
Track executions and artifacts.
When you create an Artifact
resource, you need to
specify its schema. Each type of an artifact has a unique schema. For example,
the system.Dataset
schema represents a dataset and the system.Metrics
schema represents evaluation metrics. For more information, see
How to use system schemas.
The following sample code shows how to create an
Artifact
resource that represents a model:
model_artifact = aiplatform.Artifact.create(
schema_title="system.Model",
display_name=PREPROCESSED_DATASET_NAME,
uri=PREPROCESSED_DATASET_URI,
Execution
The Execution
class represents the metadata in an
execution in Vertex AI. An execution is a step in an ML workflow.
Examples of an execution are data processing, training, and model evaluation. An
execution can consume artifacts, such as a dataset, and produce an artifact,
such as a model.
Use
aiplatform.start_execution
to create an Execution
resource. After you create an
Execution
resource, use the same
aiplatform.start_execution
method with its resume
parameter set to True
to resume it.
The following sample code shows how to create an
Execution
resource:
with aiplatform.start_execution(schema_title='system.ContainerExecution',
display_name='trainer') as execution:
execution.assign_input_artifacts([my_artifact])
model = aiplatform.Artifact.create(uri='gs://my-uri', schema_title='system.Model')
execution.assign_output_artifacts([model])
Vertex AI Experiments classes
You can use the Vertex AI SDK for Python to create and run Vertex AI Experiments. Use Vertex AI Experiments to track logged metrics and parameters to help you analyze and optimize your ML workflow. For more information, see Introduction to Vertex AI Experiments.
To learn more about how to use the Experiment
and
ExperimentRun
classes, try one of the following
tutorials:
- Build Vertex AI Experiments lineage for custom training
- Track parameters and metrics for locally trained models
- Compare pipeline runs with Vertex AI Experiments
- Get started with Vertex AI Experiments
Experiment
The Experiment
class represents an experiment in
Vertex AI. Use an experiment to analyze its
experiment runs and
pipeline runs with different
configurations, such as multiple input artifacts and hyperparameters.
There are two ways to create an Experiment
resource:
The preferred way to create an
Experiment
is by specifying a name for your experiment as a parameter when you callaiplatform.init
:# In a real world scenario it's likely you would specify more parameters # when you call aiplatform.init. This sample shows only how to use the # parameter used to create an Experiment. # Specify a name for the experiment EXPERIMENT_NAME = "your-experiment-name" # Create the experiment aiplatform.init(experiment=EXPERIMENT_NAME)
You can also create an
Experiment
by callingaiplatform.Experiment.create
.aiplatform.Experiment.create
creates theExperiment
resource but doesn't set it to a global environment. Because of this, you can't run the experiment withaiplatform.start_run
. The following sample code shows how to useaiplatform.Experiment.create
to create an experiment and then run the experiment:# Specify a name for the experiment EXPERIMENT_NAME = "your-experiment-name" EXPERIMENT_RUN_NAME = "your-run" # Create the experiment experiment = aiplatform.Experiment.create(experiment_name=EXPERIMENT_NAME) experiment_run = aiplatform.ExperimentRun.create(EXPERIMENT_RUN_NAME, experiment=EXPERIMENT_NAME)
ExperimentRun
The ExperimentRun
class represents a run of an
experiment.
The following sample code shows how to create and start an experiment run, then
use it to get information about your experiment. To delete the experiment run,
get a reference to the ExperimentRun
instance
and call its
delete
method.
# Specify your project name, location, experiment name, and run name
PROJECT_NAME = "my-project"
LOCATION = "us-central1"
EXPERIMENT_NAME = "experiment-1"
RUN_NAME = "run-1"
# Create the experiment to run
aiplatform.init(experiment_name=EXPERIMENT_NAME,
project=PROJECT_NAME,
location=LOCATION)
# Create and run an ExperimentRun resource. Next, you can use it to get
# information about your experiment. For example, you can log parameters and
# metrics with specified key-value pairs.
with aiplatform.start_run(RUN_NAME):
aiplatform.log_params({'learning_rate': 0.1, 'dropout_rate': 0.2})
aiplatform.log_metrics({'accuracy': 0.9, 'recall': 0.8})
# Get a reference to the ExperimentRun resource, get the parameters logged to
# the run, get the summary metrics logged to the run, then delete it.
with aiplatform.start_run(RUN_NAME, resume=True) as run:
run.get_params()
run.get_metrics()
run.delete()
Vertex AI TensorBoard classes
The Vertex AI SDK for Python includes classes to work with a managed version of the open source Vertex AI TensorBoard. Vertex AI TensorBoard is a tool used to monitor measurements and visualizations during your ML workflow. For more information, see Get started with Vertex AI TensorBoard.
To learn more about using the Vertex AI SDK for Python to work with Vertex AI TensorBoard, try one of the following notebook tutorials:
- Profile model training performance using Cloud Profiler
- Vertex AI TensorBoard custom training with a custom container.
- Vertex AI TensorBoard custom training with prebuilt container
- Vertex AI TensorBoard hyperparameter tuning with the HParams Dashboard
- Profile a model's training performance using Cloud Profiler
- Profile a model's training performance using Cloud Profiler in custom training with a prebuilt container
Tensorboard
The Tensorboard
class represents a managed
resource that stores Vertex AI TensorBoard experiments. You need to create a
Tensorboard
instance before the experiments can be
visualized. You can create more than one
Tensorboard
instance in a Google Cloud project.
The following sample code shows how to create a
Tensorboard
instance:
# Specify your project name, location, and the name of your Tensorboard
PROJECT_NAME = "my-project"
LOCATION = "us-central1"
TENSORBOARD_NAME = "my-tensorboard"
aiplatform.init(project=PROJECT_NAME, location=LOCATION)
tensorboard = aiplatform.Tensorboard.create(
display_name=TENSORBOARD_NAME,
project=PROJECT_NAME,
location=LOCATION,
)
TensorboardExperiment
The TensorboardExperiment
represents a
group of TensorboardRun
objects. A
TensorboardRun
instance represents the results
of a training job run in a Tensorboard.
TensorboardRun
An instance of the TensorboardRun
class maps to
a training job run in a Tensorboard with a specified set of hyperparameters, a
model definition, a dataset, and more.
TensorboardTimeSeries
The TensorboardTimeSeries
class
represents a series produced in training runs.
What's next
- Learn about the Vertex AI SDK.