Track the lineage of pipeline artifacts

When you run a pipeline using Vertex AI Pipelines, the artifacts and parameters of your pipeline run are stored using Vertex ML Metadata. Vertex ML Metadata makes it easier to analyze the lineage of your pipeline's artifacts, by saving you the difficulty of keeping track of your pipeline's metadata.

An artifact's lineage includes all the factors that contributed to its creation, as well as artifacts and metadata that are derived from this artifact. For example, a model's lineage could include the following:

  • The training, test, and evaluation data used to create the model.
  • The hyperparameters used during model training.
  • Metadata recorded from the training and evaluation process, such as the model's accuracy.
  • Artifacts that descend from this model, such as the results of batch predictions.

You can use this metadata to help answer questions like the following:

  • Why did a certain pipeline run produce an especially accurate model?
  • Which pipeline run produced the most accurate model, and what hyperparameters were used to train the model?
  • Depending on the steps in your pipeline, you might be able to use Vertex ML Metadata to answer system governance questions. For example, you could use metadata to determine which version of your model was in production at a given point in time.

If you are new to Vertex ML Metadata, read the introduction to Vertex ML Metadata.

Using the Google Cloud console to analyze the lineage of your pipeline artifacts

Use the following instructions to view the lineage graph for a pipeline artifact in the Google Cloud console.

  1. In the Google Cloud console, in the Vertex AI section, go to the Metadata page.

    Go to Metadata

    The Metadata page lists the artifacts that have been created in the default metadata store.

  2. In the Region drop-down list, select the region that your run was created in.

  3. Click the Display name of an artifact to see its lineage graph.

    A static graph showing the artifacts and executions that are a part of this lineage graph appears.

  4. Click on an artifact or execution to learn more about it.

What's next