Interfaces for Vertex AI Pipelines

To learn more, run the "Learn how to build Python function-based Kubeflow pipeline components" Jupyter notebook in one of the following environments:

Open in Colab | Open in Colab Enterprise | Open in Vertex AI Workbench user-managed notebooks | View on GitHub

This page lists the interfaces that you can use to define and run ML pipelines on Vertex AI Pipelines.

Interfaces to define a pipeline

Vertex AI Pipelines supports ML pipelines defined using the Kubeflow Pipelines (KFP) SDK or the TensorFlow Extended (TFX) SDK.

Kubeflow Pipelines (KFP) SDK

Use KFP for all use cases where you don't need to use TensorFlow Extended to process huge amounts of structured or text data.

Vertex AI Pipelines supports KFP SDK v1.8 or later. However, to use the code samples in the Vertex AI Pipelines documentation, use KFP SDK v2.

When you use the KFP SDK, you can define your ML workflow by building custom components and also by reusing prebuilt components, such as the Google Cloud Pipeline Components. Google Cloud Pipeline Components let you easily use Vertex AI services like AutoML in your ML pipeline. Vertex AI Pipelines supports Google Cloud Pipeline Components SDK v2 or later. For more information about Google Cloud Pipeline Components, see Introduction to Google Cloud Pipeline Components.

To learn how to build a pipeline using the Kubeflow Pipelines, see Build a pipeline. To learn more about Kubeflow Pipelines, see the Kubeflow Pipelines documentation.

TensorFlow Extended (TFX) SDK

Use TFX if you use TensorFlow Extended in your ML workflow to process terabytes of structured or text data. Vertex AI Pipelines supports TFX SDK v0.30.0 or later.

To learn how to build ML pipelines using TFX, see the Getting started tutorials section on the TensorFlow Extended in Production tutorials.

Interfaces to run a pipeline

After you define your ML pipeline, you can create an ML pipeline run using any of the following interfaces:

• REST API

• SDK clients

• Google Cloud console

Note: Vertex AI Pipelines doesn't support the gcloud CLI interface.

For more information about the interfaces you can use to interact with Vertex AI, see Interfaces for Vertex AI.

REST API

To create a pipeline run using REST, use the Pipelines service API. This API uses the projects.locations.pipelineJobs REST resource.

Note: Breaking changes to the Pipelines service API are communicated as preview launches. You can test the changes announced in preview, see the the API documentation for projects.locations.pipelineJobs (v1beta1). For more information about the preview launch stage, see the launch stage descriptions.

SDK Clients

Vertex AI Pipelines lets you create pipeline runs using the Vertex AI SDK for Python or client libraries.

Vertex AI SDK for Python

The Vertex AI SDK for Python (aiplatform) is the recommended SDK for programmatically working with the Pipelines service API. For more information about this SDK, see the API documentation for google.cloud.aiplatform.PipelineJob.

Client libraries

Client libraries are programmatically Generated API Clients (GAPIC) SDKs. Vertex AI Pipelines supports the following client libraries:

• Python (aiplatform v1 and v1beta1)

• Java

• Node.js

• Go

For more information, see Install the Vertex AI client libraries.

Google Cloud console (GUI)

Google Cloud console is the recommended way for reviewing and monitoring your pipeline runs. You can also perform other tasks using the Google Cloud console, such as creating, deleting and cloning pipeline runs, accessing the Template Gallery, and retrieving the billing label for a pipeline run.

Go to Pipelines in Google Cloud console

What's next

• Get started by learning how to define a pipeline using the Kubeflow Pipelines SDK.

• Learn how to run a pipeline.

• Learn about best practices for implementing custom-trained ML models on Vertex AI.