REST Resource: projects.locations.trainingPipelines

Resource: TrainingPipeline

The TrainingPipeline orchestrates tasks associated with training a Model. It always executes the training task, and optionally may also export data from Vertex AI's Dataset which becomes the training input, upload the Model to Vertex AI, and evaluate the Model.

JSON representation 
{
  "name": string,
  "displayName": string,
  "inputDataConfig": {
    object (InputDataConfig)
  },
  "trainingTaskDefinition": string,
  "trainingTaskInputs": value,
  "trainingTaskMetadata": value,
  "modelToUpload": {
    object (Model)
  },
  "state": enum (PipelineState),
  "error": {
    object (Status)
  },
  "createTime": string,
  "startTime": string,
  "endTime": string,
  "updateTime": string,
  "labels": {
    string: string,
    ...
  },
  "encryptionSpec": {
    object (EncryptionSpec)
  }
}
Fields
name

string

Output only. Resource name of the TrainingPipeline.
displayName

string

Required. The user-defined name of this TrainingPipeline.
inputDataConfig

object (InputDataConfig)

Specifies Vertex AI owned input data that may be used for training the Model. The TrainingPipeline's trainingTaskDefinition should make clear whether this config is used and if there are any special requirements on how it should be filled. If nothing about this config is mentioned in the trainingTaskDefinition, then it should be assumed that the TrainingPipeline does not depend on this configuration.
trainingTaskDefinition

string

Required. A Google Cloud Storage path to the YAML file that defines the training task which is responsible for producing the model artifact, and may also include additional auxiliary work. The definition files that can be used here are found in gs://google-cloud-aiplatform/schema/trainingjob/definition/. Note: The URI given on output will be immutable and probably different, including the URI scheme, than the one given on input. The output URI will point to a location where the user only has a read access.
trainingTaskInputs

value (Value format)

Required. The training task's parameter(s), as specified in the trainingTaskDefinition's inputs.
trainingTaskMetadata

value (Value format)

Output only. The metadata information as specified in the trainingTaskDefinition's metadata. This metadata is an auxiliary runtime and final information about the training task. While the pipeline is running this information is populated only at a best effort basis. Only present if the pipeline's trainingTaskDefinition contains metadata object.
modelToUpload

object (Model)

Describes the Model that may be uploaded (via ModelService.UploadModel) by this TrainingPipeline. The TrainingPipeline's trainingTaskDefinition should make clear whether this Model description should be populated, and if there are any special requirements regarding how it should be filled. If nothing is mentioned in the trainingTaskDefinition, then it should be assumed that this field should not be filled and the training task either uploads the Model without a need of this information, or that training task does not support uploading a Model as part of the pipeline. When the Pipeline's state becomes PIPELINE_STATE_SUCCEEDED and the trained Model had been uploaded into Vertex AI, then the modelToUpload's resource name is populated. The Model is always uploaded into the Project and Location in which this pipeline is.
state

enum (PipelineState)

Output only. The detailed state of the pipeline.
error

object (Status)

Output only. Only populated when the pipeline's state is PIPELINE_STATE_FAILED or PIPELINE_STATE_CANCELLED.
createTime

string (Timestamp format)

Output only. Time when the TrainingPipeline was created.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
startTime

string (Timestamp format)

Output only. Time when the TrainingPipeline for the first time entered the PIPELINE_STATE_RUNNING state.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
endTime

string (Timestamp format)

Output only. Time when the TrainingPipeline entered any of the following states: PIPELINE_STATE_SUCCEEDED, PIPELINE_STATE_FAILED, PIPELINE_STATE_CANCELLED.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
updateTime

string (Timestamp format)

Output only. Time when the TrainingPipeline was most recently updated.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
labels

map (key: string, value: string)

The labels with user-defined metadata to organize TrainingPipelines.

Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed.

See https://goo.gl/xmQnxf for more information and examples of labels.
encryptionSpec

object (EncryptionSpec)

Customer-managed encryption key spec for a TrainingPipeline. If set, this TrainingPipeline will be secured by this key.

Note: Model trained by this TrainingPipeline is also secured by this key if modelToUpload is not set separately.

InputDataConfig

Specifies Vertex AI owned input data to be used for training, and possibly evaluating, the Model.

JSON representation 
{
  "datasetId": string,
  "annotationsFilter": string,
  "annotationSchemaUri": string,

  // Union field split can be only one of the following:
  "fractionSplit": {
    object (FractionSplit)
  },
  "filterSplit": {
    object (FilterSplit)
  },
  "predefinedSplit": {
    object (PredefinedSplit)
  },
  "timestampSplit": {
    object (TimestampSplit)
  }
  // End of list of possible types for union field split.

  // Union field destination can be only one of the following:
  "gcsDestination": {
    object (GcsDestination)
  },
  "bigqueryDestination": {
    object (BigQueryDestination)
  }
  // End of list of possible types for union field destination.
}
Fields
datasetId

string

Required. The ID of the Dataset in the same Project and Location which data will be used to train the Model. The Dataset must use schema compatible with Model being trained, and what is compatible should be described in the used TrainingPipeline's trainingTaskDefinition. For tabular Datasets, all their data is exported to training, to pick and choose from.
annotationsFilter

string

Applicable only to Datasets that have DataItems and Annotations.

A filter on Annotations of the Dataset. Only Annotations that both match this filter and belong to DataItems not ignored by the split method are used in respectively training, validation or test role, depending on the role of the DataItem they are on (for the auto-assigned that role is decided by Vertex AI). A filter with same syntax as the one used in annotations.list may be used, but note here it filters across all Annotations of the Dataset, and not just within a single DataItem.
annotationSchemaUri

string

Applicable only to custom training with Datasets that have DataItems and Annotations.

Cloud Storage URI that points to a YAML file describing the annotation schema. The schema is defined as an OpenAPI 3.0.2 Schema Object. The schema files that can be used here are found in gs://google-cloud-aiplatform/schema/dataset/annotation/ , note that the chosen schema must be consistent with metadata of the Dataset specified by datasetId.

Only Annotations that both match this schema and belong to DataItems not ignored by the split method are used in respectively training, validation or test role, depending on the role of the DataItem they are on.

When used in conjunction with annotationsFilter, the Annotations used for training are filtered by both annotationsFilter and annotationSchemaUri.
Union field split. The instructions how the input data should be split between the training, validation and test sets. If no split type is provided, the fraction_split is used by default. split can be only one of the following:
fractionSplit

object (FractionSplit)

Split based on fractions defining the size of each set.
filterSplit

object (FilterSplit)

Split based on the provided filters for each set.
predefinedSplit

object (PredefinedSplit)

Supported only for tabular Datasets.

Split based on a predefined key.
timestampSplit

object (TimestampSplit)

Supported only for tabular Datasets.

Split based on the timestamp of the input data pieces.

Union field destination. Only applicable to Custom and Hyperparameter Tuning TrainingPipelines.

The destination of the training data to be written to.

Supported destination file formats: * For non-tabular data: "jsonl". * For tabular data: "csv" and "bigquery".

The following Vertex AI environment variables are passed to containers or python modules of the training task when this field is set:

  • AIP_DATA_FORMAT : Exported data format.
  • AIP_TRAINING_DATA_URI : Sharded exported training data uris.
  • AIP_VALIDATION_DATA_URI : Sharded exported validation data uris.
  • AIP_TEST_DATA_URI : Sharded exported test data uris. destination can be only one of the following:
gcsDestination

object (GcsDestination)

The Cloud Storage location where the training data is to be written to. In the given directory a new directory is created with name: dataset-<dataset-id>-<annotation-type>-<timestamp-of-training-call> where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. All training input data is written into that directory.

The Vertex AI environment variables representing Cloud Storage data URIs are represented in the Cloud Storage wildcard format to support sharded data. e.g.: "gs://.../training-*.jsonl"

  • AIP_DATA_FORMAT = "jsonl" for non-tabular data, "csv" for tabular data
  • AIP_TRAINING_DATA_URI = "gcsDestination/dataset---

  • AIP_VALIDATION_DATA_URI = "gcsDestination/dataset---

  • AIP_TEST_DATA_URI = "gcsDestination/dataset---
bigqueryDestination

object (BigQueryDestination)

Only applicable to custom training with tabular Dataset with BigQuery source.

The BigQuery project location where the training data is to be written to. In the given project a new dataset is created with name dataset_<dataset-id>_<annotation-type>_<timestamp-of-training-call> where timestamp is in YYYY_MM_DDThh_mm_ss_sssZ format. All training input data is written into that dataset. In the dataset three tables are created, training, validation and test.

  • AIP_DATA_FORMAT = "bigquery".
  • AIP_TRAINING_DATA_URI = "bigqueryDestination.dataset___

  • AIP_VALIDATION_DATA_URI = "bigqueryDestination.dataset___

  • AIP_TEST_DATA_URI = "bigqueryDestination.dataset___

FractionSplit

Assigns the input data to training, validation, and test sets as per the given fractions. Any of trainingFraction, validationFraction and testFraction may optionally be provided, they must sum to up to 1. If the provided ones sum to less than 1, the remainder is assigned to sets as decided by Vertex AI. If none of the fractions are set, by default roughly 80% of data is used for training, 10% for validation, and 10% for test.

JSON representation 
{
  "trainingFraction": number,
  "validationFraction": number,
  "testFraction": number
}
Fields
trainingFraction

number

The fraction of the input data that is to be used to train the Model.
validationFraction

number

The fraction of the input data that is to be used to validate the Model.
testFraction

number

The fraction of the input data that is to be used to evaluate the Model.

FilterSplit

Assigns input data to training, validation, and test sets based on the given filters, data pieces not matched by any filter are ignored. Currently only supported for Datasets containing DataItems. If any of the filters in this message are to match nothing, then they can be set as '-' (the minus sign).

Supported only for unstructured Datasets.

JSON representation 
{
  "trainingFilter": string,
  "validationFilter": string,
  "testFilter": string
}
Fields
trainingFilter

string

Required. A filter on DataItems of the Dataset. DataItems that match this filter are used to train the Model. A filter with same syntax as the one used in DatasetService.ListDataItems may be used. If a single DataItem is matched by more than one of the FilterSplit filters, then it is assigned to the first set that applies to it in the training, validation, test order.
validationFilter

string

Required. A filter on DataItems of the Dataset. DataItems that match this filter are used to validate the Model. A filter with same syntax as the one used in DatasetService.ListDataItems may be used. If a single DataItem is matched by more than one of the FilterSplit filters, then it is assigned to the first set that applies to it in the training, validation, test order.
testFilter

string

Required. A filter on DataItems of the Dataset. DataItems that match this filter are used to test the Model. A filter with same syntax as the one used in DatasetService.ListDataItems may be used. If a single DataItem is matched by more than one of the FilterSplit filters, then it is assigned to the first set that applies to it in the training, validation, test order.

PredefinedSplit

Assigns input data to training, validation, and test sets based on the value of a provided key.

Supported only for tabular Datasets.

JSON representation 
{
  "key": string
}
Fields
key

string

Required. The key is a name of one of the Dataset's data columns. The value of the key (either the label's value or value in the column) must be one of {training, validation, test}, and it defines to which set the given piece of data is assigned. If for a piece of data the key is not present or has an invalid value, that piece is ignored by the pipeline.

TimestampSplit

Assigns input data to training, validation, and test sets based on a provided timestamps. The youngest data pieces are assigned to training set, next to validation set, and the oldest to the test set.

Supported only for tabular Datasets.

JSON representation 
{
  "trainingFraction": number,
  "validationFraction": number,
  "testFraction": number,
  "key": string
}
Fields
trainingFraction

number

The fraction of the input data that is to be used to train the Model.
validationFraction

number

The fraction of the input data that is to be used to validate the Model.
testFraction

number

The fraction of the input data that is to be used to evaluate the Model.
key

string

Required. The key is a name of one of the Dataset's data columns. The values of the key (the values in the column) must be in RFC 3339 date-time format, where time-offset = "Z" (e.g. 1985-04-12T23:20:50.52Z). If for a piece of data the key is not present or has an invalid value, that piece is ignored by the pipeline.

Methods

cancel

 Cancels a TrainingPipeline.

create

 Creates a TrainingPipeline.

delete

 Deletes a TrainingPipeline.

get

 Gets a TrainingPipeline.

list

 Lists TrainingPipelines in a Location.