Resource: PipelineJob
An instance of a machine learning PipelineJob.
name
string
Output only. The resource name of the PipelineJob.
displayName
string
The display name of the Pipeline. The name can be up to 128 characters long and can consist of any UTF-8 characters.
Output only. Pipeline creation time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. Pipeline start time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. Pipeline end time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. timestamp when this PipelineJob was most recently updated.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
The spec of the pipeline.
Output only. The detailed state of the job.
Output only. The details of pipeline run. Not available in the list view.
Output only. The error that occurred during pipeline execution. Only populated when the pipeline's state is FAILED or CANCELLED.
labels
map (key: string, value: string)
The labels with user-defined metadata to organize PipelineJob.
label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed.
See https://goo.gl/xmQnxf for more information and examples of labels.
Note there is some reserved label key for Vertex AI Pipelines. - vertex-ai-pipelines-run-billing-id
, user set value will get overrided.
Runtime config of the pipeline.
Customer-managed encryption key spec for a pipelineJob. If set, this PipelineJob and all of its sub-resources will be secured by this key.
serviceAccount
string
The service account that the pipeline workload runs as. If not specified, the Compute Engine default service account in the project will be used. See https://cloud.google.com/compute/docs/access/service-accounts#default_service_account
Users starting the pipeline must have the iam.serviceAccounts.actAs
permission on this service account.
network
string
The full name of the Compute Engine network to which the Pipeline Job's workload should be peered. For example, projects/12345/global/networks/myVPC
. Format is of the form projects/{project}/global/networks/{network}
. Where {project} is a project number, as in 12345
, and {network} is a network name.
Private services access must already be configured for the network. Pipeline job will apply the network configuration to the Google Cloud resources being launched, if applied, such as Vertex AI Training or Dataflow job. If left unspecified, the workload is not peered with any network.
reservedIpRanges[]
string
A list of names for the reserved ip ranges under the VPC network that can be used for this Pipeline Job's workload.
If set, we will deploy the Pipeline Job's workload within the provided ip ranges. Otherwise, the job will be deployed to any ip ranges under the provided VPC network.
Example: ['vertex-ai-ip-range'].
Optional. Configuration for PSC-I for PipelineJob.
templateUri
string
A template uri from where the PipelineJob.pipeline_spec
, if empty, will be downloaded. Currently, only uri from Vertex Template Registry & Gallery is supported. Reference to https://cloud.google.com/vertex-ai/docs/pipelines/create-pipeline-template.
Output only. Pipeline template metadata. Will fill up fields if PipelineJob.template_uri
is from supported template registry.
scheduleName
string
Output only. The schedule resource name. Only returned if the Pipeline is created by Schedule API.
preflightValidations
boolean
Optional. Whether to do component level validations before job creation.
satisfiesPzs
boolean
Output only. reserved for future use.
satisfiesPzi
boolean
Output only. reserved for future use.
Optional. The original pipeline job id if this pipeline job is a rerun of a previous pipeline job.
Optional. The rerun configs for each task in the pipeline job. By default, the rerun will: 1. Use the same input artifacts as the original run. 2. Use the same input parameters as the original run. 3. Skip all the tasks that are already succeeded in the original run. 4. Rerun all the tasks that are not succeeded in the original run. By providing this field, users can override the default behavior and specify the rerun config for each task.
JSON representation |
---|
{ "name": string, "displayName": string, "createTime": string, "startTime": string, "endTime": string, "updateTime": string, "pipelineSpec": { object }, "state": enum ( |
PipelineJobDetail
The runtime detail of PipelineJob.
Output only. The context of the pipeline.
Output only. The context of the current pipeline run.
Output only. The runtime details of the tasks under the pipeline.
JSON representation |
---|
{ "pipelineContext": { object ( |
PipelineTaskDetail
The runtime detail of a task execution.
Output only. The system generated id of the task.
Output only. The id of the parent task if the task is within a component scope. Empty if the task is at the root level.
taskName
string
Output only. The user specified name of the task that is defined in pipelineSpec
.
Output only. Task create time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. Task start time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. Task end time.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. The detailed execution info.
Output only. state of the task.
Output only. The execution metadata of the task.
Output only. The error that occurred during task execution. Only populated when the task's state is FAILED or CANCELLED.
Output only. A list of task status. This field keeps a record of task status evolving over time.
Output only. The runtime input artifacts of the task.
Output only. The runtime output artifacts of the task.
JSON representation |
---|
{ "taskId": string, "parentTaskId": string, "taskName": string, "createTime": string, "startTime": string, "endTime": string, "executorDetail": { object ( |
PipelineTaskExecutorDetail
The runtime detail of a pipeline executor.
Union field details
.
details
can be only one of the following:
Output only. The detailed info for a container executor.
Output only. The detailed info for a custom job executor.
JSON representation |
---|
{ // Union field |
ContainerDetail
The detail of a container execution. It contains the job names of the lifecycle of a container execution.
mainJob
string
Output only. The name of the CustomJob
for the main container execution.
preCachingCheckJob
string
Output only. The name of the CustomJob
for the pre-caching-check container execution. This job will be available if the PipelineJob.pipeline_spec
specifies the pre_caching_check
hook in the lifecycle events.
failedMainJobs[]
string
Output only. The names of the previously failed CustomJob
for the main container executions. The list includes the all attempts in chronological order.
failedPreCachingCheckJobs[]
string
Output only. The names of the previously failed CustomJob
for the pre-caching-check container executions. This job will be available if the PipelineJob.pipeline_spec
specifies the pre_caching_check
hook in the lifecycle events. The list includes the all attempts in chronological order.
JSON representation |
---|
{ "mainJob": string, "preCachingCheckJob": string, "failedMainJobs": [ string ], "failedPreCachingCheckJobs": [ string ] } |
CustomJobDetail
The detailed info for a custom job executor.
job
string
Output only. The name of the CustomJob
.
failedJobs[]
string
Output only. The names of the previously failed CustomJob
. The list includes the all attempts in chronological order.
JSON representation |
---|
{ "job": string, "failedJobs": [ string ] } |
State
Specifies state of TaskExecution
Enums | |
---|---|
STATE_UNSPECIFIED |
Unspecified. |
PENDING |
Specifies pending state for the task. |
RUNNING |
Specifies task is being executed. |
SUCCEEDED |
Specifies task completed successfully. |
CANCEL_PENDING |
Specifies Task cancel is in pending state. |
CANCELLING |
Specifies task is being cancelled. |
CANCELLED |
Specifies task was cancelled. |
FAILED |
Specifies task failed. |
SKIPPED |
Specifies task was skipped due to cache hit. |
NOT_TRIGGERED |
Specifies that the task was not triggered because the task's trigger policy is not satisfied. The trigger policy is specified in the condition field of PipelineJob.pipeline_spec . |
PipelineTaskStatus
A single record of the task status.
Output only. Update time of this status.
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z"
and "2014-10-02T15:01:23.045123456Z"
.
Output only. The state of the task.
Output only. The error that occurred during the state. May be set when the state is any of the non-final state (PENDING/RUNNING/CANCELLING) or FAILED state. If the state is FAILED, the error here is final and not going to be retried. If the state is a non-final state, the error indicates a system-error being retried.
ArtifactList
A list of artifact metadata.
Output only. A list of artifact metadata.
JSON representation |
---|
{
"artifacts": [
{
object ( |
RuntimeConfig
The runtime config of a PipelineJob.
Deprecated. Use RuntimeConfig.parameter_values
instead. The runtime parameters of the PipelineJob. The parameters will be passed into PipelineJob.pipeline_spec
to replace the placeholders at runtime. This field is used by pipelines built using PipelineJob.pipeline_spec.schema_version
2.0.0 or lower, such as pipelines built using Kubeflow Pipelines SDK 1.8 or lower.
gcsOutputDirectory
string
Required. A path in a Cloud Storage bucket, which will be treated as the root output directory of the pipeline. It is used by the system to generate the paths of output artifacts. The artifact paths are generated with a sub-path pattern {job_id}/{taskId}/{outputKey}
under the specified output directory. The service account specified in this pipeline must have the storage.objects.get
and storage.objects.create
permissions for this bucket.
The runtime parameters of the PipelineJob. The parameters will be passed into PipelineJob.pipeline_spec
to replace the placeholders at runtime. This field is used by pipelines built using PipelineJob.pipeline_spec.schema_version
2.1.0, such as pipelines built using Kubeflow Pipelines SDK 1.9 or higher and the v2 DSL.
Represents the failure policy of a pipeline. Currently, the default of a pipeline is that the pipeline will continue to run until no more tasks can be executed, also known as PIPELINE_FAILURE_POLICY_FAIL_SLOW. However, if a pipeline is set to PIPELINE_FAILURE_POLICY_FAIL_FAST, it will stop scheduling any new tasks when a task has failed. Any scheduled tasks will continue to completion.
The runtime artifacts of the PipelineJob. The key will be the input artifact name and the value would be one of the InputArtifact.
JSON representation |
---|
{ "parameters": { string: { object ( |
Value
value is the value of the field.
Union field value
.
value
can be only one of the following:
An integer value.
doubleValue
number
A double value.
stringValue
string
A string value.
JSON representation |
---|
{ // Union field |
PipelineFailurePolicy
Represents the failure policy of a pipeline. Currently, the default of a pipeline is that the pipeline will continue to run until no more tasks can be executed, also known as PIPELINE_FAILURE_POLICY_FAIL_SLOW. However, if a pipeline is set to PIPELINE_FAILURE_POLICY_FAIL_FAST, it will stop scheduling any new tasks when a task has failed. Any scheduled tasks will continue to completion.
Enums | |
---|---|
PIPELINE_FAILURE_POLICY_UNSPECIFIED |
Default value, and follows fail slow behavior. |
PIPELINE_FAILURE_POLICY_FAIL_SLOW |
Indicates that the pipeline should continue to run until all possible tasks have been scheduled and completed. |
PIPELINE_FAILURE_POLICY_FAIL_FAST |
Indicates that the pipeline should stop scheduling new tasks after a task has failed. |
InputArtifact
The type of an input artifact.
Union field kind
.
kind
can be only one of the following:
artifactId
string
Artifact resource id from MLMD. Which is the last portion of an artifact resource name: projects/{project}/locations/{location}/metadataStores/default/artifacts/{artifactId}
. The artifact must stay within the same project, location and default metadatastore as the pipeline.
JSON representation |
---|
{ // Union field |
PscInterfaceConfig
Configuration for PSC-I.
networkAttachment
string
Optional. The name of the Compute Engine network attachment to attach to the resource within the region and user project. To specify this field, you must have already created a network attachment. This field is only used for resources using PSC-I.
JSON representation |
---|
{ "networkAttachment": string } |
PipelineTemplateMetadata
Pipeline template metadata if PipelineJob.template_uri
is from supported template registry. Currently, the only supported registry is Artifact Registry.
version
string
The versionName in artifact registry.
Will always be presented in output if the PipelineJob.template_uri
is from supported template registry.
Format is "sha256:abcdef123456...".
JSON representation |
---|
{ "version": string } |
PipelineTaskRerunConfig
user provided rerun config to submit a rerun pipelinejob. This includes 1. Which task to rerun 2. user override input parameters and artifacts.
Optional. The system generated id of the task. Retrieved from original run.
taskName
string
Optional. The name of the task.
Optional. The runtime input of the task overridden by the user.
skipTask
boolean
Optional. Whether to skip this task. Default value is False.
skipDownstreamTasks
boolean
Optional. Whether to skip downstream tasks. Default value is False.
JSON representation |
---|
{
"taskId": string,
"taskName": string,
"inputs": {
object ( |
Inputs
Runtime inputs data of the task.
Optional. Input artifacts.
Optional. Input parameters.
JSON representation |
---|
{
"artifacts": {
string: {
object ( |
ArtifactList
A list of artifact metadata.
Optional. A list of artifact metadata.
JSON representation |
---|
{
"artifacts": [
{
object ( |
RuntimeArtifact
The definition of a runtime artifact.
name
string
The name of an artifact.
The type of the artifact.
uri
string
The URI of the artifact.
The properties of the artifact. Deprecated. Use RuntimeArtifact.metadata
instead.
The custom properties of the artifact. Deprecated. Use RuntimeArtifact.metadata
instead.
Properties of the Artifact.
JSON representation |
---|
{ "name": string, "type": { object ( |
ArtifactTypeSchema
The definition of a artifact type in MLMD.
schemaVersion
string
The schema version of the artifact. If the value is not set, it defaults to the latest version in the system.
Union field kind
.
kind
can be only one of the following:
schemaTitle
string
The name of the type. The format of the title must be: <namespace>.<title>
. Examples: - aiplatform.Model
- acme.CustomModel
When this field is set, the type must be pre-registered in the MLMD store.
schemaUri
(deprecated)
string
Points to a YAML file stored on Cloud Storage describing the format. Deprecated. Use [PipelineArtifactTypeSchema.schema_title][] or [PipelineArtifactTypeSchema.instance_schema][] instead.
instanceSchema
string
Contains a raw YAML string, describing the format of the properties of the type.
JSON representation |
---|
{ "schemaVersion": string, // Union field |
Methods |
|
---|---|
|
Batch cancel PipelineJobs. |
|
Batch deletes PipelineJobs The Operation is atomic. |
|
Cancels a PipelineJob. |
|
Creates a PipelineJob. |
|
Deletes a PipelineJob. |
|
Gets a PipelineJob. |
|
Lists PipelineJobs in a Location. |