Manage ML metadata

This guide demonstrates how to manage your ML Metadata.

Before you begin

The first time that you use Vertex ML Metadata in a Google Cloud project, Vertex AI creates your project's metadata store.

If you want your metadata encrypted using a customer-managed encryption key (CMEK), you must create your metadata store using a CMEK before you use Vertex ML Metadata to track or analyze metadata. Use the create a metadata store that uses a CMEK instructions to configure your project's metadata store.

Artifact management

Create an artifact

Use the following instructions to create an artifact.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the artifact is created. The default metadata store is named default.
  • ARTIFACT_ID: (Optional.) The ID of the artifact record. If the artifact ID is not specified, Vertex ML Metadata created a unique identifier for this artifact.
  • DISPLAY_NAME: The artifact's display name. This field may contain up to 128 Unicode characters.
  • URI: (Optional.) The location where the artifact is stored.
  • ARTIFACT_STATE: (Optional.) A value from the State enumeration that represents the current state of the artifact. This field is managed by client applications. Vertex ML Metadata does not check the validity of state transitions.
  • METADATA_SCHEMA_TITLE: The title of the schema that describes the metadata field.
  • METADATA_SCHEMA_VERSION: (Optional.) The version of the schema that describes the metadata field.
  • METADATA: (Optional.) Properties that describe the artifact, such as the type of dataset.
  • DESCRIPTION: (Optional.) A description of the execution.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/artifacts?artifactId=ARTIFACT_ID

Request JSON body:

{
  "displayName": "DISPLAY_NAME",
  "uri": "URI",
  "state": "ARTIFACT_STATE",
  "schemaTitle": "METADATA_SCHEMA_TITLE",
  "schemaVersion": "METADATA_SCHEMA_VERSION",
  "metadata": {
    METADATA
  },
  "description": "DESCRIPTION"
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/12345/locations/us-central1/metadataStores/default/artifacts/example-artifact",
  "displayName": "Example artifact",
  "uri": "gs://your_bucket_name/artifacts/dataset.csv",
  "etag": "67891011",
  "createTime": "2021-05-18T00:29:24.344Z",
  "updateTime": "2021-05-18T00:29:24.344Z",
  "state": "LIVE",
  "schemaTitle": "system.Dataset",
  "schemaVersion": "0.0.1",
  "metadata": {
    "payload_format": "CSV"
  },
  "description": "Description of the example artifact."
}

Look up an existing artifact

Artifacts represent data used or produced by your ML workflow, such as datasets and models. Use the following instructions to look-up an existing artifact.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the artifact is created. The default metadata store is named default.
  • PAGE_SIZE: (Optional.) The maximum number of artifacts to return. If this value is not specified, a maximum of 100 records are returned.
  • PAGE_TOKEN: (Optional.) A page token from a previous MetadataService.ListArtifacts call. Specify this token to get the next page of results.
  • FILTER: Specifies the conditions required to include an artifact in the result set.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/artifacts?pageSize=PAGE_SIZE&pageToken=PAGE_TOKEN&filter=FILTER

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "artifacts": [
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/artifacts/example-artifact",
      "displayName": "Example artifact",
      "uri": "gs://your_bucket_name/artifacts/dataset.csv",
      "etag": "67891011",
      "createTime": "2021-05-18T00:33:13.833Z",
      "updateTime": "2021-05-18T00:33:13.833Z",
      "state": "LIVE",
      "schemaTitle": "system.Dataset",
      "schemaVersion": "0.0.1",
      "metadata": {
        "payload_format": "CSV"
      },
      "description": "Description of the example artifact."
    },
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/artifacts/example-artifact-2",
      "displayName": "Another example artifact",
      "uri": "gs://your_bucket_name/artifacts/dataset-2.csv",
      "etag": "67891012",
      "createTime": "2021-05-18T00:29:24.344Z",
      "updateTime": "2021-05-18T00:29:24.344Z",
      "state": "LIVE",
      "schemaTitle": "system.Dataset",
      "schemaVersion": "0.0.1",
      "metadata": {
        "payload_format": "CSV"
      },
      "description": "Description of the other example artifact."
    }
  ]
}

Delete an existing artifact

Use the following instructions to delete an artifact.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the artifact was created. The default metadata store is named default.
  • ARTIFACT_ID: The ID of the artifact record to delete.

HTTP method and URL:

DELETE https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/artifacts/ARTIFACT_ID

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/artifacts/ARTIFACT_ID/operations/85544",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.DeleteOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T20:05:30.179713Z",
      "updateTime": "2021-07-21T20:05:30.179713Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

Purge artifacts

Use the following instructions to delete multiple artifacts based on a filter condition.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the artifacts were created. The default metadata store is named default.
  • FILTER: Specifies the conditions required by the artifacts to be deleted. For example:
    • Filters for all artifacts that contain example in the display name: "display_name = \"*example*\"".
    • Filters for all artifacts created before 2020-11-19T11:30:00-04:00: "create_time < \"2020-11-19T11:30:00-04:00\"".
  • FORCE: Indicates whether to perform the actual purge or not. If the flag is set to false, the method will return a sample of artifact names that would be deleted.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/artifacts:purge

Request JSON body:

{
  "filter": "FILTER",
  "force": FORCE
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/operations/11531",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeArtifactsMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T21:02:33.757991Z",
      "updateTime": "2021-07-21T21:02:33.757991Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeArtifactsResponse",
    "purgeCount": "15"
  }
}

Execution management

Create an execution

Executions represent a step in your ML workflow. Use the following instructions to create an execution.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the execution is created. The default metadata store is named default.
  • EXECUTION_ID: (Optional.) The ID of the execution record. If the execution ID is not specified, Vertex ML Metadata created a unique identifier for this execution.
  • DISPLAY_NAME: The execution's display name. This field may contain up to 128 Unicode characters.
  • EXECUTION_STATE: (Optional.) A value from the State enumeration that represents the current state of the execution. This field is managed by client applications. Vertex ML Metadata does not check the validity of state transitions.
  • METADATA_SCHEMA_TITLE: The title of the schema that describes the metadata field.
  • METADATA_SCHEMA_VERSION: (Optional.) The version of the schema that describes the metadata field.
  • METADATA: (Optional.) Properties that describe the execution, such as the execution parameters.
  • DESCRIPTION: (Optional.) A description of the execution.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/executions?executionId=EXECUTION_ID

Request JSON body:

{
  "displayName": "DISPLAY_NAME",
  "state": "EXECUTION_STATE",
  "schemaTitle": "METADATA_SCHEMA_TITLE",
  "schemaVersion": "METADATA_SCHEMA_VERSION",
  "metadata": {
    METADATA
  },
  "description": "DESCRIPTION"
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/12345/locations/us-central1/metadataStores/default/executions/example-execution",
  "displayName": "Example Execution",
  "etag": "67891011",
  "createTime": "2021-05-18T00:04:49.659Z",
  "updateTime": "2021-05-18T00:04:49.659Z",
  "schemaTitle": "system.Run",
  "schemaVersion": "0.0.1",
  "metadata": {},
  "description": "Description of the example execution."
}

Look up an existing execution

Use the following instructions to look-up an existing execution.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the execution is created. The default metadata store is named default.
  • PAGE_SIZE: (Optional.) The maximum number of executions to return. If this value is not specified, a maximum of 100 records are returned.
  • PAGE_TOKEN: (Optional.) A page token from a previous MetadataService.ListExecutions call. Specify this token to get the next page of results.
  • FILTER: Specifies the conditions required to include an execution in the result set.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/executions?pageSize=PAGE_SIZE&pageToken=PAGE_TOKEN&filter=FILTER

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "executions": [
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/executions/example-execution-1",
      "displayName": "Example execution 1",
      "etag": "67891011",
      "createTime": "2021-05-18T00:06:56.177Z",
      "updateTime": "2021-05-18T00:06:56.177Z",
      "schemaTitle": "system.Run",
      "schemaVersion": "0.0.1",
      "metadata": {},
      "description": "Descrption of the example execution."
    },
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/executions/example-execution-2",
      "displayName": "Example execution 2",
      "etag": "67891011",
      "createTime": "2021-05-18T00:04:49.659Z",
      "updateTime": "2021-05-18T00:04:49.659Z",
      "schemaTitle": "system.Run",
      "schemaVersion": "0.0.1",
      "metadata": {},
      "description": "Descrption of the example execution."
    }
  ]
}

Delete an existing execution

Use the following instructions to delete an execution.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the execution was created. The default metadata store is named default.
  • EXECUTION_ID: The ID of the execution record to delete.

HTTP method and URL:

DELETE https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/executions/EXECUTION_ID

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/executions/EXECUTION_ID/operations/85544",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.DeleteOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T20:05:30.179713Z",
      "updateTime": "2021-07-21T20:05:30.179713Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

Purge executions

Use the following instructions to delete multiple executions based on a filter condition.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the executions is created. The default metadata store is named default.
  • FILTER: Specifies the conditions required by the executions to be deleted. For example:
    • Filters for all executions that contain example in the display name: "display_name = \"*example*\"".
    • Filters for all executions created before 2020-11-19T11:30:00-04:00: "create_time < \"2020-11-19T11:30:00-04:00\"".
  • FORCE: Indicates whether to perform the actual purge or not. If the flag is set to false, the method will return a sample of execution names that would be deleted.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/executions:purge

Request JSON body:

{
  "filter": "FILTER",
  "force": FORCE
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/operations/11531",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeExecutionsMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T21:02:45.757991Z",
      "updateTime": "2021-07-21T21:02:45.757991Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeExecutionsResponse",
    "purgeCount": "2"
  }
}

Context management

Create a context

Contexts let you group sets of artifacts and executions together. Use the following instructions to create a context.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the execution is created. The default metadata store is named default.
  • CONTEXT_ID: (Optional.) The ID of the context record. If the context ID is not specified, Vertex ML Metadata created a unique identifier for this context.
  • DISPLAY_NAME: The context's display name. This field may contain up to 128 Unicode characters.
  • Specify the PARENT_CONTEXT resource name for any parent contexts. A context can not have more than 10 parent contexts.
  • METADATA_SCHEMA_TITLE: The title of the schema that describes the metadata field.
  • METADATA_SCHEMA_VERSION: (Optional.) The version of the schema that describes the metadata field.
  • METADATA: (Optional.) Properties that describe the context.
  • DESCRIPTION: (Optional.) A description of the execution.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/contexts?contextId=CONTEXT_ID

Request JSON body:

{
  "displayName": "DISPLAY_NAME:",
  "parentContexts": [
    "PARENT_CONTEXT_1",
    "PARENT_CONTEXT_2"
  ],
  "schemaTitle": "METADATA_SCHEMA_TITLE",
  "schemaVersion": "METADATA_SCHEMA_VERSION",
  "metadata": {
    METADATA
  },
  "description": "DESCRIPTION"
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/12345/locations/us-central1/metadataStores/default/contexts/example-context",
  "displayName": "Example context:",
  "etag": "67891011",
  "createTime": "2021-05-18T01:52:51.642Z",
  "updateTime": "2021-05-18T01:52:51.642Z",
  "schemaTitle": "system.Experiment",
  "schemaVersion": "0.0.1",
  "metadata": {},
  "description": "Description of the example context."
}

Look up an existing context

Use the following instructions to look-up an existing context.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the context is created. The default metadata store is named default.
  • PAGE_SIZE: (Optional.) The maximum number of contexts to return. If this value is not specified, a maximum of 100 records are returned.
  • PAGE_TOKEN: (Optional.) A page token from a previous MetadataService.ListExecutions call. Specify this token to get the next page of results.
  • FILTER: Specifies the conditions required to include a context in the result set.

HTTP method and URL:

GET https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/contexts?pageSize=PAGE_SIZE&pageToken=PAGE_TOKEN&filter=FILTER

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "contexts": [
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/contexts/experiment-1",
      "displayName": "Experiment 1",
      "etag": "67891011",
      "createTime": "2021-05-18T22:36:02.153Z",
      "updateTime": "2021-05-18T22:36:02.153Z",
      "parentContexts": [],
      "schemaTitle": "system.Experiment",
      "schemaVersion": "0.0.1",
      "metadata": {}
    },
    {
      "name": "projects/12345/locations/us-central1/metadataStores/default/contexts/pipeline-run-1",
      "displayName": "Pipeline run 1",
      "etag": "67891011",
      "createTime": "2021-05-18T22:35:02.600Z",
      "updateTime": "2021-05-18T22:35:02.600Z",
      "parentContexts": [],
      "schemaTitle": "system.PipelineRun",
      "schemaVersion": "0.0.1",
      "metadata": {}
    }
  ]
}

Delete an existing context

Use the following instructions to delete a context.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the context was created. The default metadata store is named default.
  • CONTEXT_ID: The ID of the context record to delete.

HTTP method and URL:

DELETE https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/contexts/CONTEXT_ID

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/contexts/CONTEXT_ID/operations/85544",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.DeleteOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T20:05:30.179713Z",
      "updateTime": "2021-07-21T20:05:30.179713Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

Purge contexts

Use the following instructions to delete multiple contexts based on a filter condition.

Before using any of the request data, make the following replacements:

  • LOCATION: Your region.
  • PROJECT: Your project ID or project number.
  • METADATA_STORE: The metadata store ID where the contexts were created. The default metadata store is named default.
  • FILTER: Specifies the conditions required by the contexts to be deleted. For example:
    • Filters for all contexts that contain example in the display name: "display_name = \"*example*\"".
    • Filters for all contexts created before 2020-11-19T11:30:00-04:00: "create_time < \"2020-11-19T11:30:00-04:00\"".
  • FORCE: Indicates whether to perform the actual purge or not. If the flag is set to false, the method will return a sample of context names that would be deleted.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/contexts:purge

Request JSON body:

{
  "filter": "FILTER",
  "force": FORCE
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT/locations/LOCATION/metadataStores/METADATA_STORE/operations/11531",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeContextsMetadata",
    "genericMetadata": {
      "createTime": "2021-07-21T21:02:40.757991Z",
      "updateTime": "2021-07-21T21:02:40.757991Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1beta1.PurgeContextsResponse",
    "purgeCount": "5"
  }
}

What's next