Class ImageDataset (1.37.0)

ImageDataset(
    dataset_name: str,
    project: typing.Optional[str] = None,
    location: typing.Optional[str] = None,
    credentials: typing.Optional[google.auth.credentials.Credentials] = None,
)

A managed image dataset resource for Vertex AI.

Use this class to work with a managed image dataset. To create a managed image dataset, you need a datasource file in CSV format and a schema file in YAML format. A schema is optional for a custom model. You put the CSV file and the schema into Cloud Storage buckets.

Use image data for the following objectives:

Single-label classification. For more information, see Prepare image training data for single-label classification.
Multi-label classification. For more information, see Prepare image training data for multi-label classification.
Object detection. For more information, see Prepare image training data for object detection.

The following code shows you how to create an image dataset by importing data from a CSV datasource file and a YAML schema file. The schema file you use depends on whether your image dataset is used for single-label classification, multi-label classification, or object detection.

my_dataset = aiplatform.ImageDataset.create(
    display_name="my-image-dataset",
    gcs_source=['gs://path/to/my/image-dataset.csv'],
    import_schema_uri=['gs://path/to/my/schema.yaml']
)

Properties

create_time

Time this resource was created.

display_name

Display name of this resource.

encryption_spec

Customer-managed encryption key options for this Vertex AI resource.

If this is set, then all resources created by this Vertex AI resource will be encrypted with the provided encryption key.

gca_resource

The underlying resource proto representation.

labels

User-defined labels containing metadata about this resource.

Read more about labels at https://goo.gl/xmQnxf

metadata_schema_uri

The metadata schema uri of this dataset resource.

name

Name of this resource.

resource_name

Full qualified resource name.

update_time

Time this resource was last updated.

Methods

ImageDataset

ImageDataset(
    dataset_name: str,
    project: typing.Optional[str] = None,
    location: typing.Optional[str] = None,
    credentials: typing.Optional[google.auth.credentials.Credentials] = None,
)

Retrieves an existing managed dataset given a dataset name or ID.

Parameters
Name	Description
`dataset_name`	`str` Required. A fully-qualified dataset resource name or dataset ID. Example: "projects/123/locations/us-central1/datasets/456" or "456" when project and location are initialized or passed.
`project`	`str` Optional project to retrieve dataset from. If not set, project set in aiplatform.init will be used.
`location`	`str` Optional location to retrieve dataset from. If not set, location set in aiplatform.init will be used.
`credentials`	`auth_credentials.Credentials` Custom credentials to use to retrieve this Dataset. Overrides credentials set in aiplatform.init.

create

create(
    display_name: typing.Optional[str] = None,
    gcs_source: typing.Optional[typing.Union[str, typing.Sequence[str]]] = None,
    import_schema_uri: typing.Optional[str] = None,
    data_item_labels: typing.Optional[typing.Dict] = None,
    project: typing.Optional[str] = None,
    location: typing.Optional[str] = None,
    credentials: typing.Optional[google.auth.credentials.Credentials] = None,
    request_metadata: typing.Optional[typing.Sequence[typing.Tuple[str, str]]] = (),
    labels: typing.Optional[typing.Dict[str, str]] = None,
    encryption_spec_key_name: typing.Optional[str] = None,
    sync: bool = True,
    create_request_timeout: typing.Optional[float] = None,
) -> google.cloud.aiplatform.datasets.image_dataset.ImageDataset

Creates a new image dataset.

Optionally imports data into the dataset when a source and import_schema_uri are passed in.

Parameters
Name	Description
`display_name`	`str` Optional. The user-defined name of the dataset. The name must contain 128 or fewer UTF-8 characters.
`gcs_source`	`Union[str, Sequence[str]]` Optional. The URI to one or more Google Cloud Storage buckets that contain your datasets. For example, `str: "gs://bucket/file.csv"` or `Sequence[str]: ["gs://bucket/file1.csv", "gs://bucket/file2.csv"]`.
`import_schema_uri`	`str` Optional. A URI for a YAML file stored in Cloud Storage that describes the import schema used to validate the dataset. The schema is an OpenAPI 3.0.2 Schema object.
`data_item_labels`	`Dict` Optional. A dictionary of label information. Each dictionary item contains a label and a label key. Each image in the dataset includes one dictionary of label information. If a data item is added or merged into a dataset, and that data item contains an image that's identical to an image that’s already in the dataset, then the data items are merged. If two identical labels are detected during the merge, each with a different label key, then one of the label and label key dictionary items is randomly chosen to be into the merged data item. Images and documents are compared using their binary data (bytes), not on their content. If annotation labels are referenced in a schema specified by the `import_schema_url` parameter, then the labels in the `data_item_labels` dictionary are overriden by the annotations.
`project`	`str` Optional. The name of the Google Cloud project to which this `ImageDataset` is uploaded. This overrides the project that was set by `aiplatform.init`.
`location`	`str` Optional. The Google Cloud region where this dataset is uploaded. This region overrides the region that was set by `aiplatform.init`.
`credentials`	`auth_credentials.Credentials` Optional. The credentials that are used to upload the `ImageDataset`. These credentials override the credentials set by `aiplatform.init`.
`request_metadata`	`Sequence[Tuple[str, str]]` Optional. Strings that contain metadata that's sent with the request.
`labels`	`Dict[str, str]` Optional. Labels with user-defined metadata to organize your Vertex AI Tensorboards. The maximum length of a key and of a value is 64 unicode characters. Labels and keys can contain only lowercase letters, numeric characters, underscores, and dashes. International characters are allowed. No more than 64 user labels can be associated with one Tensorboard (system labels are excluded). For more information and examples of using labels, see Using labels to organize Google Cloud Platform resources. System reserved label keys are prefixed with `aiplatform.googleapis.com/` and are immutable.
`encryption_spec_key_name`	`Optional[str]` Optional. The Cloud KMS resource identifier of the customer managed encryption key that's used to protect the dataset. The format of the key is `projects/my-project/locations/my-region/keyRings/my-kr/cryptoKeys/my-key`. The key needs to be in the same region as where the compute resource is created. If `encryption_spec_key_name` is set, this image dataset and all of its sub-resources are secured by this key. This `encryption_spec_key_name` overrides the `encryption_spec_key_name` set by `aiplatform.init`.
`sync`	`bool` If `true`, the `create` method creates an image dataset synchronously. If `false`, the `create` method creates an image dataset asynchronously.
`create_request_timeout`	`float` Optional. The number of seconds for the timeout of the create request.

Returns
Type	Description
`image_dataset (ImageDataset)`	An instantiated representation of the managed `ImageDataset` resource.

delete

delete(sync: bool = True) -> None

Deletes this Vertex AI resource. WARNING: This deletion is permanent.

Parameter
Name	Description
`sync`	`bool` Whether to execute this deletion synchronously. If False, this method will be executed in concurrent Future and any downstream object will be immediately returned and synced when the Future has completed.

export_data

export_data(output_dir: str) -> typing.Sequence[str]

Exports data to output dir to GCS.

Parameter

Name Description

output_dir

str

Required. The Google Cloud Storage location where the output is to be written to. In the given directory a new directory will be created with name: export-data- where timestamp is in YYYYMMDDHHMMSS format. All export output will be written into that directory. Inside that directory, annotations with the same schema will be grouped into sub directories which are named with the corresponding annotations' schema title. Inside these sub directories, a schema.yaml will be created to describe the output format. If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.

Returns
Type	Description
`exported_files (Sequence[str])`	All of the files that are exported in this export operation.

import_data

import_data(
    gcs_source: typing.Union[str, typing.Sequence[str]],
    import_schema_uri: str,
    data_item_labels: typing.Optional[typing.Dict] = None,
    sync: bool = True,
    import_request_timeout: typing.Optional[float] = None,
) -> google.cloud.aiplatform.datasets.dataset._Dataset

Upload data to existing managed dataset.

Parameters
Name	Description
`gcs_source`	`Union[str, Sequence[str]]` Required. Google Cloud Storage URI(-s) to the input file(s). May contain wildcards. For more information on wildcards, see https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames. .. rubric:: Examples str: "gs://bucket/file.csv" Sequence[str]: ["gs://bucket/file1.csv", "gs://bucket/file2.csv"]
`import_schema_uri`	`str` Required. Points to a YAML file stored on Google Cloud Storage describing the import format. Validation will be done against the schema. The schema is defined as an `OpenAPI 3.0.2 Schema Object https://tinyurl.com/y538mdwt`__.
`data_item_labels`	`Dict` Labels that will be applied to newly imported DataItems. If an identical DataItem as one being imported already exists in the Dataset, then these labels will be appended to these of the already existing one, and if labels with identical key is imported before, the old label value will be overwritten. If two DataItems are identical in the same import data operation, the labels will be combined and if key collision happens in this case, one of the values will be picked randomly. Two DataItems are considered identical if their content bytes are identical (e.g. image bytes or pdf bytes). These labels will be overridden by Annotation labels specified inside index file referenced by `import_schema_uri`, e.g. jsonl file. This arg is not for specifying the annotation name or the training target of your data, but for some global labels of the dataset. E.g., 'data_item_labels={"aiplatform.googleapis.com/ml_use":"training"}' specifies that all the uploaded data are used for training.
`sync`	`bool` Whether to execute this method synchronously. If False, this method will be executed in concurrent Future and any downstream object will be immediately returned and synced when the Future has completed.
`import_request_timeout`	`float` Optional. The timeout for the import request in seconds.

Returns
Type	Description
`dataset (Dataset)`	Instantiated representation of the managed dataset resource.

list

list(
    filter: typing.Optional[str] = None,
    order_by: typing.Optional[str] = None,
    project: typing.Optional[str] = None,
    location: typing.Optional[str] = None,
    credentials: typing.Optional[google.auth.credentials.Credentials] = None,
) -> typing.List[google.cloud.aiplatform.base.VertexAiResourceNoun]

List all instances of this Dataset resource.

Example Usage:

aiplatform.TabularDataset.list( filter='labels.my_key="my_value"', order_by='display_name' )

Parameters
Name	Description
`filter`	`str` Optional. An expression for filtering the results of the request. For field names both snake_case and camelCase are supported.
`order_by`	`str` Optional. A comma-separated list of fields to order by, sorted in ascending order. Use "desc" after a field name for descending. Supported fields: `display_name`, `create_time`, `update_time`
`project`	`str` Optional. Project to retrieve list from. If not set, project set in aiplatform.init will be used.
`location`	`str` Optional. Location to retrieve list from. If not set, location set in aiplatform.init will be used.
`credentials`	`auth_credentials.Credentials` Optional. Custom credentials to use to retrieve list. Overrides credentials set in aiplatform.init.

to_dict

to_dict() -> typing.Dict[str, typing.Any]

Returns the resource proto as a dictionary.

update

update(
    *,
    display_name: typing.Optional[str] = None,
    labels: typing.Optional[typing.Dict[str, str]] = None,
    description: typing.Optional[str] = None,
    update_request_timeout: typing.Optional[float] = None
) -> google.cloud.aiplatform.datasets.dataset._Dataset

Update the dataset. Updatable fields:

display_name
description
labels

Parameters
Name	Description
`display_name`	`str` Optional. The user-defined name of the Dataset. The name can be up to 128 characters long and can be consist of any UTF-8 characters.
`labels`	`Dict[str, str]` Optional. Labels with user-defined metadata to organize your Tensorboards. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. No more than 64 user labels can be associated with one Tensorboard (System labels are excluded). See https://goo.gl/xmQnxf for more information and examples of labels. System reserved label keys are prefixed with "aiplatform.googleapis.com/" and are immutable.
`description`	`str` Optional. The description of the Dataset.
`update_request_timeout`	`float` Optional. The timeout for the update request in seconds.

Returns
Type	Description
`dataset (Dataset)`	Updated dataset.

wait

wait()

Helper method that blocks until all futures are complete.