This page provides best practices for creating your datastore and populating it with document information.
Overview
Your datastore contains a collection of document objects, such as videos or
news articles, that you have uploaded to Discovery for Media. This is
represented in Discovery Engine by the DataStore resource.
A document is any item that you upload to Discovery for Media to be
recommended. For Discovery for Media, you might upload a video, news article,
music file, or podcast as a document. The Document object in
Discovery Engine captures all metadata information of items to be
recommended.
The data you import into Discovery for Media has a direct effect on the quality of the resulting model, and therefore on the quality of the results Discovery for Media provides. In general, the more accurate and specific information you can provide, the higher quality your model.
Your datastore should be kept up to date. You can upload datastore changes as often as needed; ideally, every day for a datastore with a high rate of change. There is no charge for uploading datastore information. For more information, see Keeping your datastore up to date.
You can have one datastore per Google Cloud project. After you enable Discovery for Media in the Discovery Engine console and accept Discovery Engine terms of use, the console prompts you to create the default datastore.
Documents
The datastore is a collection of document objects. The Document object in
Discovery Engine captures all metadata information of items to be
recommended.
A document is any item you upload to Discovery for Media to be recommended, such as a video or news article.
JSON Schema for Document
JSON Schema is a JSON-based format that describes the structure of JSON documents. When using Media Recommendations, documents must use the predefined JSON schema shown below.
For more about the JSON Schema, also see the JSON Schema specification documentation at json-schema.org.
Documents are uploaded with either a JSON or Struct data representation. Make sure the document JSON or Struct conforms to the following JSON schema. The JSON schema uses JSON Schema 2020-12 for validation.
{ "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { "title": { "type": "string", }, "description": { "type": "string", }, "language_code": { "type": "string", }, "categories": { "type": "array", "items": { "type": "string", } }, "uri": { "type": "string", }, "images": { "type": "array", "items": { "type": "object", "properties": { "uri": { "type": "string", }, "name": { "type": "string", } }, } }, "media_type": { "type": "string", }, "in_languages": { "type": "array", "items": { "type": "string", } }, "country_of_origin": { "type": "string", }, "content_index": { "type": "integer", }, "persons": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", }, "role": { "type": "string", }, "custom_role": { "type": "string", }, "rank": { "type": "integer", }, "uri": { "type": "string", } }, "required": ["name", "role"], } }, "organizations": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", }, "role": { "type": "string", }, "custom_role": { "type": "string", }, "rank": { "type": "integer", }, "uri": { "type": "string", } }, "required": ["name", "role"], } }, "hash_tags": { "type": "array", "items": { "type": "string", } }, "filter_tags": { "type": "array", "items": { "type": "string", } }, "duration": { "type": "string", }, "content_rating": { "type": "array", "items": { "type": "string", } }, "aggregate_ratings": { "type": "array", "items": { "type": "object", "properties": { "rating_source": { "type": "string", }, "rating_score": { "type": "number", }, "rating_count": { "type": "integer", } }, "required": ["rating_source"], } }, "availability_start_time": { "type": "string", }, "production_year": { "type": "integer", } }, "required": ["title", "uri", "availability_start_time"], }
Sample JSON Document object
The following example shows an example of a JSON Document object.
{ "title": "Test document title", "description": "Test document description", "language_code": "en-US", "categories": [ "sports > clip", "sports > highlight" ], "uri": "http://www.example.com", "images": [ { "uri": "http://example.com/img1", "name": "image_1" } ], "media_type": "sports-game", "in_languages": [ "en-US" ], "country_of_origin": "US", "content_index": 0, "persons": [ { "name": "sports person", "role": "player", "rank": 0, "uri": "http://example.com/person" }, ], "organizations": [ { "name": "sports team", "role": "team", "rank": 0, "uri": "http://example.com/team" }, ], "hash_tags": [ "tag1" ], "filter_tags": [ "filter_tag" ], "duration": "100s", "production_year": 1900, "content_rating": [ "PG-13" ], "aggregate_ratings": [ { "rating_source": "imdb", "rating_score": 4.5, "rating_count": 1250 } ], "availability_start_time": "2022-08-26T23:00:17Z" }
Document fields
This section lists the field values you provide when you create document items in your datastore. The values should correspond with the values used in your internal document database, and should accurately reflect the item represented, because they are included in training your models.
All document information you provide can be used to improve the quality of recommendations. Be sure to provide as many fields as possible.
Document object fields
The following fields are top-level API fields for the Document object. Also
refer to these fields on the Document reference page.
| Field | Notes |
|---|---|
name
|
The full, unique resource name of the document. Required for all
Document methods except for create and
import. During import, the name is automatically generated
and does not need to be manually provided.
|
id
|
The document ID used by your internal database. The ID field must
be unique across your entire datastore. The same value is used when you
record a user event, and is also returned by the recommend
method.
|
schemaId
|
Required. The identifier of the schema located in the same datastore. Should be set as "default_schema", which is automatically created when the default datastore is created. |
parentDocumentId
|
The ID of the parent document. For top level (root) documents,
parent_document_id can be empty or can point to itself. For
child documents, parent_document_id should point to a valid
root document.
|
Key properties
The following properties are defined using the predefined JSON Schema format for Discovery for Media.
For more about JSON properties, also see the Understanding JSON Schema documentation for properties at json-schema.org.
The following table defines flat properties.
| Field name | Notes |
|---|---|
title
|
String - required Document title from your database. A UTF-8 encoded string. Limited to 1000 characters. |
categories
|
String - required Document categories. This property is repeated for supporting one document belonging to several parallel categories. Use the full category path for higher quality results.
To represent the full path of a category, use the For example:
A document can contain at most 250 categories. Each category is a UTF-8 encoded string with a length limit of 5000 characters. |
uri
|
String - required URI of the document. Length limit of 5000 characters. |
description
|
String - optional Description of the document. Length limit of 5000 characters. |
language_code
|
String - optional Language of the title/description and other string attributes. Use language tags defined by BCP 47. For document recommendation, this field is ignored and the model automatically detects the text language. The document can include text in different languages, but duplicating documents to provide text in multiple languages can result in degraded model performance.
For document search this field is in use. It defaults to "en-US" if unset.
For example, |
The following table defines hierarchical key properties.
| Field name | Notes |
|---|---|
images
|
Object - optional - repeated Root key property for encapsulating the image related properties. |
images.uri
|
String - optional URI of the image. Length limit of 5,000 characters. |
images.name
|
String - optional Name of the image. Length limit of 128 characters. |
The following table defines flat key media properties.
| Field name | Notes |
|---|---|
duration
|
String - optional
Duration of the media content. Duration should be encoded as a string.
Encoding should be the same as the |
availability_start_time
|
String - required The time that the content is available to the end-users. This field identifies the freshness of a content for end-users. The timestamp should conform to RFC 3339 standard. For example:
|
media_type
|
String - optional Top-level category. Supported types: episode, movie, concert, event, live-event, broadcast, tv-series, video-game, clip, vlog, audio, audio-book, music, album , articles, news, radio, podcast, book, sports-game |
in_languages
|
String - optional - repeated Language of the media contents. Use language tags defined by BCP 47.
For example: |
country_of_origin
|
String - optional Media document country of origin. Length limit of 128 characters.
For example: |
content_index
|
Int - optional Content index of the media document. Content index field can be used to order the documents relative to others. For example, episode number can be used as the content index. Content index should be a non-negative integer.
For example: |
filter_tags
|
String - optional - repeated Filter tags for the document. At most 250 values are allowed per document with a length limit of 1000 characters. Otherwise, an INVALID_ARGUMENT error is returned.
This tag can be used for filtering recommendation results by passing the
tag as part of the
For example: |
hash_tags
|
String - optional - repeated Hashtags for the document. At most 100 values are allowed per document, with a length limit of 5000 characters.
For example: |
content_rating
|
String - optional - repeated The content rating, used for content advisory systems and content filtering based on the audience. At most 100 values are allowed per document with a length limit of 128 characters.
This tag can be used for filtering recommendation results by passing the
tag as part of the
For example: |
The following table defines hierarchical key media properties.
| Field name | Notes |
|---|---|
persons
|
Object - optional - repeated Root key property for encapsulating the person-related properties.
For example:
|
persons.name
|
String - required Name of the person. |
persons.role
|
String - required The role of the person in the media. Supported values: director, actor, player, team, league, editor, author, character, contributor, creator, editor, funder, producer, provider, publisher, sponsor, translator, music-by, channel, custom-role.
If none of the supported values are applied to |
persons.custom_role
|
String - optional
|
persons.rank
|
Int - optional
Used for role ranking. For example, for first actor,
|
persons.uri
|
String - optional URI of the person. |
organizations
|
Object - optional - repeated
Root key property for encapsulating the
For example:
|
organizations.name
|
String - required Name of the organization. |
organizations.role
|
String - required The role of the organization in the media. Supported values: director, actor, player, team, league, editor, author, character, contributor, creator, editor, funder, producer, provider, publisher, sponsor, translator, music-by, channel, custom-role.
If none of the supported values are applied to |
organizations.custom_role
|
String - optional
|
organizations.rank
|
String - optional
Used for role ranking. For example, for first publisher:
|
organizations.uri
|
String - optional URI of the organization. |
aggregate_ratings
|
Object - optional - repeated
Root key property for encapsulating the
|
aggregate_ratings.rating_source
|
String - required
The source for rating. For example, |
aggregate_ratings.rating_score
|
Double - optional The aggregated rating. The rating should be normalized to the [1, 5] range. |
aggregate_ratings.rating_count
|
Int - optional The number of individual reviews. Should be a non-negative value. |
Document levels
Document levels determine the hierarchy in your datastore. Typically, you should have a single-level datastore or two-level datastore. Only two layers are supported.
For example, you can have a single-level datastore where each document is an individual item. Alternatively, you might choose a two-level datastore that contains both groups of items and individual items.
Document-level types
There are two document-level types:
Parent items are what Discovery for Media returns in recommendations. Parents can be individual items or groups of similar items.
Child items are versions of a group's parent document. Children can only be individual items. For example, if the parent document is "Example TV Show", children could be "Episode 1" and "Episode 2".
About datastore hierarchy
When planning your datastore hierarchy, decide if your datastore should contain only parents or parents and children. The key point to remember is that recommendations only return parent items.
For example, a parent-only datastore might work well for audiobooks, where a recommendations panel returns a selection of individual audiobooks. However, a parent-only datastore for TV shows might return several out-of-order episodes of that show in the recommendation panel.
The TV show datastore could work better with both parents and children, with episodes as children and a parent representing a TV show as a group representing all those episodes. This two-level datastore allows the recommendation panel to show a range of similar TV shows. The end-user can click a particular show to select an episode to watch.
If you determine that your datastore should have both parents and children, that
is, groups and singular items, but you only have singular items now, you need to
create parents for the groups. The minimum information that you need to provide
for a parent is id, title, and categories. For more information, see the
section Document fields.
Datastore import
If your datastore is in Cloud Storage or BigQuery or some other storage, you can do a bulk data import.
For detailed information about how to upload a datastore, see Import datastore information.
Branches
Although Discovery for Media has datastore branching, Media Recommendations does not support multiple datastore branches.
When using Media Recommendations, do not use multiple datastore branches.
Always use branch 0.
BigQuery schema for Discovery for Media
When importing a datastore from BigQuery, use the Discovery for Media schema below to create a BigQuery table with the correct format and load it with your documents data. Then, import the documents.
[ { "name": "id", "mode": "REQUIRED", "type": "STRING", "fields": [] }, { "name": "schemaId", "mode": "REQUIRED", "type": "STRING", "fields": [] }, { "name": "parentDocumentId", "mode": "NULLABLE", "type": "STRING", "fields": [] }, { "name": "jsonData", "mode": "NULLABLE", "type": "STRING", "fields": [] } ]