Check data quality for media recommendations

This page describes how to find out whether various metrics for your media data meet their requirement thresholds.

About checking media data quality

Because recent user events are so important for media recommendations, you must regularly check the quality of your ingested data and user events. You can do this by reviewing the Optimization tab for your media recommendations app to determine what improvements you can make to your data in order to optimize for better quality recommendations.

If a metric's threshold isn't met, then the metric has a warning status. Then, you need to review the metric and its description to determine what action you should take to improve your media quality.

All models and objectives need to pass the General quality metric thresholds. Some model and objectives have additional App-specific quality metrics and thresholds. The general quality metrics are the same for all apps using the same data store, but app-specific quality metrics vary according to the app's model and objectives.

For information about the recommendation models and objectives, see About media app recommendations types.

Check data quality

Console

To check the quality of your media recommendations data, follow these steps:

  1. In the Google Cloud console, go to the Agent Builder page.

    Agent Builder

  2. Click the name of the media recommendations app that you want check data quality for.

  3. In the navigation menu, click Data quality and click the Optimization tab. This page shows the status of various metrics for the data associated with your app.

  4. Review the General quality and the App-specific quality statuses at the top of the page. The summary status at the top of the page shows as a warning if one or more metrics has exceeded its threshold.

    The two metrics tables (General quality and the App-specific quality) list the individual metrics.

    optimization tab shows one warning metric for the general quality doc_with_same_title_percentage

  5. In the metrics tables, click View details for more information about any metrics in the warning state.

  6. Optional: If you want to see the threshold for a compliant metric, click View Details. Thresholds for compliant metrics are not shown in the metrics table.

REST

Use the requirements:checkRequirement method to check the quality of your media recommendations data, as shown.

To check the quality from the command line, follow these steps:

  1. Find your data store ID. If you already have your data store ID, skip to the next step.

    1. In the Google Cloud console, go to the Agent Builder page and in the navigation menu, click Data Stores.

      Go to the Data Stores page

    2. Click the name of your data store.

    3. On the Data page for your data store, get the data store ID.

  2. Run the following curl command to learn if your media recommendations meets the thresholds for the general metrics:

    curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "X-GFE-SSL: yes" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/requirements:checkRequirement" \
-d '{
      "location": "projects/PROJECT_ID/locations/global",
      "requirementType": "discoveryengine.googleapis.com/media_recs/general/all/warning",
      "resources": [
        {
          "labels": {
            "branch_id": "0",
            "collection_id": "default_collection",
            "datastore_id": "DATA_STORE_ID",
            "location_id": "global",
            "project_number": "PROJECT_ID"
          },
          "type": "discoveryengine.googleapis.com/Branch"
        },
        {
          "labels": {
            "collection_id": "default_collection",
            "datastore_id": "DATA_STORE_ID",
            "location_id": "global",
            "project_number": "PROJECT_ID"
          },
          "type": "discoveryengine.googleapis.com/DataStore"
        }
      ]
    }'
    • PROJECT_ID: the ID of your Google Cloud project.
    • DATA_STORE_ID: the ID of the Vertex AI Search data store.

  3. Review the output:

    1. Look for the value of result:

      • If the value is SUCCESS, then your data passes the general requirements; continue to step 4.

      • If the value is WARNING, continue to step b.

      • If you don't see result in the output, there are a couple possible reasons:

        • The PROJECT_ID or DATA_STORE_ID in the request is incorrect.

        • Some metric values are unavailable. Try again in 6 hours or reach out to a customer engineer for help.

    2. Look for the expression (requirement.Condition.Expression): If this expression evaluates to false, then there is a problem with your data.

      The value of the metrics are in the requirementCondition.metricResults.value field. The warning threshold values are in the thresholdBindings.thresholdValues fields. The description fields can help you understand the purpose of the metric.

      For example, the value of doc_with_same_title_percentage is 30.47 and the warning threshold for doc_with_same_title_percentage_threshold is 1. There is a data problem that so many of the titles in the data store are the same, and this needs to be investigated.

  4. If the model and objective combination used for your recommendations app appears in this table, then you also need to call the check requirement method, updated with the values for your model and objective:

    Model Objective MODEL_OBJ
    Others You May Like Conversion rate oyml/cvr
    Recommended for You Conversion rate rfy/cvr
    More Like This Conversion rate mlt/cvr
    Most Popular Conversion rate mp/cvr
    Others You May Like Watch duration per session oyml/wdps
    Recommended for You Watch duration per session rfy/wdps
    More Like This Watch duration per session mlt/wdps 

    curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "X-GFE-SSL: yes" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/requirements:checkRequirement" \
-d '{
      "location": "projects/PROJECT_ID/locations/global",
      "requirementType": "discoveryengine.googleapis.com/media_recs/MODEL_OBJ/warning",
      "resources": [
        {
          "labels": {
            "branch_id": "0",
            "collection_id": "default_collection",
            "datastore_id": "DATA_STORE_ID",
            "location_id": "global",
            "project_number": "PROJECT_ID"
          },
          "type": "discoveryengine.googleapis.com/Branch"
        },
        {
          "labels": {
            "collection_id": "default_collection",
            "datastore_id": "DATA_STORE_ID",
            "location_id": "global",
            "project_number": "PROJECT_ID"
          },
          "type": "discoveryengine.googleapis.com/DataStore"
        }
      ]
    }'
    • PROJECT_ID: the ID of your Google Cloud project.
    • DATA_STORE_ID: the ID of the Vertex AI Search data store.
    • MODEL_OBJ: See the preceding table to choose the correct value for your recommendations app.

  5. Review the output:

    1. Look for the value of result:

      • If the value is SUCCESS, then your data is good enough.

      • If the value is WARNING, continue to step b.

      • If you don't see result in the output, there are a couple possible reasons:

        • The PROJECT_ID or DATA_STORE_ID in the request is incorrect.

        • Some metric values are unavailable. Try again in 6 hours or reach out to a customer engineer for help.

    2. Look the expression (requirement.Condition.Expression). If this expression evaluates to false, then there is a problem with your data.

      The value of the metrics can be found in the requirementCondition.metricResults.value field, and the warning threshold values, in the thresholdBindings.thresholdValues fields. The description fields can help you understand the purpose of the metric.