This is the unified documentation for Retail API. This includes Recommendations AI, Retail Search, and the unified Retail console (which is applicable to both Recommendations AI and Retail Search users). To use the new console or Retail Search while they are in the restricted GA phase, submit a form here to contact Cloud sales. If you are using the v1beta version of Recommendations AI, migrate to the GA version: Migrating to the Retail API from beta.

To see documentation for only Recommendations AI and the Recommendations AI-only console, go to the How-to guides for Recommendations AI and the API reference documentation for Recommendations AI.

Importing historical user events

This page describes how to import user event data from past events in bulk. Retail API models require user event data for training.

After you've set up real-time event recording, it can take a considerable amount of time to record sufficient user event data to train your models. You can accelerate initial model training by importing user event data from past events in bulk. Before doing so, review the best practices for recording user events and the Before you begin section on this page.

The import procedures on this page apply to both Recommendations AI and Retail Search. Once you import data to the Retail API, both services are able to use those events, so you don't need to import the same data twice if you use both services.

You can:

Before you begin

To avoid import errors and ensure the Retail API has sufficient data to generate good results, review the following information before importing your user events.

Event import considerations

This section describes the methods can be used for batch importing of your historical user events, when you might use each method, and some of their limitations.

Cloud Storage Description Import data in a JSON format from files loaded in a Cloud Storage bucket. Each file must be 2 GB or smaller, and up to 100 files at a time can be imported. The import can be done using the Cloud Console or cURL. Uses the Product JSON data format, which allows custom attributes.
When to use If you need higher volumes of data to be loaded in a single step.
Limitations If your data is in Google Analytics or Merchant Center, that data can only be exported to BigQuery and requires the extra step of then importing it to Cloud Storage.
BigQuery Description Import data from a previously loaded BigQuery table that uses the Retail schema. Can be performed using Cloud Console or cURL.
When to use If you are also using analytics or preprocessing event data before importing it.
Limitations Requires the extra step of creating a BigQuery table that maps to the Retail schema. If you have a high volume of user events, also consider that BigQuery is a higher cost resource than Cloud Storage.
BigQuery with Google Analytics 360 Description Import pre-existing data from Google Analytics into Retail.
When to use If you have Google Analytics and track conversions for recommendations or searches. No additional schema mapping is required.
Limitations Only a subset of attributes is available, so some advanced Retail features cannot be used. Tracking impressions in Google Analytics is required if you plan to use Retail Search.
Inline import Description Import using a call to the userEvents.import method.
When to use If you want to have the increased privacy of having all authentication occur on the backend and are capable of performing a backend import.
Limitations Usually more complicated than a web import.

Importing user events from Cloud Storage

Import user events from Cloud Storage using the Cloud Console or the userEvents.import method.

Console

  1. Go to the Retail Data page in the Google Cloud Console.

    Go to the Data page

  2. Click Import to open the Import panel.

  3. Choose User events.

  4. Select the branch you will upload your events to.

  5. Select Cloud Storage as the data source.

  6. Enter the Cloud Storage location of your data.

  7. (Optional) Enter the location of a Cloud Storage bucket in your project as a temporary location for your data.

    If not specified, a default location is used. If specified, the BigQuery and Cloud Storage bucket have to be in the same region.

  8. (Optional) Choose whether to schedule a recurring upload of your user events.

    This option is only available if you have successfully imported user events to the Retail API at least once before.

  9. Click Import.

cURL

Use the userEvents.import method to import your user events.

  1. Create a data file for the input parameters for the import. Use the GcsSource object to point to your Cloud Storage bucket.

    You can provide multiple files, or just one.

    • INPUT_FILE: A file or files in Cloud Storage containing your user event data. See User events for examples of each user event type format. Make sure each user event is on its own single line, with no line breaks.
    • ERROR_DIRECTORY: A Cloud Storage directory for error information about the import.

    The input file fields must be in the format gs://<bucket>/<path-to-file>/. The error directory must be in the format gs://<bucket>/<folder>/. If the error directory does not exist, the Retail API creates it. The bucket must already exist.

    {
    "inputConfig":{
     "gcsSource": {
       "inputUris": ["INPUT_FILE_1", "INPUT_FILE_2"],
      },
      "errorsConfig":{"gcsPrefix":"ERROR_DIRECTORY"}
    }
    
  2. Import your catalog information to the Retail API by making a POST request to the userEvents:import REST method, providing the name of the data file.

    export GOOGLE_APPLICATION_CREDENTIALS=/tmp/my-key.json
    
    curl -X POST \
         -v \
         -H "Content-Type: application/json; charset=utf-8" \
         -H "Authorization: Bearer "$(gcloud auth application-default print-access-token)"" \
         --data @./DATA_FILE.json \
      "https://retail.googleapis.com/v2/projects/[PROJECT_ID]/locations/global/catalogs/default_catalog/userEvents:import"
      }
    }'
    

Importing user events from BigQuery

Import user events from BigQuery using the Cloud Console or the userEvents.import method.

Console

  1. Go to the Retail Data page in the Google Cloud Console.

    Go to the Data page

  2. Click Import to open the Import panel.

  3. Choose User events.

  4. Select the branch you will upload your events to.

  5. Select BigQuery as the data source.

  6. Enter the BigQuery table where your data is located.

  7. (Optional) Enter the location of a Cloud Storage bucket in your project as a temporary location for your data.

    If not specified, a default location is used. If specified, the BigQuery and Cloud Storage bucket have to be in the same region.

  8. (Optional) Choose whether to schedule a recurring upload of your user events.

    This option is only available if you have successfully imported user events to the Retail API at least once before.

  9. Click Import.

cURL

Use the userEvents.import method to import your user events.

When importing your events, use the value user_event for dataSchema.

export GOOGLE_APPLICATION_CREDENTIALS=/tmp/my-key.json

curl \
  -v \
  -X POST \
  -H "Content-Type: application/json; charset=utf-8" \
  -H "Authorization: Bearer "$(gcloud auth application-default print-access-token)"" \
  "https://retail.googleapis.com/v2/projects/[PROJECT_ID]/locations/global/catalogs/default_catalog/userEvents:import" \
  --data '{
    "inputConfig": {
      "bigQuerySource": {
        "datasetId": "DATASET_ID",
        "tableId": "TABLE_ID",
        "dataSchema": "user_event"
    }
  }
}'

If your BigQuery dataset belongs to a different project from your Retail API project, then follow the instructions in Setting up access to your BigQuery dataset to give your Retail service account a BigQuery Data Editor role for your BigQuery project. Modify the import request to specify the BigQuery project ID:

"bigQuerySource": {
     "projectId": "BQ_PROJECT_ID",
   }

Importing Google Analytics 360 user events with BigQuery

You can import Google Analytics 360 user events if you have integrated Google Analytics 360 with BigQuery and use Enhanced Ecommerce.

The following procedures assume you are familiar with using BigQuery and Google Analytics 360.

Before you begin

Check your data source

  1. Make sure the user event data you will import is correctly formatted in a BigQuery table you have access to.

    The table should have the format project_id:ga360_export_dataset.ga_sessions_YYYYMMDD.

    See the Google Analytics documentation for more about the table format.

  2. In the BigQuery Google Cloud Console, select the table from the Explorer panel to preview the table.

    Check that:

    1. The clientId column has a valid value (for example, 123456789.123456789).

      Note that this value is different from the full _ga cookie value (which has a format such as GA1.3.123456789.123456789).

    2. The hits.transaction.currencyCode column has a valid currency code.

    3. If you plan to import search events, check that either a hits.page.searchKeyword or hits.customVariable.searchQuery column is present.

      Importing search events is supported, but they don't map from Analytics 360 in the same way that other event types do because Analytics 360 doesn't natively support the search event type. During import, Retail API constructs search events from Analytics 360 by combining information from the search query and if present, the product impression.

      The search query is derived from either hits.page.searchKeyword, or from hits.customVariables.customVarValue if hits.customVariables.customVarName is searchQuery. The product impression is taken from hits.product if hits.product.isImpressions is TRUE.

  3. Check the consistency of item IDs between the uploaded catalog and the Analytics 360 user event table.

    Using any product ID from the hits.product.productSKU column in the BigQuery table preview, use the product.get method to make sure the same product is in your uploaded catalog.

    export GOOGLE_APPLICATION_CREDENTIALS=/tmp/my-key.json
    
       curl \
         -v \
         -X GET \
         -H "Content-Type: application/json; charset=utf-8" \
         -H "Authorization: Bearer "$(gcloud auth application-default print-access-token)"" \
         "https://retail.googleapis.com/v2/projects/[PROJECT_NUMBER]/locations/global/catalogs/default_catalog/branches/default_branch/products/PRODUCT_ID"
    

Import your Google Analytics 360 events

Import your user events by including the data for the events in your call to the userEvents.import method.

For dataSchema, use the value user_event_ga360.

curl

export GOOGLE_APPLICATION_CREDENTIALS=/tmp/my-key.json
curl \
  -v \
  -X POST \
  -H "Content-Type: application/json; charset=utf-8" \
  -H "Authorization: Bearer "$(gcloud auth application-default print-access-token)"" \
  "https://retail.googleapis.com/v2/projects/[PROJECT_ID]/locations/global/catalogs/default_catalog/userEvents:import" \
  --data '{
    "inputConfig": {
      "bigQuerySource": {
        "datasetId": "some_ga360_export_dataset",
        "tableId": "ga_sessions_YYYYMMDD",
        "dataSchema": "user_event_ga360"
    }
  }
}'

Java

public static String importUserEventsFromBigQuerySource()
    throws IOException, InterruptedException, ExecutionException {
  UserEventServiceClient userEventsClient = getUserEventServiceClient();

  BigQuerySource bigQuerySource = BigQuerySource.newBuilder()
      .setProjectId(PROJECT_ID)
      .setDatasetId(DATASET_ID)
      .setTableId(TABLE_ID)
      .setDataSchema("user_event")
      .build();

  UserEventInputConfig inputConfig = UserEventInputConfig.newBuilder()
      .setBigQuerySource(bigQuerySource)
      .build();

  ImportUserEventsRequest importRequest = ImportUserEventsRequest.newBuilder()
      .setParent(DEFAULT_CATALOG_NAME)
      .setInputConfig(inputConfig)
      .build();

  String operationName = userEventsClient
      .importUserEventsAsync(importRequest).getName();

  userEventsClient.shutdownNow();
  userEventsClient.awaitTermination(2, TimeUnit.SECONDS);

  return operationName;
}

If your BigQuery dataset belongs to a different project from your Retail project, then follow the instructions in Setting up access to your BigQuery dataset to give your Retail service account a BigQuery Data Editor role for your BigQuery project. Modify the import request to specify the the BigQuery project ID:

"bigQuerySource": {
     "projectId": "GA360_BQ_PROJECT_ID",
   }

Importing user events inline

curl

You import user events inline by including the data for the events in your call to the userEvents.import method.

The easiest way to do this is to put your user event data into a JSON file and provide the file to cURL.

For the formats of the user event types, see User events.

  1. Create the JSON file:

    {
    "inputConfig": {
      "userEventInlineSource": {
          "userEvents": [
            {
              <userEvent1>>
            },
            {
              <userEvent2>
            },
            ....
          ]
        }
      }
    }
    
  2. Call the POST method:

    curl -X POST \
         -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
         -H "Content-Type: application/json; charset=utf-8" \
         --data @./data.json \
      "https://retail.googleapis.com/v2/projects/[PROJECT_ID]/locations/global/catalogs/default_catalog/userEvents:import"
    

Java

public static String importUserEventsFromInlineSource(
    List<UserEvent> userEventsToImport)
    throws IOException, InterruptedException, ExecutionException {
  UserEventServiceClient userEventsClient = getUserEventServiceClient();

  UserEventInlineSource inlineSource = UserEventInlineSource.newBuilder()
      .addAllUserEvents(userEventsToImport)
      .build();

  UserEventInputConfig inputConfig = UserEventInputConfig.newBuilder()
      .setUserEventInlineSource(inlineSource)
      .build();

  ImportUserEventsRequest importRequest = ImportUserEventsRequest.newBuilder()
      .setParent(DEFAULT_CATALOG_NAME)
      .setInputConfig(inputConfig)
      .build();

  String operationName = userEventsClient
      .importUserEventsAsync(importRequest).getName();

  userEventsClient.shutdownNow();
  userEventsClient.awaitTermination(2, TimeUnit.SECONDS);

  return operationName;
}

Historical catalog data

You can also import the historical catalog data that appears in your historical user events. This historical catalog data can be helpful because the Retail API can utilize past product information to enrich the user events, which can improve model accuracy.

For more details, see Import historical catalog data.

What's next