Google Play transfers

The BigQuery Data Transfer Service for Google Play allows you to automatically schedule and manage recurring load jobs for Google Play reporting data.

Supported Reports

The BigQuery Data Transfer Service for Google Play currently supports the following monthly reporting options:

Detailed reports

Aggregated reports

For information on how Google Play reports are transformed into BigQuery tables, see Google Play report transformations.

Reporting option	Support
Supported API version	N/A
Schedule	Daily, at the time the transfer is first created (default) You can configure the time of day.
Refresh window	Last 7 days Not configurable
Maximum backfill duration	No limit While Google Play has no known data retention limits, the BigQuery Data Transfer Service has limits on how many days can be requested in a single backfill. For information on backfills, see Set up a backfill.

Before you begin

Before you create a Google Play transfer:

Verify that you have completed all actions required to enable the BigQuery Data Transfer Service.
If you are creating the transfer by using the classic BigQuery web UI, allow pop-ups in your browser from bigquery.cloud.google.com so that you can view the permissions window. You must allow the BigQuery Data Transfer Service permission to manage your transfer.
Create a BigQuery dataset to store the Google Play data.
Find your Cloud Storage bucket:
- In the Google Play console, select any report, such as Reviews. You'll find the ID for your Cloud Storage bucket near the bottom of the page. Your bucket ID begins with gs://. For example, for the Reviews report, you'll see:
```
gs://pubsite_prod_rev_01234567890987654321/reviews
```
  To set up your transfer, you'll copy only the unique ID that comes between gs:// and /reviews:
```
pubsite_prod_rev_01234567890987654321
```
If you intend to setup transfer run notifications for Pub/Sub, you must have pubsub.topics.setIamPolicy permissions. Pub/Sub permissions are not required if you just set up email notifications. For more information, see BigQuery Data Transfer Service run notifications.

Required permissions

BigQuery: Ensure that the person creating the transfer has the following permissions in BigQuery:
- bigquery.transfers.update permissions to create the transfer.
The bigquery.admin predefined Cloud IAM role includes bigquery.transfers.update permissions. For more information on Cloud IAM roles in BigQuery Data Transfer Service, see Access control reference.
Google Play: Ensure that you have the following permissions in Google Play:
- You must have reporting access in the Google Play console.
The Google Cloud team does NOT have the ability to generate or grant access to Google Play files on your behalf. See Contact Google Play support for help accessing Google Play files.

Setting up a Google Play transfer

Setting up a Google Play transfer requires a:

Cloud Storage bucket. Steps for locating your Cloud Storage bucket are described in Before you begin. Your Cloud Storage bucket begins with pubsite_prod_rev. For example: pubsite_prod_rev_01234567890987654321.
Table suffix: A user-friendly name for all data sources loading into the same dataset. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

To set up a Google Play transfer:

Console

Go to the BigQuery web UI in the Cloud Console.

Go to the Cloud Console
Click Transfers.
Click Create Transfer.
On the Create Transfer page:
- In the Source type section, for Source, choose Google Play.
- In the Transfer config name section, for Display name, enter a name for the transfer such as My Transfer. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
- In the Schedule options section, for Schedule, leave the default value (Start now) or click Start at a set time.
  - For Repeats, choose an option for how often to run the transfer.
    - Daily (default)
    - Weekly
    - Monthly
    - Custom
    - On-demand
  - For Start date and run time, enter the date and time to start the transfer. If you choose Start now, this option is disabled.
- In the Destination settings section, for Destination dataset, choose the dataset you created to store your data.
- In the Data source details section:
  - For Cloud Storage bucket, enter the ID for your Cloud Storage bucket.
  - For Table suffix, enter a suffix such as 'MT' (for 'My Transfer').
- (Optional) In the Notification options section:
  - Click the toggle to enable email notifications. When you enable this option, the transfer administrator receives an email notification when a transfer run fails.
  - For Select a Pub/Sub topic, choose your topic name or click Create a topic. This option configures Pub/Sub run notifications for your transfer.
Click Save.

Classic UI

Go to the BigQuery web UI.

Go to the BigQuery web UI
Click Transfers.
Click Add Transfer.
On the New Transfer page:
- For Source, choose Google Play.
- For Display name, enter a name for the transfer such as 'My Transfer'. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
- For Schedule, leave the default value (every 24 hours at transfer creation time) or click Edit and change the time.
- For Destination dataset, choose the appropriate dataset.
- For Cloud Storage bucket, enter your Cloud Storage bucket.
- For Table suffix, enter a suffix such as 'MT' (for 'My Transfer').
- (Optional) Expand the Advanced section and configure run notifications for your transfer.
  - For Pub/Sub topic, enter your topic name, for example, 'projects/myproject/topics/mytopic'.
  - Check Send email notifications to allow email notifications of transfer run failures.
  - Don't check Disabled when setting up a transfer. See Working with transfers to disable existing transfers.
Click Add. Note that by clicking the Add button you are granting Google Play Reports read and write access to the destination BigQuery dataset you selected.
When prompted, click Allow to give the BigQuery Data Transfer Service permission to view your Google Play reporting data and to access and manage the data in BigQuery. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

CLI

Enter the bq mk command and supply the transfer creation flag — --transfer_config. The following flags are also required:

--target_dataset
--display_name
--params
--data_source

bq mk \
--transfer_config \
--project_id=project_id \
--target_dataset=dataset \
--display_name=name \
--params='parameters' \
--data_source=data_source

Where:

project_id is your project ID. If --project_id isn't specified, the default project is used.
dataset is the target dataset for the transfer configuration.
name is the display name for the transfer configuration. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
parameters contains the parameters for the created transfer configuration in JSON format. For example: --params='{"param":"param_value"}'. For Google Play, you must supply the bucket and table_suffix, parameters. bucket is the Cloud Storage bucket that contains your Play report files.
data_source is the data source: play.

For example, the following command creates a Google Play transfer named My Transfer using Cloud Storage bucket pubsite_prod_rev_01234567890987654321 and target dataset mydataset. The transfer is created in the default project:

bq mk \
--transfer_config \
--target_dataset=mydataset \
--display_name='My Transfer' \
--params='{"bucket":"pubsite_prod_rev_01234567890987654321","table_suffix":"MT"}' \
--data_source=play

The first time you run the command, you will receive a message like the following:

[URL omitted] Please copy and paste the above URL into your web browser and follow the instructions to retrieve an authentication code.

Follow the instructions in the message and paste the authentication code on the command line.

API

Use the projects.locations.transferConfigs.create method and supply an instance of the TransferConfig resource.

Troubleshooting Google Play transfer set up

If you are having issues setting up your transfer, see Troubleshooting BigQuery Data Transfer Service transfer setup.

Querying your data

When your data is transferred to BigQuery, the data is written to ingestion-time partitioned tables. For more information, see Introduction to partitioned tables.

If you query your tables directly instead of using the auto-generated views, you must use the _PARTITIONTIME pseudo-column in your query. For more information, see Querying partitioned tables.

What's next

To see how your Google Play reports are transferred to BigQuery, see Google Play report transformations.
For an overview of BigQuery Data Transfer Service, see Introduction to BigQuery Data Transfer Service.
For information on using transfers including getting information about a transfer configuration, listing transfer configurations, and viewing a transfer's run history, see Working with transfers.