BigQuery Data Transfer Service for YouTube

Overview

The BigQuery Data Transfer Service for YouTube allows you to automatically schedule and manage recurring load jobs for YouTube Channel and Content Owner reporting data.

The Google BigQuery Data Transfer Service automatically loads these monthly ad performance report data and channel report data into your BigQuery project.

If you currently use the YouTube Reporting API and have existing reporting jobs, the BigQuery Data Transfer Service additionally loads this report data. If you do not have existing reporting jobs, only the automatically generated reporting data is loaded into BigQuery.

The BigQuery Data Transfer Service automatically loads data for the YouTube system managed reports.

By default, automated YouTube data transfers adhere to the following schedule:

  • Data transfers for YouTube channel reporting - Daily at approximately 10:00 UTC
  • Data transfers for YouTube Content Owner reporting - Daily at approximately 09:00 UTC

If you create new reporting jobs using the YouTube Reporting API, the BigQuery Data Transfer Service automatically loads the reporting data into BigQuery within 48 hours. For more information on creating new reporting jobs using the YouTube Reporting API, see: jobs.create method.

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Projects page

  3. Enable billing for your project.

    Enable billing

  4. BigQuery is automatically enabled in new projects. To activate BigQuery in a pre-existing project, Enable the BigQuery API.

    Enable the API

  5. Create a BigQuery dataset to store the YouTube data.
  6. The user creating your transfers should be a project EDITOR or OWNER for the transfers to succeed. You can confirm project-level permissions in the IAM & Admin console.
  7. IAM & Admin

Before using the BigQuery Transfer Service, you must complete the enrollment form. To complete the form, you need the project number for the project containing your transfers.

  1. Open the Cloud Platform Console.
  2. Cloud Platform Console
  3. If your transfer project isn't selected, at the top of the window, click the project name.
  4. In the Select project dialog, select your project and then click Open.
  5. On the Dashboard tab, in the Project Info box, note your project number (below your project ID). The project number is a string of numerals.
  6. Project number
  7. Complete the BigQuery Data Transfer Service enrollment form.

The enrollment request must be completed by the BigQuery team before you can use the service. Requests are generally completed within 1 business day but may take up to 3 business days depending on the volume of requests. Please contact your Cloud Sales Representative if you have questions about the status of your enrollment request.

Set up a YouTube data transfer

To set up a YouTube data transfer, you:

  • Grant project-level permissions to the BigQuery Data Transfer Service.
  • Create the data transfer.

Grant project-level permissions to the BigQuery Data Transfer Service

To allow the BigQuery Data Transfer Service to run load jobs for YouTube, you must grant the BigQuery User and Bigquery Data Editor IAM roles to the bigquery-connectors@googlegroups.com group. You must be a Project Owner to grant IAM roles to users and groups. For more information on granting IAM roles at the project level, see Granting, Changing, and Revoking Access to Project Members.

To assign project-level permissions to the BigQuery Transfer Service:

Web UI

  1. Open the IAM page in the Google Cloud Platform Console.

    Open the IAM page

  2. Click Select a project.

  3. Select your project and click Open.

  4. On the IAM page, click Add.

  5. In the Add members dialog:

    • For Members type the group name: bigquery-connectors@googlegroups.com.
    • For Roles, click Select a role, select BigQuery > BigQuery User and BigQuery Data Editor.
    • Click Add.

Command-line

  1. To add a single binding to the project's IAM policy, type the following command. Replace [PROJECT_ID] with your project ID.

    gcloud projects add-iam-policy-binding [PROJECT_ID] --member group:bigquery-connectors@googlegroups.com --role roles/bigquery.user
    
  2. The command outputs the updated policy:

    bindings:
    - members:
      - group:bigquery-connectors@googlegroups.com
        role: roles/bigquery.user
    
  3. Type the command again to add the bigquery.dataEditor role. Replace [PROJECT_ID] with your project ID.

    gcloud projects add-iam-policy-binding [PROJECT_ID] --member group:bigquery-connectors@googlegroups.com --role roles/bigquery.dataEditor
    
  4. The command outputs the updated policy:

    bindings:
    - members:
      - group:bigquery-connectors@googlegroups.com
        role: roles/bigquery.dataEditor
    

For more information on BigQuery access controls, see: Access control.

Create a transfer for YouTube Channel reporting

To set up a YouTube data transfer for Channel reporting, you must own a YouTube Channel. Setting up a data transfer for YouTube Channel reporting requires your channel page ID. The page ID used in the data transfer is not the YouTube channel ID in your channel URL.

To retrieve your channel page ID:

  1. Open Google Plus.
  2. At the top right of the window, click the account icon, and choose the account associated with your YouTube channel.
  3. Retrieve the page id from the end of the URL. For example, if the URL is https://plus.google.com/b/123123123/, the page id is 123123123.

To create a YouTube data transfer for YouTube Channel reporting:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. If prompted, click Enable to grant permissions to the BigQuery Data Transfer Service.

  4. Click Add Transfer.

  5. On the New Transfer page:

    • For Source, choose YouTube Channel Reports.
    • For Destination, choose the appropriate dataset.
    • For Display Name, enter a name for the transfer such as My Transfer.
    • For Channel Page ID, enter your channel page ID.
    • For Table Suffix, enter a suffix such as MT. This suffix is appended to the job ID to create the table name, for example [job_id]_[suffix]. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

      Channel transfer

  6. Click Add.

  7. When prompted, click Allow to give the BigQuery Transfer Service permission to manage your YouTube reporting data. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

    Allow transfer

    Once the transfer is added, you can edit the transfer, delete the transfer, update the credentials, schedule a backfill, or view the transfer properties such as the next scheduled transfer in the BigQuery web UI.

    Transfer properties

    As your scheduled transfers are run, the BigQuery web UI displays the run history, including successful transfers and transfers that fail. Scheduled runs more than 90 days old are automatically deleted from the run history.

    Run history

Create a transfer for YouTube Content Owner reporting

To set up a YouTube data transfer for Content Owner reporting, you must be a Content Manager or a Content Owner. A Content Owner is an umbrella account that owns one or more YouTube channels and the videos on those channels. A Content Manager is a person who has been granted rights to administer the content for a Content Owner.

To retrieve your content owner ID: When you create a YouTube Content Owner reporting transfer, your Content Owner ID is required. When you log in to YouTube as a Content Owner or Manager, your ID appears in the URL after o=. For example, if the URL is https://youtube.com/dashboard?0=AbCDE_8FghIjK, the Content Owner ID is AbCDE_8FghIjK. For more information on your Content Manager account, see: Configure Content Manager account settings.

To create a data transfer for YouTube Content Owner reporting:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. If prompted, click Enable to grant permissions to the BigQuery Data Transfer Service.

  4. Click Add Transfer.

  5. On the New Transfer page:

    • For Source, choose YouTube Content Owner Reports.
    • For Destination, choose the appropriate dataset.
    • For Display Name, enter a name for the transfer such as My Transfer. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
    • For Content Owner ID, enter your Content Owner ID.
    • For Table Suffix, enter a suffix such as MT. This suffix is appended to the job ID to create the table name, for example [job_id]_[suffix]. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

      Channel transfer

  6. Click Add.

  7. When prompted, click Allow to give the BigQuery Transfer Service permission to view your YouTube reporting data and to access and manage the data in BigQuery. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

    Allow transfer

    Once the transfer is added, you can edit the transfer, delete the transfer, update the credentials, schedule a backfill, or view the transfer properties such as the next scheduled transfer in the BigQuery web UI.

    Transfer properties

    As your scheduled transfers are run, the BigQuery web UI displays the run history, including successful transfers and transfers that fail. Scheduled runs more than 90 days old are automatically deleted from the run history.

    Run history

Set up a backfill

To transfer historical YouTube data to BigQuery, you must request a backfill. You can schedule YouTube data transfer backfills up to the past 180 days.

To set up a backfill:

  1. Go to the BigQuery web UI.
  2. BigQuery web UI
  3. Click Transfers.
  4. On the Transfer page, click the appropriate transfer.
  5. Click Schedule Backfill.
  6. Schedule backfill
  7. In the Schedule Run dialog:
    • For Data Start Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • For Data End Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • Click Ok.
    • Schedule backfill

Send feedback about...

BigQuery Documentation