BigQuery Data Transfer Service for YouTube

Overview

The BigQuery Data Transfer Service for YouTube allows you to automatically schedule and manage recurring load jobs for YouTube Channel and Content Owner reporting data.

The BigQuery Data Transfer Service automatically loads the following YouTube reports into BigQuery:

Channel Reports:

  • Video reports provide statistics for all user activity related to a channel's videos. For example, the number of times that your videos are viewed.
  • Playlist reports provide statistics related to video views that occur via a playlist.

Content Owner Reports:

  • Video reports provide statistics for all user activity that is related to a channel's videos or a content owner's videos. For example, the number of times that your videos are viewed.
  • Playlist reports provide statistics that are related to video views that occur via a playlist.
  • Ad rate reports provide impression-based metrics for ads that ran during video playbacks. Each video playback can yield multiple impressions.
  • Estimated revenue reports provide the total estimated revenue for videos from Google-sold advertising sources and from non-advertising sources. Estimated revenue reports also contain some ad performance metrics.
  • Asset reports provide user activity metrics that are related to videos that are linked to a content owner's assets. A video is included in a content owner's report if the content owner has claimed that video as a match to one of the content owner's assets. The video could have been uploaded by the content owner or by another YouTube user.

If you currently use the YouTube Reporting API and have existing reporting jobs, the BigQuery Data Transfer Service additionally loads your report data. If you do not have existing reporting jobs, setting up the transfer automatically enables YouTube reporting.

The BigQuery Data Transfer Service automatically loads data for the YouTube system managed reports.

By default, automated YouTube data transfers adhere to the following schedule:

  • Data transfers for YouTube channel reporting - Daily at approximately 10:00 UTC
  • Data transfers for YouTube Content Owner reporting - Daily at approximately 09:00 UTC

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Projects page

  3. Enable billing for your project.

    Enable billing

  4. BigQuery is automatically enabled in new projects. To activate BigQuery in a pre-existing project, Enable the BigQuery API.

    Enable the API

  5. Create a BigQuery dataset to store the YouTube data.
  6. The user creating your transfers should be granted the bigquery.admin role for the transfers to succeed. You can apply roles using the IAM & Admin console.
  7. IAM & Admin

Before using the BigQuery Transfer Service, you must complete the enrollment form. To complete the form, you need the project number for the project containing your transfers.

  1. Open the Cloud Platform Console.
  2. Cloud Platform Console
  3. If your transfer project isn't selected, at the top of the window, click the project name.
  4. In the Select project dialog, select your project and then click Open.
  5. On the Dashboard tab, in the Project Info box, note your project number (below your project ID). The project number is a string of numerals.
  6. Project number
  7. Complete the BigQuery Data Transfer Service enrollment form.

The enrollment request must be completed by the BigQuery team before you can use the service. Requests are generally completed within 1 business day but may take up to 3 business days depending on the volume of requests. Please contact your Cloud Sales Representative if you have questions about the status of your enrollment request.

Set up a YouTube data transfer

You can set up two types of YouTube data transfers:

  • YouTube channel reporting
  • YouTube Content Owner reporting

Create a transfer for YouTube Channel reporting

To set up a YouTube data transfer for Channel reporting, you must own a YouTube Channel. Setting up a data transfer for YouTube Channel reporting requires your channel page ID. The page ID used in the data transfer is not the YouTube channel ID in your channel URL.

To retrieve your channel page ID:

  1. Open Google Plus.
  2. At the top right of the window, click the account icon, and choose the account associated with your YouTube channel.
  3. Retrieve the page id from the end of the URL. For example, if the URL is https://plus.google.com/b/123123123/, the page id is 123123123.

To create a YouTube data transfer for YouTube Channel reporting:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. Click Enable to grant permissions to the BigQuery Data Transfer Service. The first time you enable transfers in a project, you are required to grant permissions to the Data Transfer Service. After permissions are granted, you are not prompted again.

    Enable transfers

  4. Click Add Transfer.

  5. On the New Transfer page:

    • For Source, choose YouTube Channel Reports.
    • For Destination, choose the appropriate dataset.
    • For Display Name, enter a name for the transfer such as My Transfer.
    • For Channel Page ID, enter your channel page ID.
    • For Table Suffix, enter a suffix such as MT. This suffix is appended to the job ID to create the table name, for example [job_id]_[suffix]. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

      Channel transfer

  6. Click Add.

  7. When prompted, click Allow to give the BigQuery Transfer Service permission to manage your YouTube reporting data. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

    Allow transfer

    Once the transfer is added, you can edit the transfer, delete the transfer, update the credentials, schedule a backfill, or view the transfer properties such as the next scheduled transfer in the BigQuery web UI.

    Transfer properties

    As your scheduled transfers are run, the BigQuery web UI displays the run history, including successful transfers and transfers that fail. Scheduled runs more than 90 days old are automatically deleted from the run history.

    Run history

If you are having issues setting up your transfer, see Troubleshooting BigQuery Data Transfer Service Transfer Setup.

Create a transfer for YouTube Content Owner reporting

To set up a YouTube data transfer for Content Owner reporting, you must be a Content Manager or a Content Owner. A Content Owner is an umbrella account that owns one or more YouTube channels and the videos on those channels. A Content Manager is a person who has been granted rights to administer the content for a Content Owner.

To retrieve your content owner ID: When you create a YouTube Content Owner reporting transfer, your Content Owner ID is required. When you log in to YouTube as a Content Owner or Manager, your ID appears in the URL after o=. For example, if the URL is https://youtube.com/dashboard?0=AbCDE_8FghIjK, the Content Owner ID is AbCDE_8FghIjK. For more information on your Content Manager account, see: Configure Content Manager account settings.

To create a data transfer for YouTube Content Owner reporting:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. Click Enable to grant permissions to the BigQuery Data Transfer Service. The first time you enable transfers in a project, you are required to grant permissions to the Data Transfer Service. After permissions are granted, you are not prompted again.

    Enable transfers

  4. Click Add Transfer.

  5. On the New Transfer page:

    • For Source, choose YouTube Content Owner Reports.
    • For Destination, choose the appropriate dataset.
    • For Display Name, enter a name for the transfer such as My Transfer. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
    • For Content Owner ID, enter your Content Owner ID.
    • For Table Suffix, enter a suffix such as MT. This suffix is appended to the job ID to create the table name, for example [job_id]_[suffix]. The suffix is used to prevent separate transfers from writing to the same tables. The table suffix must be unique across all transfers that load data into the same dataset, and the suffix should be short to minimize the length of the resulting table name.

      Channel transfer

  6. Click Add.

  7. When prompted, click Allow to give the BigQuery Transfer Service permission to view your YouTube reporting data and to access and manage the data in BigQuery. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

    Allow transfer

    Once the transfer is added, you can edit the transfer, delete the transfer, update the credentials, schedule a backfill, or view the transfer properties such as the next scheduled transfer in the BigQuery web UI.

    Transfer properties

    As your scheduled transfers are run, the BigQuery web UI displays the run history, including successful transfers and transfers that fail. Scheduled runs more than 90 days old are automatically deleted from the run history.

    Run history

Set up a backfill

To transfer historical YouTube data to BigQuery, you must request a backfill. The BigQuery Data Transfer Service supports backfilling up to 180 days at a time. When backfilling large date ranges, break your backfill requests into 180 day chunks and wait for the previous backfill request to finish before creating another one.

To set up a backfill:

  1. Go to the BigQuery web UI.
  2. BigQuery web UI
  3. Click Transfers.
  4. On the Transfer page, click the appropriate transfer.
  5. Click Schedule Backfill.
  6. Schedule backfill
  7. In the Schedule Run dialog:
    • For Data Start Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • For Data End Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • Click Ok.
    • Schedule backfill

Send feedback about...