BigQuery Data Transfer Service for DoubleClick Bid and Campaign Manager

Overview

The BigQuery Data Transfer Service for DoubleClick allows you to automatically schedule and manage recurring load jobs for DoubleClick reporting data. The BigQuery Data Transfer Service for DoubleClick Bid and Campaign Manager supports the following DoubleClick reporting data:

To schedule a BigQuery data transfer for DoubleClick Bid Manager or DoubleClick Campaign Manager, you must have access to:

By default, BigQuery data transfers for DoubleClick Bid and Campaign Manager adhere to the following schedule:

  • Transfers run every 8 hours.
  • By default, data for the past 2 days is refreshed on a daily basis.

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Projects page

  3. Enable billing for your project.

    Enable billing

  4. BigQuery is automatically enabled in new projects. To activate BigQuery in a pre-existing project, Enable the BigQuery API.

    Enable the API

  5. Create a BigQuery dataset to store the DoubleClick data.
  6. Work with your DoubleClick sales or support contact to setup access to DoubleClick Data Transfer v2.0.
  7. Obtain the Google Cloud Storage bucket URI for your DoubleClick Data Transfer files from your DoubleClick administrator. The bucket name should look like this: dcdt_-dcm_account1234.
  8. Ensure the user setting up your DoubleClick Campaign Manager transfers is a member of the google group that administers the Google Cloud Storage bucket containing your DoubleClick data.
  9. The user creating your transfers should be a project EDITOR or OWNER for the transfers to succeed. You can confirm project-level permissions in the IAM & Admin console.
  10. IAM & Admin

Before using the BigQuery Transfer Service, you must complete the enrollment form. To complete the form, you need the project number for the project containing your transfers.

  1. Open the Cloud Platform Console.
  2. Cloud Platform Console
  3. If your transfer project isn't selected, at the top of the window, click the project name.
  4. In the Select project dialog, select your project and then click Open.
  5. On the Dashboard tab, in the Project Info box, note your project number (below your project ID). The project number is a string of numerals.
  6. Project number
  7. Complete the BigQuery Data Transfer Service enrollment form.

The enrollment request must be completed by the BigQuery team before you can use the service. Requests are generally completed within 1 business day but may take up to 3 business days depending on the volume of requests. Please contact your Cloud Sales Representative if you have questions about the status of your enrollment request.

Set up a BigQuery data transfer for DoubleClick Campaign or Bid Manager

Setting up a BigQuery data transfer for DoubleClick Bid or Campaign Manager requires your DoubleClick Network, Advertiser, or Floodlight ID. Network ID (for DoubleClick Campaign Manager) and Partner ID (for DoubleClick Bid Manager) are synonymous, and are the parent in the hierarchy.

DoubleClick Campaign Manager allows multiple DoubleClick Data Transfer configurations, each containing its own file scope. Each scope consists of core files (Impression, Click, Activity, Rich Media) and match table lookups.

Other file scopes are subsets:

  • Advertiser - DoubleClick Data Transfer files are filtered for a specific Advertiser
  • Floodlight - DoubleClick Data Transfer files are filtered for a specific Floodlight

To retrieve your DoubleClick ID, you can use the Google Cloud Storage web UI to examine the files in your DoubleClick Data Transfer Cloud Storage bucket. The ID is imbedded in the file name. For example, in a file named dcm_account1234_activity_*, the ID is 1234.

To create a BigQuery data transfer for DoubleClick Campaign or Bid Manager:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. If prompted, click Enable to grant permissions to the BigQuery Data Transfer Service.

  4. Click Add Transfer.

  5. On the New Transfer page:

    • For Source, choose DCM Data Transfer.
    • For Destination, choose the appropriate dataset.
    • For Display Name, enter a name for the transfer such as My Transfer. The transfer name can be any value that allows you to easily identify the transfer if you need to modify it later.
    • For GCS Bucket, enter the name of the Cloud Storage bucket that stores your Data Transfer V2.0 files. When you enter the bucket name, do not include gs://.
    • For DoubleClick ID, enter the appropriate ID.

      Channel transfer

  6. Click Add.

  7. When prompted, click Allow to give the BigQuery Data Transfer Service permission to manage your DoubleClick campaigns. You must allow pop-ups from bigquery.cloud.google.com to view the permissions window.

    Allow transfer

    Once the transfer is added, you can edit the transfer, delete the transfer, update the credentials, schedule a backfill, or view the transfer properties such as the next scheduled transfer in the BigQuery web UI.

    Transfer properties

As your scheduled transfers are run, the BigQuery web UI displays the run history, including successful transfers and transfers that fail. Scheduled runs more than 90 days old are automatically deleted from the run history.

To view the run history:

  1. Go to the BigQuery web UI.

    Go to the BigQuery web UI

  2. Click Transfers.

  3. On the Transfers page, click the appropriate transfer in the list.

  4. On the properties page, the run history appears below the transfer properties.

    Run history

If you are having issues setting up your transfer, see Troubleshooting BigQuery Data Transfer Service Transfer Setup.

Set up a backfill

To transfer historical DoubleClick data to BigQuery, you must request a backfill. You can schedule DoubleClick Bid and Campaign Manager backfills up to the earliest file in your Cloud Storage bucket.

To set up a backfill:

  1. Go to the BigQuery web UI.
  2. BigQuery web UI
  3. Click Transfers.
  4. On the Transfer page, click the appropriate transfer.
  5. Click Schedule Backfill.
  6. Schedule backfill
  7. In the Schedule Run dialog:
    • For Data Start Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • For Data End Date, enter or choose the appropriate date in MM/DD/YYYY format.
    • Click Ok.
    • Schedule backfill

DoubleClick sample queries

You can use the following DoubleClick Bid and Campaign Manager sample queries to analyze your transferred data. You can also use the queries in a visualization tool such as Google Cloud Datalab or Google Data Studio 360.

These samples use BigQuery’s support for standard SQL. Use the #standardSQL tag to let BigQuery know you want to use standard SQL. For more information about the #standardSQL prefix, see Setting a query prefix.

In each of the following queries, replace [DATASET] with your dataset name. Replace [DOUBLECLICK_ID] with your DoubleClick ID.

Latest campaigns

The following sample query retrieves the latest campaigns.

Web UI

#standardSQL
SELECT Campaign, Campaign_ID FROM `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
WHERE _DATA_DATE = _LATEST_DATE

Command-line

bq query --use_legacy_sql=false '
SELECT Campaign, Campaign_ID FROM `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
WHERE _DATA_DATE = _LATEST_DATE'

Impressions and distinct users by campaign

The following sample query analyzes the number of impressions and distinct users by campaign over the past 30 days.

Web UI

#standardSQL
# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
SELECT Campaign_ID, DATA_DATE AS Date, count(*) AS count, count(distinct User_ID) AS du
FROM `[DATASET].impression[DOUBLECLICK_ID]` WHERE
  _DATA_DATE BETWEEN [START_DATE] AND [END_DATE]
GROUP BY Campaign_ID, Date

Command-line

# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
bq query --use_legacy_sql=false '
SELECT Campaign_ID, _DATA_DATE AS Date, count(*) AS count, count(distinct User_ID) AS du
FROM `[DATASET].impression_[DOUBLECLICK_ID]` WHERE
  _DATA_DATE BETWEEN [START_DATE] AND [END_DATE]
GROUP BY Campaign_ID, Date'

Latest campaigns ordered by campaign and date

The following sample query analyzes the latest campaigns in the past 30 days, ordered by campaign and date.

Web UI

#standardSQL
# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
SELECT Campaign, Campaign_ID, Date FROM (
SELECT Campaign, Campaign_ID FROM `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
  WHERE _DATA_DATE = _LATEST_DATE
), (
SELECT date AS Date
  FROM `bigquery-public-data.common_us.date_greg`
  WHERE Date BETWEEN [START_DATE] AND [END_DATE]
)
ORDER BY
  Campaign_ID, Date

Command-line

# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
bq query --use_legacy_sql=false '
SELECT Campaign, Campaign_ID, Date FROM (
SELECT Campaign, Campaign_ID FROM `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
  WHERE _DATA_DATE = _LATEST_DATE
), (
SELECT date AS Date
  FROM `bigquery-public-data.common_us.date_greg`
  WHERE Date BETWEEN [START_DATE] AND [END_DATE]
)
ORDER BY
  Campaign_ID, Date'

Impressions and distinct users by campaign within a date range

The following sample query analyzes the number of impressions and distinct users by campaign between [START_DATE] and [END_DATE].

Web UI

#standardSQL
# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
SELECT
  base.*,
  imp.count AS imp_count,
  imp.du AS imp_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN [START_DATE]
      AND [END_DATE] ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].impression[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN [START_DATE]
    AND [END_DATE]
  GROUP BY
    Campaign_ID,
    Date ) AS imp
ON
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
WHERE
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
ORDER BY
  base.Campaign_ID,
  base.Date

Command-line

# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
bq query --use_legacy_sql=false '
SELECT
  base.*,
  imp.count AS imp_count,
  imp.du AS imp_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN [START_DATE]
      AND [END_DATE] ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].impression[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN [START_DATE]
    AND [END_DATE]
  GROUP BY
    Campaign_ID,
    Date ) AS imp
ON
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
WHERE
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
ORDER BY
  base.Campaign_ID,
  base.Date'

Impressions, clicks, activities and distinct users by campaign

The following sample query analyzes the number of impressions, clicks, activities, and distinct users by campaign over the past 30 days.

Web UI

#standardSQL
# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
SELECT
  base.*,
  imp.count AS imp_count,
  imp.du AS imp_du,
  click.count AS click_count,
  click.du AS click_du,
  activity.count AS activity_count,
  activity.du AS activity_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
      AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY) ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].impression[DOUBLECLICK_ID]`
  WHERE
    DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS imp
ON
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
LEFT JOIN (
  SELECT
    Campaign_ID,
    DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].click[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS click
ON
  base.Campaign_ID = click.Campaign_ID
  AND base.Date = click.Date
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].activity[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS activity
ON
  base.Campaign_ID = activity.Campaign_ID
  AND base.Date = activity.Date
WHERE
  base.Campaign_ID IN [CAMPAIGN_LIST]
  AND (base.Date = imp.Date
    OR base.Date = click.Date
    OR base.Date = activity.Date)
ORDER BY
  base.Campaign_ID,
  base.Date

Command-line

# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
bq query --use_legacy_sql=false '
SELECT
  base.*,
  imp.count AS imp_count,
  imp.du AS imp_du,
  click.count AS click_count,
  click.du AS click_du,
  activity.count AS activity_count,
  activity.du AS activity_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
      AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY) ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].impression[DOUBLECLICK_ID]`
  WHERE
    DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS imp
ON
  base.Campaign_ID = imp.Campaign_ID
  AND base.Date = imp.Date
LEFT JOIN (
  SELECT
    Campaign_ID,
    DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].click[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS click
ON
  base.Campaign_ID = click.Campaign_ID
  AND base.Date = click.Date
LEFT JOIN (
  SELECT
    Campaign_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].activity[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Date ) AS activity
ON
  base.Campaign_ID = activity.Campaign_ID
  AND base.Date = activity.Date
WHERE
  base.Campaign_ID IN [CAMPAIGN_LIST]
  AND (base.Date = imp.Date
    OR base.Date = click.Date
    OR base.Date = activity.Date)
ORDER BY
  base.Campaign_ID,
  base.Date'

Campaign activity

The following sample query analyzes campaign activity over the past 30 days. In this query, replace [CAMPAIGN_LIST] with a comma separated list of all the DoubleClick campaigns of interest within the scope of the query.

Web UI

#standardSQL
# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
SELECT
  base.*,
  activity.count AS activity_count,
  activity.du AS activity_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      mt_at.Activity_Group,
      mt_ac.Activity,
      mt_ac.Activity_Type,
      mt_ac.Activity_Sub_Type,
      mt_ac.Activity_ID,
      mt_ac.Activity_Group_ID
    FROM
      `[DATASET].match_table_activity_cats[DOUBLECLICK_ID]` AS mt_ac
    JOIN (
      SELECT
        Activity_Group,
        Activity_Group_ID
      FROM
        `[DATASET].match_table_activity_types_[DOUBLECLICK_ID]`
      WHERE
        _DATA_DATE = _LATEST_DATE ) AS mt_at
    ON
      mt_at.Activity_Group_ID = mt_ac.Activity_Group_ID
    WHERE
      _DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN [START_DATE]
      AND [END_DATE] ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    Activity_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].activity_[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Activity_ID,
    Date ) AS activity
ON
  base.Campaign_ID = activity.Campaign_ID
  AND base.Activity_ID = activity.Activity_ID
  AND base.Date = activity.Date
WHERE
  base.Campaign_ID IN [CAMPAIGN_LIST]
  AND base.Activity_ID = activity.Activity_ID
ORDER BY
  base.Campaign_ID,
  base.Activity_Group_ID,
  base.Activity_ID,
  base.Date

Command-line

# START_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
# END_DATE = DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
bq query --use_legacy_sql=false '
SELECT
  base.*,
  activity.count AS activity_count,
  activity.du AS activity_du
FROM (
  SELECT
    *
  FROM (
    SELECT
      Campaign,
      Campaign_ID
    FROM
      `[DATASET].match_table_campaigns_[DOUBLECLICK_ID]`
    WHERE
      DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      mt_at.Activity_Group,
      mt_ac.Activity,
      mt_ac.Activity_Type,
      mt_ac.Activity_Sub_Type,
      mt_ac.Activity_ID,
      mt_ac.Activity_Group_ID
    FROM
      `[DATASET].match_table_activity_cats[DOUBLECLICK_ID]` AS mt_ac
    JOIN (
      SELECT
        Activity_Group,
        Activity_Group_ID
      FROM
        `[DATASET].match_table_activity_types_[DOUBLECLICK_ID]`
      WHERE
        _DATA_DATE = _LATEST_DATE ) AS mt_at
    ON
      mt_at.Activity_Group_ID = mt_ac.Activity_Group_ID
    WHERE
      _DATA_DATE = _LATEST_DATE ),
    (
    SELECT
      date AS Date
    FROM
      `bigquery-public-data.common_us.date_greg`
    WHERE
      Date BETWEEN [START_DATE]
      AND [END_DATE] ) ) AS base
LEFT JOIN (
  SELECT
    Campaign_ID,
    Activity_ID,
    _DATA_DATE AS Date,
    COUNT() AS count,
    COUNT(DISTINCT User_ID) AS du
  FROM
    `[DATASET].activity_[DOUBLECLICK_ID]`
  WHERE
    _DATA_DATE BETWEEN DATE_ADD(CURRENT_DATE(), INTERVAL -31 DAY)
    AND DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY)
  GROUP BY
    Campaign_ID,
    Activity_ID,
    Date ) AS activity
ON
  base.Campaign_ID = activity.Campaign_ID
  AND base.Activity_ID = activity.Activity_ID
  AND base.Date = activity.Date
WHERE
  base.Campaign_ID IN [CAMPAIGN_LIST]
  AND base.Activity_ID = activity.Activity_ID
ORDER BY
  base.Campaign_ID,
  base.Activity_Group_ID,
  base.Activity_ID,
  base.Date'

Send feedback about...

BigQuery Documentation