BigQuery to Bigtable template

The BigQuery to Bigtable template is a batch pipeline that copies data from a BigQuery table into an existing Bigtable table. The template can either read the entire table or read specific records using a supplied query.

Pipeline requirements

Template parameters

Required parameters

  • readIdColumn : The name of the BigQuery column storing the unique identifier of the row.
  • bigtableWriteInstanceId : The ID of the Bigtable instance that contains the table.
  • bigtableWriteTableId : The ID of the Bigtable table to write to.
  • bigtableWriteColumnFamily : The name of the column family of the Bigtable table to write data into.

Optional parameters

  • inputTableSpec : The BigQuery table to read from. Format: projectId:datasetId.tablename. If you specify inputTableSpec, the template reads the data directly from BigQuery storage by using the BigQuery Storage Read API (https://cloud.google.com/bigquery/docs/reference/storage). For information about limitations in the Storage Read API, see https://cloud.google.com/bigquery/docs/reference/storage#limitations. You must specify either inputTableSpec or query. If you set both parameters, the template uses the query parameter. (Example: bigquery-project:dataset.input_table).
  • outputDeadletterTable : The BigQuery table for messages that failed to reach the output table, in the format <PROJECT_ID>:<DATASET_NAME>.<DEADLETTER_TABLE>. If a table doesn't exist, is is created during pipeline execution. If not specified, <outputTableSpec>_error_records is used. (Example: your-project-id:your-dataset.your-table-name).
  • query : The SQL query to use to read data from BigQuery. If the BigQuery dataset is in a different project than the Dataflow job, specify the full dataset name in the SQL query, for example: <PROJECT_ID>.<DATASET_NAME>.<TABLE_NAME>. By default, the query parameter uses GoogleSQL (https://cloud.google.com/bigquery/docs/introduction-sql), unless useLegacySql is true. You must specify either inputTableSpec or query. If you set both parameters, the template uses the query parameter. (Example: select * from sampledb.sample_table).
  • useLegacySql : Set to true to use legacy SQL. This parameter only applies when using the query parameter. Defaults to: false.
  • queryLocation : Needed when reading from an authorized view without underlying table's permission. (Example: US).
  • queryTempDataset : With this option, you can set an existing dataset to create the temporary table to store the results of the query. (Example: temp_dataset).
  • bigtableRpcAttemptTimeoutMs : The timeout for each Bigtable RPC attempt in milliseconds.
  • bigtableRpcTimeoutMs : The total timeout for a Bigtable RPC operation in milliseconds.
  • bigtableAdditionalRetryCodes : The additional retry codes. (Example: RESOURCE_EXHAUSTED,DEADLINE_EXCEEDED).
  • bigtableWriteAppProfile : The ID of the Bigtable application profile to use for the export. If you do not specify an app profile, Bigtable uses the default app profile (https://cloud.google.com/bigtable/docs/app-profiles#default-app-profile) of the instance.
  • bigtableWriteProjectId : The ID of the Google Cloud project that contains the Bigtable instanceto write data to.
  • bigtableBulkWriteLatencyTargetMs : The latency target of Bigtable in milliseconds for latency-based throttling.
  • bigtableBulkWriteMaxRowKeyCount : The maximum number of row keys in a Bigtable batch write operation.
  • bigtableBulkWriteMaxRequestSizeBytes : The maximum bytes to include per Bigtable batch write operation.

Run the template

Console

  1. Go to the Dataflow Create job from template page.
  2. Go to Create job from template
  3. In the Job name field, enter a unique job name.
  4. Optional: For Regional endpoint, select a value from the drop-down menu. The default region is us-central1.

    For a list of regions where you can run a Dataflow job, see Dataflow locations.

  5. From the Dataflow template drop-down menu, select the BigQuery to Bigtable template.
  6. In the provided parameter fields, enter your parameter values.
  7. Click Run job.

gcloud

In your shell or terminal, run the template:

gcloud dataflow flex-template run JOB_NAME \
    --project=PROJECT_ID \
    --region=REGION_NAME \
    --template-file-gcs-location=gs://dataflow-templates-REGION_NAME/VERSION/flex/BigQuery_to_Bigtable \
    --parameters \
readIdColumn=READ_COLUMN_ID,\
inputTableSpec=INPUT_TABLE_SPEC,\
bigtableWriteInstanceId=BIGTABLE_INSTANCE_ID,\
bigtableWriteTableId=BIGTABLE_TABLE_ID,\
bigtableWriteColumnFamily=BIGTABLE_COLUMN_FAMILY

Replace the following:

  • PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job
  • JOB_NAME: a unique job name of your choice
  • REGION_NAME: the region where you want to deploy your Dataflow job—for example, us-central1
  • VERSION: the version of the template that you want to use

    You can use the following values:

  • READ_COLUMN_ID: your BigQuery unique id column.
  • INPUT_TABLE_SPEC: your BigQuery table name.
  • BIGTABLE_INSTANCE_ID: your Bigtable instance id.
  • BIGTABLE_TABLE_ID: your Bigtable table id.
  • BIGTABLE_COLUMN_FAMILY: your Bigtable table column family.

API

To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch.

POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch
{
   "launch_parameter": {
      "jobName": "JOB_NAME",
      "parameters": {
          "readIdColumn": "READ_COLUMN_ID",
          "inputTableSpec": "INPUT_TABLE_SPEC",
          "bigtableWriteInstanceId": "BIGTABLE_INSTANCE_ID",
          "bigtableWriteTableId": "BIGTABLE_TABLE_ID",
          "bigtableWriteColumnFamily": "BIGTABLE_COLUMN_FAMILY"
      },
      "containerSpecGcsPath": "gs://dataflow-templates-LOCATION/VERSION/flex/BigQuery_to_Bigtable",
   }
}

Replace the following:

  • PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job
  • JOB_NAME: a unique job name of your choice
  • LOCATION: the region where you want to deploy your Dataflow job—for example, us-central1
  • VERSION: the version of the template that you want to use

    You can use the following values:

  • READ_COLUMN_ID: your BigQuery unique id column.
  • INPUT_TABLE_SPEC: your BigQuery table name.
  • BIGTABLE_INSTANCE_ID: your Bigtable instance id.
  • BIGTABLE_TABLE_ID: your Bigtable table id.
  • BIGTABLE_COLUMN_FAMILY: your Bigtable table column family.

What's next