BigQuery Data Transfer API Connector Overview

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery Data Transfer API

Schedule queries or transfer external data from SaaS applications to Google BigQuery on a regular basis. To learn more, see the BigQuery Data Transfer API documentation.

BigQuery Data Transfer connector sample

YAML

# This workflow creates a new dataset and a new table inside that dataset, which are required
# for the BigQuery Data Transfer Job to run. It creates a new TransferJob configuration and starts
# a manual run of the transfer (30 seconds after the config is created).
# The transferRun is a blocking LRO.
# All resources get deleted once the transfer run completes.
#
# On success, it returns "SUCCESS".
#
# Features included in this test:
# - BigQuery Data Transfer connector
# - Waiting for long-running transfer run to complete
#
# This workflow expects following items to be provided through input argument for execution:
#   - projectID (string)
#     - The user project ID.
#   - datasetID (string)
#     - The dataset name, expected to have an unique value to avoid the
#       instance being referred by multiple tests.
#   - tableID (string)
#     - The table name, expected to have an unique value to avoid the
#       instance being referred by multiple tests.
#   - runConfigDisplayName (string)
#     - The transfer run configuration display name.
#
# Expected successful output: "SUCCESS"
main:
  params: [args]
  steps:
    - init:
        assign:
          - project_id: ${args.projectID}
          - destination_dataset: ${args.datasetID}
          - destination_table: ${args.tableID}
          - run_config_display_name: ${args.runConfigDisplayName}
          - run_config_data_source_id: "google_cloud_storage"
          - location: "us"
          - data_path_template: "gs://xxxxxx-bucket/xxxxx/xxxx"
    - create_dataset:
        call: googleapis.bigquery.v2.datasets.insert
        args:
          projectId: ${project_id}
          body:
            datasetReference:
              datasetId: ${destination_dataset}
              projectId: ${project_id}
    - create_table:
        call: googleapis.bigquery.v2.tables.insert
        args:
          datasetId: ${destination_dataset}
          projectId: ${project_id}
          body:
            tableReference:
              datasetId: ${destination_dataset}
              projectId: ${project_id}
              tableId: ${destination_table}
            schema:
              fields:
                - name: "column1"
                  type: "STRING"
                - name: "column2"
                  type: "STRING"
    - list_config:
        call: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.list
        args:
          parent: ${"projects/" + project_id + "/locations/us"}
    - create_run_config:
        call: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.create
        args:
          parent: ${"projects/" + project_id + "/locations/" + location}
          body:
            displayName: ${run_config_display_name}
            schedule: "every day 19:22"
            scheduleOptions:
              disableAutoScheduling: true
            destinationDatasetId: ${destination_dataset}
            dataSourceId: ${run_config_data_source_id}
            params:
              destination_table_name_template: ${destination_table}
              file_format: "CSV"
              data_path_template: ${data_path_template}
        result: config
    - get_time_in_30s:
        assign:
          - now_plus_30s: ${time.format(sys.now() + 30)}
    - start_run:
        call: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.startManualRuns
        args:
          parent: ${config.name}
          body:
            requestedRunTime: ${now_plus_30s}
        result: runsResp
    - remove_run_config:
        call: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.delete
        args:
          name: ${config.name}
    - delete_table:
        call: googleapis.bigquery.v2.tables.delete
        args:
          datasetId: ${destination_dataset}
          projectId: ${project_id}
          tableId: ${destination_table}
    - delete_dataset:
        call: googleapis.bigquery.v2.datasets.delete
        args:
          projectId: ${project_id}
          datasetId: ${destination_dataset}
    - the_end:
        return: "SUCCESS"

JSON

{
  "main": {
    "params": [
      "args"
    ],
    "steps": [
      {
        "init": {
          "assign": [
            {
              "project_id": "${args.projectID}"
            },
            {
              "destination_dataset": "${args.datasetID}"
            },
            {
              "destination_table": "${args.tableID}"
            },
            {
              "run_config_display_name": "${args.runConfigDisplayName}"
            },
            {
              "run_config_data_source_id": "google_cloud_storage"
            },
            {
              "location": "us"
            },
            {
              "data_path_template": "gs://xxxxxx-bucket/xxxxx/xxxx"
            }
          ]
        }
      },
      {
        "create_dataset": {
          "call": "googleapis.bigquery.v2.datasets.insert",
          "args": {
            "projectId": "${project_id}",
            "body": {
              "datasetReference": {
                "datasetId": "${destination_dataset}",
                "projectId": "${project_id}"
              }
            }
          }
        }
      },
      {
        "create_table": {
          "call": "googleapis.bigquery.v2.tables.insert",
          "args": {
            "datasetId": "${destination_dataset}",
            "projectId": "${project_id}",
            "body": {
              "tableReference": {
                "datasetId": "${destination_dataset}",
                "projectId": "${project_id}",
                "tableId": "${destination_table}"
              },
              "schema": {
                "fields": [
                  {
                    "name": "column1",
                    "type": "STRING"
                  },
                  {
                    "name": "column2",
                    "type": "STRING"
                  }
                ]
              }
            }
          }
        }
      },
      {
        "list_config": {
          "call": "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.list",
          "args": {
            "parent": "${\"projects/\" + project_id + \"/locations/us\"}"
          }
        }
      },
      {
        "create_run_config": {
          "call": "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.create",
          "args": {
            "parent": "${\"projects/\" + project_id + \"/locations/\" + location}",
            "body": {
              "displayName": "${run_config_display_name}",
              "schedule": "every day 19:22",
              "scheduleOptions": {
                "disableAutoScheduling": true
              },
              "destinationDatasetId": "${destination_dataset}",
              "dataSourceId": "${run_config_data_source_id}",
              "params": {
                "destination_table_name_template": "${destination_table}",
                "file_format": "CSV",
                "data_path_template": "${data_path_template}"
              }
            }
          },
          "result": "config"
        }
      },
      {
        "get_time_in_30s": {
          "assign": [
            {
              "now_plus_30s": "${time.format(sys.now() + 30)}"
            }
          ]
        }
      },
      {
        "start_run": {
          "call": "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.startManualRuns",
          "args": {
            "parent": "${config.name}",
            "body": {
              "requestedRunTime": "${now_plus_30s}"
            }
          },
          "result": "runsResp"
        }
      },
      {
        "remove_run_config": {
          "call": "googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.delete",
          "args": {
            "name": "${config.name}"
          }
        }
      },
      {
        "delete_table": {
          "call": "googleapis.bigquery.v2.tables.delete",
          "args": {
            "datasetId": "${destination_dataset}",
            "projectId": "${project_id}",
            "tableId": "${destination_table}"
          }
        }
      },
      {
        "delete_dataset": {
          "call": "googleapis.bigquery.v2.datasets.delete",
          "args": {
            "projectId": "${project_id}",
            "datasetId": "${destination_dataset}"
          }
        }
      },
      {
        "the_end": {
          "return": "SUCCESS"
        }
      }
    ]
  }
}

Module: googleapis.bigquerydatatransfer.v1.projects.dataSources

Functions
checkValidCreds	Returns true if valid credentials exist for the given data source and requesting user.
get	Retrieves a supported data source and returns its settings.
list	Lists supported data sources and returns their settings.

Module: googleapis.bigquerydatatransfer.v1.projects.locations

Functions
get	Gets information about a location.
list	Lists information about the supported locations for this service.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.dataSources

Functions
checkValidCreds	Returns true if valid credentials exist for the given data source and requesting user.
get	Retrieves a supported data source and returns its settings.
list	Lists supported data sources and returns their settings.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs

Functions
create	Creates a new data transfer configuration.
delete	Deletes a data transfer configuration, including any associated transfer runs and logs.
get	Returns information about a data transfer config.
list	Returns information about all transfer configs owned by a project in the specified location.
patch	Updates a data transfer configuration. All fields must be set, even if they are not updated.
startManualRuns	Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.runs

Functions
delete	Deletes the specified transfer run.
get	Returns information about the particular transfer run.
list	Returns information about running and completed transfer runs.

Module: googleapis.bigquerydatatransfer.v1.projects.locations.transferConfigs.runs.transferLogs

Functions
list	Returns log messages for the transfer run.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs

Functions
create	Creates a new data transfer configuration.
delete	Deletes a data transfer configuration, including any associated transfer runs and logs.
get	Returns information about a data transfer config.
list	Returns information about all transfer configs owned by a project in the specified location.
patch	Updates a data transfer configuration. All fields must be set, even if they are not updated.
startManualRuns	Start manual transfer runs to be executed now with schedule_time equal to current time. The transfer runs can be created for a time range where the run_time is between start_time (inclusive) and end_time (exclusive), or for a specific run_time.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs.runs

Functions
delete	Deletes the specified transfer run.
get	Returns information about the particular transfer run.
list	Returns information about running and completed transfer runs.

Module: googleapis.bigquerydatatransfer.v1.projects.transferConfigs.runs.transferLogs

Functions
list	Returns log messages for the transfer run.