BigQuery API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery API

A data platform for customers to create, manage, share and query data. To learn more, see the BigQuery API documentation.

You can use a page token in a workflow to paginate through the results of a query to a BigQuery dataset.

BigQuery connector sample

YAML

# This workflow demonstrates how to use the BigQuery connector:
# Create a dataset and then insert a table with data from a public dataset
# Delete both the table and dataset
# Expected output: "SUCCESS"
- init:
    assign:
      - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
      - dataset_id: "example_dataset"
      - table_id: "example_table"
      - query: "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
      - create_disposition: "CREATE_IF_NEEDED"  # creates table if it doesn't exist
      - write_disposition: "WRITE_TRUNCATE"  # truncates table if it already exists
- create_dataset:
    call: googleapis.bigquery.v2.datasets.insert
    args:
      projectId: ${project_id}
      body:
        datasetReference:
          datasetId: ${dataset_id}
          projectId: ${project_id}
        access[].role: "roles/bigquery.dataViewer"
        access[].specialGroup: "projectReaders"
- insert_table_into_dataset:
    call: googleapis.bigquery.v2.jobs.insert
    args:
      projectId: ${project_id}
      body:
        configuration:
          query:
            query: ${query}
            destinationTable:
              projectId: ${project_id}
              datasetId: ${dataset_id}
              tableId: ${table_id}
            create_disposition: ${create_disposition}
            write_disposition: ${write_disposition}
            allowLargeResults: true
            useLegacySql: false
- delete_table_from_dataset:
    call: googleapis.bigquery.v2.tables.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
      tableId: ${table_id}
- delete_dataset:
    call: googleapis.bigquery.v2.datasets.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
- the_end:
    return: "SUCCESS"

JSON

[
  {
    "init": {
      "assign": [
        {
          "project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"
        },
        {
          "dataset_id": "example_dataset"
        },
        {
          "table_id": "example_table"
        },
        {
          "query": "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
        },
        {
          "create_disposition": "CREATE_IF_NEEDED"
        },
        {
          "write_disposition": "WRITE_TRUNCATE"
        }
      ]
    }
  },
  {
    "create_dataset": {
      "call": "googleapis.bigquery.v2.datasets.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "datasetReference": {
            "datasetId": "${dataset_id}",
            "projectId": "${project_id}"
          },
          "access[].role": "roles/bigquery.dataViewer",
          "access[].specialGroup": "projectReaders"
        }
      }
    }
  },
  {
    "insert_table_into_dataset": {
      "call": "googleapis.bigquery.v2.jobs.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "configuration": {
            "query": {
              "query": "${query}",
              "destinationTable": {
                "projectId": "${project_id}",
                "datasetId": "${dataset_id}",
                "tableId": "${table_id}"
              },
              "create_disposition": "${create_disposition}",
              "write_disposition": "${write_disposition}",
              "allowLargeResults": true,
              "useLegacySql": false
            }
          }
        }
      }
    }
  },
  {
    "delete_table_from_dataset": {
      "call": "googleapis.bigquery.v2.tables.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}",
        "tableId": "${table_id}"
      }
    }
  },
  {
    "delete_dataset": {
      "call": "googleapis.bigquery.v2.datasets.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}"
      }
    }
  },
  {
    "the_end": {
      "return": "SUCCESS"
    }
  }
]

Module: googleapis.bigquery.v2.datasets

Functions
delete	Deletes the dataset specified by the datasetId value. Before you can delete a dataset, you must delete all its tables, either manually or by specifying deleteContents. Immediately after deletion, you can create another dataset with the same name.
get	Returns the dataset specified by datasetID.
insert	Creates a new empty dataset.
list	Lists all datasets in the specified project to which you have been granted the READER dataset role.
patch	Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource. This method supports patch semantics.
update	Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource.

Module: googleapis.bigquery.v2.jobs

Functions
cancel	Requests that a job be cancelled. This call will return immediately, and the client will need to poll for the job status to see if the cancel completed successfully. Cancelled jobs may still incur costs.
delete	Requests the deletion of the metadata of a job. This call returns when the job's metadata is deleted.
get	Returns information about a specific job. Job information is available for a six month period after creation. Requires that you're the person who ran the job, or have the Is Owner project role.
getQueryResults	Retrieves the results of a query job.
insert	Starts a new asynchronous job. Requires the Can View project role.
list	Lists all jobs that you started in the specified project. Job information is available for a six month period after creation. The job list is sorted in reverse chronological order, by job creation time. Requires the Can View project role, or the Is Owner project role if you set the allUsers property.
query	Runs a BigQuery SQL query synchronously and returns query results if the query completes within a specified timeout.

Module: googleapis.bigquery.v2.models

Functions
delete	Deletes the model specified by modelId from the dataset.
get	Gets the specified model resource by model ID.
list	Lists all models in the specified dataset. Requires the READER dataset role. After retrieving the list of models, you can get information about a particular model by calling the models.get method.
patch	Patch specific fields in the specified model.

Module: googleapis.bigquery.v2.projects

Functions
getServiceAccount	Returns the email address of the service account for your project used for interactions with Google Cloud KMS.
list	Lists all projects to which you have been granted any project role.

Module: googleapis.bigquery.v2.routines

Functions
delete	Deletes the routine specified by routineId from the dataset.
get	Gets the specified routine resource by routine ID.
insert	Creates a new routine in the dataset.
list	Lists all routines in the specified dataset. Requires the READER dataset role.
update	Updates information in an existing routine. The update method replaces the entire Routine resource.

Module: googleapis.bigquery.v2.rowAccessPolicies

Functions
getIamPolicy	Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
list	Lists all row access policies on the specified table.
testIamPermissions	Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.

Module: googleapis.bigquery.v2.tabledata

Functions
insertAll	Streams data into BigQuery one record at a time without needing to run a load job. Requires the WRITER dataset role.
list	Retrieves table data from a specified set of rows. Requires the READER dataset role.

Module: googleapis.bigquery.v2.tables

Functions
delete	Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.
get	Gets the specified table resource by table ID. This method does not return the data in the table, it only returns the table resource, which describes the structure of this table.
getIamPolicy	Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
insert	Creates a new, empty table in the dataset.
list	Lists all tables in the specified dataset. Requires the READER dataset role.
patch	Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource. This method supports patch semantics.
setIamPolicy	Sets the access control policy on the specified resource. Replaces any existing policy. Can return NOT_FOUND, INVALID_ARGUMENT, and PERMISSION_DENIED errors.
testIamPermissions	Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a NOT_FOUND error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
update	Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource.