BigQuery API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

BigQuery API

A data platform for customers to create, manage, share and query data. To learn more, see the BigQuery API documentation.

You can use a page token in a workflow to paginate through the results of a query to a BigQuery dataset.

BigQuery connector sample

YAML

# This workflow demonstrates how to use the BigQuery connector.
# The workflow creates a new dataset and then issues a simple query to insert a table with
# some data from bigquery-public-data.usa_names.usa_1910_2013.
# The new table and dataset are both deleted in the following steps.
# Expected successful output: "SUCCESS"

- init:
    assign:
      - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
      - dataset_id: "example_dataset"
      - table_id: "example_table"
      - query: "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
      - create_disposition: "CREATE_IF_NEEDED"  # create a new one if table doesn't exist
      - write_disposition: "WRITE_TRUNCATE"  # truncate it if the table already exists
- create_dataset:
    call: googleapis.bigquery.v2.datasets.insert
    args:
      projectId: ${project_id}
      body:
        datasetReference:
          datasetId: ${dataset_id}
          projectId: ${project_id}
        access[].role: "roles/bigquery.dataViewer"
        access[].specialGroup: "projectReaders"
- insert_table_into_dataset:
    call: googleapis.bigquery.v2.jobs.insert
    args:
      projectId: ${project_id}
      body:
        configuration:
          query:
            query: ${query}
            destinationTable:
              projectId: ${project_id}
              datasetId: ${dataset_id}
              tableId: ${table_id}
            create_disposition: ${create_disposition}
            write_disposition: ${write_disposition}
            allowLargeResults: true
            useLegacySql: false
- delete_table_from_dataset:
    call: googleapis.bigquery.v2.tables.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
      tableId: ${table_id}
- delete_dataset:
    call: googleapis.bigquery.v2.datasets.delete
    args:
      projectId: ${project_id}
      datasetId: ${dataset_id}
- the_end:
    return: "SUCCESS"

JSON

[
  {
    "init": {
      "assign": [
        {
          "project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"
        },
        {
          "dataset_id": "example_dataset"
        },
        {
          "table_id": "example_table"
        },
        {
          "query": "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 5000;"
        },
        {
          "create_disposition": "CREATE_IF_NEEDED"
        },
        {
          "write_disposition": "WRITE_TRUNCATE"
        }
      ]
    }
  },
  {
    "create_dataset": {
      "call": "googleapis.bigquery.v2.datasets.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "datasetReference": {
            "datasetId": "${dataset_id}",
            "projectId": "${project_id}"
          },
          "access[].role": "roles/bigquery.dataViewer",
          "access[].specialGroup": "projectReaders"
        }
      }
    }
  },
  {
    "insert_table_into_dataset": {
      "call": "googleapis.bigquery.v2.jobs.insert",
      "args": {
        "projectId": "${project_id}",
        "body": {
          "configuration": {
            "query": {
              "query": "${query}",
              "destinationTable": {
                "projectId": "${project_id}",
                "datasetId": "${dataset_id}",
                "tableId": "${table_id}"
              },
              "create_disposition": "${create_disposition}",
              "write_disposition": "${write_disposition}",
              "allowLargeResults": true,
              "useLegacySql": false
            }
          }
        }
      }
    }
  },
  {
    "delete_table_from_dataset": {
      "call": "googleapis.bigquery.v2.tables.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}",
        "tableId": "${table_id}"
      }
    }
  },
  {
    "delete_dataset": {
      "call": "googleapis.bigquery.v2.datasets.delete",
      "args": {
        "projectId": "${project_id}",
        "datasetId": "${dataset_id}"
      }
    }
  },
  {
    "the_end": {
      "return": "SUCCESS"
    }
  }
]

Module: googleapis.bigquery.v2.datasets

Functions
`delete`	Deletes the dataset specified by the datasetId value. Before you can delete a dataset, you must delete all its tables, either manually or by specifying deleteContents. Immediately after deletion, you can create another dataset with the same name.
`get`	Returns the dataset specified by datasetID.
`insert`	Creates a new empty dataset.
`list`	Lists all datasets in the specified project to which you have been granted the READER dataset role.
`patch`	Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource. This method supports patch semantics.
`update`	Updates information in an existing dataset. The update method replaces the entire dataset resource, whereas the patch method only replaces fields that are provided in the submitted dataset resource.

Module: googleapis.bigquery.v2.jobs

Functions
`cancel`	Requests that a job be cancelled. This call will return immediately, and the client will need to poll for the job status to see if the cancel completed successfully. Cancelled jobs may still incur costs.
`delete`	Requests the deletion of the metadata of a job. This call returns when the job's metadata is deleted.
`get`	Returns information about a specific job. Job information is available for a six month period after creation. Requires that you're the person who ran the job, or have the Is Owner project role.
`getQueryResults`	Retrieves the results of a query job.
`insert`	Starts a new asynchronous job. Requires the Can View project role.
`list`	Lists all jobs that you started in the specified project. Job information is available for a six month period after creation. The job list is sorted in reverse chronological order, by job creation time. Requires the Can View project role, or the Is Owner project role if you set the allUsers property.
`query`	Runs a BigQuery SQL query synchronously and returns query results if the query completes within a specified timeout.

Module: googleapis.bigquery.v2.models

Functions
`delete`	Deletes the model specified by modelId from the dataset.
`get`	Gets the specified model resource by model ID.
`list`	Lists all models in the specified dataset. Requires the READER dataset role. After retrieving the list of models, you can get information about a particular model by calling the models.get method.
`patch`	Patch specific fields in the specified model.

Module: googleapis.bigquery.v2.projects

Functions
`getServiceAccount`	Returns the email address of the service account for your project used for interactions with Google Cloud KMS.
`list`	Lists all projects to which you have been granted any project role.

Module: googleapis.bigquery.v2.routines

Functions
`delete`	Deletes the routine specified by routineId from the dataset.
`get`	Gets the specified routine resource by routine ID.
`insert`	Creates a new routine in the dataset.
`list`	Lists all routines in the specified dataset. Requires the READER dataset role.
`update`	Updates information in an existing routine. The update method replaces the entire Routine resource.

Module: googleapis.bigquery.v2.rowAccessPolicies

Functions
`getIamPolicy`	Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
`list`	Lists all row access policies on the specified table.
`testIamPermissions`	Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a `NOT_FOUND` error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.

Module: googleapis.bigquery.v2.tabledata

Functions
`insertAll`	Streams data into BigQuery one record at a time without needing to run a load job. Requires the WRITER dataset role.
`list`	Retrieves table data from a specified set of rows. Requires the READER dataset role.

Module: googleapis.bigquery.v2.tables

Functions
`delete`	Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.
`get`	Gets the specified table resource by table ID. This method does not return the data in the table, it only returns the table resource, which describes the structure of this table.
`getIamPolicy`	Gets the access control policy for a resource. Returns an empty policy if the resource exists and does not have a policy set.
`insert`	Creates a new, empty table in the dataset.
`list`	Lists all tables in the specified dataset. Requires the READER dataset role.
`patch`	Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource. This method supports patch semantics.
`setIamPolicy`	Sets the access control policy on the specified resource. Replaces any existing policy. Can return `NOT_FOUND`, `INVALID_ARGUMENT`, and `PERMISSION_DENIED` errors.
`testIamPermissions`	Returns permissions that a caller has on the specified resource. If the resource does not exist, this will return an empty set of permissions, not a `NOT_FOUND` error. Note: This operation is designed to be used for building permission-aware UIs and command-line tools, not for authorization checking. This operation may "fail open" without warning.
`update`	Updates information in an existing table. The update method replaces the entire table resource, whereas the patch method only replaces fields that are provided in the submitted table resource.