Create a BigQuery connection

Stay organized with collections Save and categorize content based on your preferences.

You're viewing Apigee X documentation.
View Apigee Edge documentation.

The BigQuery connector lets you insert, read, update and delete rows in a BigQuery table and use the resulting output in an integration. This page provides a reference for configuring a BigQuery connection and using the BigQuery connection with Apigee Integration.

Google BigQuery is a serverless, scalable enterprise data warehouse that enables super-fast SQL queries using the processing power of Google's infrastructure. Because there is no infrastructure to manage, you can focus on analyzing data to find meaningful insights using familiar SQL without the need for a database administrator.

For a hands-on introduction, try one of the BigQuery quickstarts.

Before you begin

Before creating a BigQuery connection to use in your integration, do the following in your Google Cloud project:

  • Enable the BigQuery API for your account.
  • Create a BigQuery dataset and table with schema.
  • Grant the following roles to the service account that you want to use to create the connection:
    • roles/bigquery.dataEditor
    • roles/bigquery.readSessionUser
    • roles/secretmanager.viewer
    • roles/secretmanager.secretAccessor
  • Enable the following services:
    • apigee.googleapis.com (Apigee API)
    • secretmanager.googleapis.com (Secret Manager API)
    • connectors.googleapis.com (Connector Platform API)

    If these services have not been enabled for your project previously, you are prompted to enable them when creating the connection in the Create Connection page.

Create a connection

  1. In the Cloud console page, select or create a Google Cloud project.
  2. Open the connections page .
  3. Click + CREATE NEW to open the Create Connection page.
  4. Configure the connection:
    1. In the Create Connection section, complete the following:
      • Connector: Select GCP > BigQuery from the drop down list of available connectors.
      • Connector version: Select the connector version from the drop down list of available versions.
      • In the Connection Name field, enter a name for the connection instance.

        Connection names must meet the following criteria:

        • Connection names can use letters, numbers, or hyphens.
        • Letters must be lower-case.
        • Connection names must begin with a letter and end with a letter or number.
        • Connection names cannot exceed 63 characters.
      • Optionally, add a Description of the connection instance.
      • Service Account: Select a service account that has the required roles.
      • Project ID: Enter the ID of the Google Cloud project where the data resides.
      • Dataset ID: Enter the ID of the BigQuery Dataset.
      • Optionally, click + ADD LABEL to add a label to the Connection in the form of a key/value pair.
      • Click NEXT.
    2. Location: Choose the location for the connection.
      • Select a location where Apigee is available from the Region drop-down list.

        Supported regions for connectors include:

        • asia-east1
        • asia-northeast1
        • asia-south1
        • asia-southeast1
        • australia-southeast1
        • europe-west1
        • europe-west2
        • europe-west3
        • europe-west4
        • europe-west6
        • northamerica-northeast1
        • southamerica-east1
        • us-central1
        • us-east1
        • us-east4
        • us-west1
        • us-west2
      • Click NEXT.
    3. Authentication: Provide authentication credentials for your connection instance, if required. The BigQuery connection does not require authentication configuration.
    4. Click NEXT.
    5. Review: Review your connections's configuration details. In this section, the Connection details and Authentication details of the new connection are displayed for your review.
  5. Click Create.

Create a Connectors task

After you create a BigQuery connection, it becomes available in Apigee integrations. You can then use the connection to configure the Connectors tasks in your integrations. For more information, see Configure the Connectors task.