Configure a BigQuery (Preview) connection

You're viewing Apigee X documentation.
View Apigee Edge documentation.

The BigQuery Connector (Preview) type lets you insert, read, update and delete rows in a BigQuery table and use the resulting output in an integration. This page provides a reference for configuring a BigQuery (Preview) connection and using the BigQuery (Preview) connection with Apigee Integration

About Google BigQuery

Google BigQuery is a serverless, scalable enterprise data warehouse that enables super-fast SQL queries using the processing power of Google's infrastructure. Because there is no infrastructure to manage, you can focus on analyzing data to find meaningful insights using familiar SQL without the need for a database administrator.

For a hands-on introduction, try one of the BigQuery quickstarts.

Before you begin

Before creating a BigQuery (Preview) connection to use in your integration, do the following:

  • Enable the BigQuery API for your account.
  • Create a BigQuery dataset and table with schema.
  • Grant the Compute Engine default service account for your project the following roles in order to access the dataset:
    • roles/bigquery.dataEditor
    • roles/bigquery.readSessionUser

Enable required services

Connectors requires that you enable the following Google services:

Name Title
apigee.googleapis.com Apigee API
secretmanager.googleapis.com Secret Manager API
connectors.googleapis.com Connector Platform API

If these services have not been enabled for your project previously, you will be prompted to enable them before using the Connectors UI.

Create a BigQuery (Preview) connection

To configure a Connection:

  1. In the Cloud console, go to Apigee > Connectors page and select or create a Google Cloud project.

    Go to the Connections page

  2. Click + CREATE NEW to open the Create Connection page.
  3. Configure the connection:
    1. In the Create Connection section, complete the following:
      • Connector: Select the Connector type from the drop down list of available Connector types. For this example, select GCP > BigQuery (Preview).
      • Connector version: Select the Connector version from the drop down list of available versions.
      • In the Connection Name field, enter a name for the Connection instance.

        Connection names must meet the following criteria:

        • Connection names can use letters, numbers, or hyphens.
        • Letters must be lower-case.
        • Connection names must begin with a letter and end with a letter or number.
        • Connection names cannot exceed 63 characters.
      • Optionally, add a Description of the connection instance.
      • Project ID: Enter the of the Google Cloud project where the data resides.
      • Dataset ID: Enter the ID of the BigQuery Dataset.
      • Optionally, click + ADD LABEL to add a label to the Connection in the form of a key/value pair.
      • Click NEXT.
    2. Location: Choose the location for the connection.
      • Select a location where Apigee is available from the Region drop-down list.

        Supported regions for Connectors include:

        • us-central1
        • us-east1
        • us-west1
        • europe-west1
        • asia-east1
      • Click NEXT.
    3. Authentication: Provide authentication credentials for your connection instance, if required. The BigQuery (Preview) connection does not require authentication configuration.
    4. Click NEXT.
    5. Review: Review your connections's configuration details. In this section, the Connection details and Authentication details of the new connection are displayed for your review.
  4. Click Create.

Create a Connectors task with a BigQuery (Preview) connection

To configure the Connectors task in Apigee Integration to use the BigQuery connection:

  1. Go to the Apigee UI and sign in.
  2. Select your organization using the drop-down menu in the upper left corner of the UI.
  3. Click Develop > Integrations.
  4. Search for or select the integration to edit from the Integrations list and click to open the integration designer.
  5. Click +Add a task/trigger > Tasks to view the list of available tasks.
  6. Drag the Connectors task element to the integration designer.
  7. Click the Connectors task element on the designer to view the Connectors task configuration pane.
  8. Optionally, click the to edit the task name. This enables you to change the task name from the generic Connectors to a meaningful name for your integration.
  9. On the Configuration tab in the Configuration section, click the Configure task button to open the Configure connector task pane.
    1. In the Connection column, choose the BigQuery (Preview) connection you created in a previous step from the list of available connections.
    2. Once a connection is chosen, the Type column appears. Select either an Entity or Action from the list of available Entities and Actions.
      • An Entity can be thought of as an object, or a collection of properties, in the connected application or service, that can be exposed to an integration through the Connector interface.

        Selecting an Entity from the available list generates a list of available Operations that are available for the Entity.

        Select an Operation from the available list.

      • An Action is a first class function that is made available to the integration through the Connector interface. Functions are defined in the connected application or service and define a change or series of changes that can be made to an entity or entities.

        Select an Action from the available list.

    3. Click Done to complete the connection configuration and close the pane.
  10. The Connectors task configuration pane for a BigQuery (Preview) connection also displays Task Input variables and Task Output variable options automatically generated by the Entity and Operation or Action selected in the previous step for use in the connection. These variables are accessible as inputs to the current task or outputs to subsequent tasks or conditionals configured in the current integration.

    To configure the Task Input or Task Output, click on the Task input or Task output fields to select a variable or add a new variable.

  11. You can configure various retry strategies to handle errors in a task. The retry strategies allow you to specify how to rerun the task in case of an error. For more information, see Error handling strategies.