Authenticate to BigQuery Engine for Apache Flink

This document describes how to authenticate to BigQuery Engine for Apache Flink programmatically. How you authenticate to BigQuery Engine for Apache Flink depends on the interface you use to access the API and the environment where your code is running.

After authenticating to BigQuery Engine for Apache Flink, you must grant specific permissions to access Google Cloud resources. For more information, see BigQuery Engine for Apache Flink security and permissions.

For more information about Google Cloud authentication, see the authentication overview.

API access

BigQuery Engine for Apache Flink supports programmatic access. You can access the API in the following ways:

Google Cloud CLI

When you use the gcloud CLI to access BigQuery Engine for Apache Flink, you log in to the gcloud CLI with a user account, which provides the credentials used by the gcloud CLI commands.

If your organization's security policies prevent user accounts from having the required permissions, you can use service account impersonation.

For more information, see Authenticate for using the gcloud CLI. For more information about using the gcloud CLI with BigQuery Engine for Apache Flink, see the gcloud CLI reference pages.

Set up authentication for BigQuery Engine for Apache Flink

How you set up authentication depends on the environment where your code is running.

The following options for setting up authentication are the most commonly used. For more options and information about authentication, see Authentication methods.

For a local development environment

You can set up credentials for a local development environment in the following ways:

Client libraries or third-party tools

Set up Application Default Credentials (ADC) in your local environment:

  1. Install the Google Cloud CLI, then initialize it by running the following command: 

    gcloud init

  2. If you're using a local shell, then create local authentication credentials for your user account: 

    gcloud auth application-default login

    You don't need to do this if you're using Cloud Shell.

    A sign-in screen appears. After you sign in, your credentials are stored in the local credential file used by ADC.

For more information about working with ADC in a local environment, see Local development environment.

On Google Cloud

To authenticate a workload running on Google Cloud, you use the credentials of the service account attached to the compute resource where your code is running, such as a Compute Engine virtual machine (VM) instance. This approach is the preferred authentication method for code running on a Google Cloud compute resource.

For most services, you must attach the service account when you create the resource that will run your code; you cannot add or replace the service account later. Compute Engine is an exception—it lets you attach a service account to a VM instance at any time.

Use the gcloud CLI to create a service account and attach it to your resource:

  1. Install the Google Cloud CLI, then initialize it by running the following command: 

    gcloud init

  2. Set up authentication:

    1. Create the service account: 

      gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

      Replace SERVICE_ACCOUNT_NAME with a name for the service account.

    2. To provide access to your project and your resources, grant a role to the service account: 

      gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE

      Replace the following:

      • SERVICE_ACCOUNT_NAME: the name of the service account
      • PROJECT_ID: the project ID where you created the service account
      • ROLE: the role to grant
    3. To grant another role to the service account, run the command as you did in the previous step.

    4. Grant the required role to the principal that will attach the service account to other resources. 

      gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com --member="user:USER_EMAIL" --role=roles/iam.serviceAccountUser

      Replace the following:

      • SERVICE_ACCOUNT_NAME: the name of the service account
      • PROJECT_ID: the project ID where you created the service account
      • USER_EMAIL: the email address for a Google Account

  3. Create the resource that will run your code, and attach the service account to that resource. For example, if you use Compute Engine:

    Create a Compute Engine instance. Configure the instance as follows:
    • Replace INSTANCE_NAME with your preferred instance name.
    • Set the --zone flag to the zone in which you want to create your instance.
    • Set the --service-account flag to the email address for the service account that you created. 
    gcloud compute instances create INSTANCE_NAME --zone=ZONE --service-account=SERVICE_ACCOUNT_EMAIL

For more information about authenticating to Google APIs, see Authentication methods.

Access control for BigQuery Engine for Apache Flink

After you authenticate to BigQuery Engine for Apache Flink, you must be authorized to access Google Cloud resources. BigQuery Engine for Apache Flink uses Identity and Access Management (IAM) for authorization.

For more information about the roles for BigQuery Engine for Apache Flink, see Access control with IAM. For more information about IAM and authorization, see IAM overview.

What's next