Create and manage deployments

A BigQuery Engine for Apache Flink deployment is a dedicated and isolated environment where your Apache Flink jobs run. The deployment contains the compute resources needed to run jobs and the configurations for managing jobs. For more information about Apache Flink deployments, see Deployment in the Apache Flink documentation.

Required APIs

To create and manage a BigQuery Engine for Apache Flink deployment, you must enable the BigQuery Engine for Apache Flink and Compute Engine APIs.

gcloud services enable managedflink.googleapis.com compute.googleapis.com

Required roles and permissions

This section describes the roles required to manage your deployments. For more information about BigQuery Engine for Apache Flink roles, see BigQuery Engine for Apache Flink predefined roles.

Create, update, and delete

To get the permissions that you need to create, update, and delete a deployment, ask your administrator to grant you the Managed Flink Admin (roles/managedflink.admin) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to create, update, and delete a deployment. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create, update, and delete a deployment:

  • Create a deployment: managedflink.deployments.create
  • Update a deployment: managedflink.deployments.update
  • Delete a deployment: managedflink.deployments.delete

You might also be able to get these permissions with custom roles or other predefined roles.

Get and list

To get the permissions that you need to retrieve information about a deployment and list deployments, ask your administrator to grant you the Managed Flink Viewer (roles/managedflink.viewer) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to retrieve information about a deployment and list deployments. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to retrieve information about a deployment and list deployments:

  • Get details about a deployment: managedflink.deployments.get
  • List deployments: managedflink.deployments.list

You might also be able to get these permissions with custom roles or other predefined roles.

Properties of a deployment

When you create a BigQuery Engine for Apache Flink deployment, you must specify the following properties.

Deployment ID

The name or ID of the BigQuery Engine for Apache Flink deployment that you are creating. You can modify the display name of your deployment, but you can't modify the deployment ID.

Project ID

The ID of the Google Cloud project for the deployment that you are creating. Your deployment is created in the project that you specify. For information about Google Cloud project IDs and project numbers, see Identifying projects.

Location

The location where you are creating the deployment. The location must be one of the supported Google Cloud regions. You can't change the location of a deployment. For a list of available locations, see BigQuery Engine for Apache Flink locations.

Virtual Private Cloud network

A Virtual Private Cloud(VPC) network is a virtual version of a physical network that is implemented inside of Google's production network. Sometimes called a network, a VPC provides connectivity for resources in a project. To learn more about VPC, see VPC network overview. To learn more about specifying a network for BigQuery Engine for Apache Flink deployments, see Specify a network and subnetwork.

Although the default network has configurations that allow deployments to run jobs, for security reasons, we recommend that you create a separate network for BigQuery Engine for Apache Flink. The default network is not secure, because it is pre-populated with firewall rules that allow incoming connections to instances.

  • You can select an auto mode VPC network in your project with the network parameter.

  • You can specify a network using only its name and not the complete URL.

Subnetwork

Specify a subnetwork by using the name of the subnetwork.

Create a deployment

To create a deployment, follow these steps. After you create the deployment, you can run jobs in the deployment. For more information, see Create a job in an existing deployment.

The first time you create either a deployment or an on-demand job in a project or in a subnet, the creation can take 30 minutes or more to complete. After that, it takes less time to create a new deployment or job.

gcloud

To create a deployment by using the gcloud CLI, use the gcloud alpha managed-flink deployments create command.

gcloud alpha managed-flink deployments create DEPLOYMENT_ID \
  --project=PROJECT_ID \
  --location=REGION \
  --network-config-vpc=NETWORK_NAME \
  --network-config-subnetwork=SUBNET_NAME \
  --max-slots=TASK_SLOTS

Replace the following:

  • DEPLOYMENT_ID: the name of your BigQuery Engine for Apache Flink deployment
  • PROJECT_ID: your BigQuery Engine for Apache Flink project ID
  • REGION: a BigQuery Engine for Apache Flink region, like us-central1
  • NETWORK_NAME: the name of your network. To use the default network, enter default.
  • SUBNET_NAME: the name of your subnet. To use the default subnet, enter default.
  • TASK_SLOTS: the maximum number of task slots available to the deployment

Update a deployment

You can modify the number of task slots available to your deployments. To modify any other deployment properties, you must delete the deployment and create a new one.

gcloud

To update a deployment, use the gcloud alpha managed-flink deployments update command.

gcloud alpha managed-flink deployments update \
  --deployments=DEPLOYMENT_ID \
  --project=PROJECT_ID \
  --location=REGION \
  --max-slots=TASK_SLOTS

Replace the following:

  • PROJECT_ID: your BigQuery Engine for Apache Flink project ID
  • REGION: a BigQuery Engine for Apache Flink region, like us-central1
  • DEPLOYMENT_ID: the name of your BigQuery Engine for Apache Flink deployment
  • TASK_SLOTS: the maximum number of task slots available to the deployment

Get details about a deployment

console

To get information about a deployment in the Google Cloud console, follow these steps:

  1. In the Google Cloud console, go to the BigQuery Engine for Apache Flink Deployments page.

    Go to Deployments

  2. To open the Deployment details page, click the name of a deployment. On the Deployment details page, you can see details about the deployment and jobs in the deployment.

gcloud

To retrieve information about a deployment by using the gcloud CLI, use the gcloud alpha managed-flink deployments describe command. This command retrieves the initial deployment implementation and the state of the deployment.

gcloud alpha managed-flink deployments describe \
  DEPLOYMENT_ID \
  --project=PROJECT_ID \
  --location=REGION

Replace the following:

  • DEPLOYMENT_ID: the name of your BigQuery Engine for Apache Flink deployment
  • PROJECT_ID: your BigQuery Engine for Apache Flink project ID
  • REGION: the region that the BigQuery Engine for Apache Flink job is in

List deployments

console

To get information about a deployment, in the Google Cloud console, go to the BigQuery Engine for Apache Flink Deployments page.

Go to Deployments

gcloud

To list the deployments in a project by using the gcloud CLI, use the gcloud alpha managed-flink deployments list command. This command lists all of the deployments in the region and project specified.

gcloud alpha managed-flink deployments list \
  REGION \
  --project=PROJECT_ID

Replace the following:

  • PROJECT_ID: your BigQuery Engine for Apache Flink project ID
  • REGION: the region that the BigQuery Engine for Apache Flink job is in

Delete deployments

gcloud

To delete a deployment by using the gcloud CLI, use the gcloud alpha managed-flink deployments delete command.

  • When you delete a deployment, the slot quota assigned to the deployment is released.
  • To delete a deployment that contains existing jobs, use the force_deletion parameter.
gcloud alpha managed-flink deployments delete \
  DEPLOYMENT_ID \
  --project=PROJECT_ID \
  --location=REGION

Replace the following:

  • DEPLOYMENT_ID: the name of your BigQuery Engine for Apache Flink deployment
  • PROJECT_ID: your BigQuery Engine for Apache Flink project ID
  • REGION: the region that the BigQuery Engine for Apache Flink job is in

What's next