Workload management using legacy reservations

Note: Legacy reservations, including access to flat-rate billing or certain commitment lengths, are only available to allow-listed customers. To determine if you have access to these legacy features, contact your administrator.

This page describes how to use BigQuery Reservations to help manage your BigQuery workloads.

Choose a billing model

At any point in time, you can choose to switch between on-demand and flat-rate billing models, or you can choose to mix and match the two models together. The two pricing models are complementary to each other. On-demand billing is efficient; whereas, flat-rate is predictable.

Consider the following factors when choosing flat-rate billing:

• Cost. Google recommends that your monthly analytics spend be in the ballpark of the minimum BigQuery flat-rate commitment (currently, 100 slots per month).
• Efficiency. Consider on-demand if you value efficiency because you pay per query.
• Predictability. You might prefer flat-rate if you value monthly predictability of pricing because you pay one flat monthly fee.
• Resources. Flat-rate and on-demand billing models differ in how they offer capacity. For more information, see Estimating the amount of slots to purchase.

As a new customer, you might choose to start with flat-rate. At any point in time, you can observe what your cost would have been with on-demand billing, and at any point in time you can choose the model that works for you.

You can also experiment with Flex slots to test out price and performance of flat-rate billing for your organization before settling on a steady-state workload configuration. Flex slots are a special commitment type:

• Commitment duration is only 60 seconds.
• You can cancel Flex slots any time thereafter.
• You are charged only for the seconds your commitment was deployed.

You can combine Flex slot commitments with other types of commitments. Flex slots are ideal for short-term use cases that range from minutes to days at a time:

• Planning for major calendar events, such as the tax season, Black Friday, popular media events, and video game launches.
• Meeting cyclical periods of high demand for analytics, like Monday mornings.
• Completing your data warehouse evaluations and dialing in the optimal number of slots to use.

You can also combine on-demand and flat-rate billing models, at the workload level or at the business unit level, by organizing workloads or business units into different projects or folders and assigning each one to a reservation or to on-demand billing.

Note: On-demand pricing gives you access to 2,000 slots or more per Google Cloud project. With flat-rate pricing, you can commit to fewer than 2,000 slots, but your queries might be less performant, depending on your workload demands.

You can use flat-rate in one region and on-demand in another region. By default, all projects use on-demand billing. Within a region, you can opt a project, folder, or organization into flat-rate billing by assigning it to a reservation. For example, if you purchase a slot commitment in the US multi-region and assign your organization to the default reservation, your organization will be on flat- rate billing in the US multi-region, but will remain on on-demand billing in all other regions.

Within a region, you can combine flat-rate and on-demand billing by explicitly assigning projects to a reservation. Any project not assigned to a reservation remains on on-demand billing. You can also explicitly assign a project to use on-demand billing by assigning the reservation ID none. This is useful if you assign a folder or an organization to a reservation, but want some projects within that folder or organization to use on-demand billing. For more information, see Assign a project to None.

Projects with on-demand billing use capacity that's separate from your committed capacity. Those projects do not affect the availability of your committed capacity.

Expired commitments

If you have flat-rate commitments, they will be removed unless a renewal plan is specified. To make sure you don't lose any capacity, the extra slots are moved to the baseline of a system-created reservation called system-created-Enterprise. After the commitments expire, your bill consists of three parts:

1. Remaining commitments.
2. Baseline slots that are not covered by the remaining commitments.
3. Scaled slots managed by autoscaling.

Scenario 1: Commitments are equal to total baseline

You have one commitment that expires with 100 slots, and one reservation with 100 baseline slots.

The 100 slots are removed and you are charged based on the 100 baseline.

Scenario 2: Commitments larger than total baseline

You have one commitment that expires with 200 slots, and one reservation with 100 baseline slots.

The 200 slots are removed and system-created-Enterprise is created with 100 baseline. You are charged based on the total 200 baseline.

Scenario 3: Commitments with Annual flat-rate renewal plan

You have one Annual flat-rate commitment that expires with 100 slots and Annual flat-rate renewal plan.

The 100 slots are moved into an Enterprise Annual commitment with Annual renewal plan.

Creating an administration project

When you create commitments and reservations, they are associated with a Google Cloud project. This project manages the BigQuery Reservations resources, and is the primary source of billing for these resources. This project does not have to be the same project that holds your BigQuery jobs.

Google recommends creating a dedicated project for Reservations resources. This project is called the administration project, because it centralizes the billing and management of your commitments. Give this project a descriptive name like bq-COMPANY_NAME-admin. Then create one or more separate projects to hold your BigQuery jobs.

Only projects within the same Organization resource as the administration project can be assigned to a reservation. If the administration project is not part of an Organization, then only that project can use the slots.

The administration project is billed for the committed slots. Projects that use the slots are billed for storage but not billed for the slots. You can purchase more than one type of plan (for example, monthly and annual) and put the slots into the same administration project.

Note: After you create a commitment, you can't move it to a different administration project.

We recommend creating one administration project for all reservations. Using a single administration project simplifies how you manage billing and allocate slots. We also recommend that you enable the BigQuery Reservations API only on the administration project so that all commitments are managed through this project.

Estimating how many slots to purchase

BigQuery is architected to scale linearly with increased resources. Depending on the workload, incremental capacity is likely to give you incremental benefits. Therefore, choosing the optimal number of slots to purchase depends on your requirements for performance, throughput, and utility.

You can use Flex slots to experiment with the best configuration of slots. For example, you can test your workload with 500 slots, then 1,000, then 1,500, and 2,000, and observe the impact on performance.

You can also examine the current slot usage of your projects, along with the desired monthly price that you want to pay. On-demand workloads have a soft slot cap of 2,000 slots, but it is important to check how many slots are actually being used by your projects by using INFORMATION_SCHEMA.JOBS* views, Cloud Logging, the Jobs API, or BigQuery Audit logs. For more information, see Visualizing slots available and slots allocated.

After you purchase slots and run your workloads for at least seven days, you can use the slot estimator (Preview) to analyze performance and model the effect of adding or reducing slots. For more information, see Estimate slot capacity requirements.

Managing your workloads and departments using reservations

You can use BigQuery Reservations to isolate your committed capacity across workloads, teams, or departments by creating additional reservations and assigning projects to these reservations. A reservation is an isolated pool of resources with the added benefit of being able to take advantage of idle capacity across your entire organization.

For example, you might have a total committed capacity of 1,000 slots and three workload types: data science, ELT, and BI.

• You can create a ds reservation with 500 slots, and assign all relevant Google Cloud projects to the ds reservation.
• You can create an elt reservation with 300 slots, and assign projects you use for ELT workloads to the elt reservation.
• You can create a bi reservation with 200 slots, and assign projects connected to your BI tools to the bi reservation.

Instead of partitioning your capacity across workloads, you might choose to create reservations for individual teams or departments.

Managing reservations in different regions

Reservations are a regional resource. Slots purchased and reservations created in one region cannot be used in any other region. Project, folders, and organizations can all be assigned to reservations in one region and run on- demand in another. To manage reservations in another region, you must change the region in the BigQuery Capacity Management page:

1. In the BigQuery console, click Reservations.
2. Click the Location picker and select a region where you want to manage reservations.
3. After a region is selected, you can purchase slots, create reservations, and assign a project to a reservation.

BigQuery Reservations for complex organizations

BigQuery Reservations is an organization-scoped resource. You purchase committed capacity centrally to be used across your entire organization. You can require individual departments or divisions to manage their BigQuery Reservations by purchasing committed capacity and distributing that capacity across the department or division. The administration project is associated with a Cloud Billing account, which is billed for the capacity.

You can use a distinct Google Cloud organization for each of your departments or divisions. In this scenario, you define an administration project for each organization and manage BigQuery Reservations for that organization from the administration project. There is no sharing of committed or idle capacity across organizations.

Idle and unallocated slots are only shared across reservations created in the same administration project. If you use more than one administration project, slots are not shared across reservations in different administration projects.