Introduction to reservations

BigQuery offers two models of workload management for data analysis (compute). The following models define how you manage workloads and how you are billed:

  • On-demand pricing: By default, you are billed for the data scanned by your queries. You have a fixed, per-project query processing capacity, and your cost is based on the number of bytes processed by each query.

  • Capacity-based pricing: You can switch to a reservation model in which you purchase dedicated or autoscaling query processing capacity at a discounted rate. Capacity is measured in slots, over a period of time. You then allocate reserved query capacity to different workloads or parts of your organization. There is no charge for bytes processed when using the capacity-based model.

Creating a capacity commitment is optional when working with reservations that use BigQuery editions, but it can reduce costs for steady-state workloads.

You can combine both billing models. For example, you might run some workloads with on-demand pricing and others with capacity-based pricing. As the billing model is specified per project, this would require that you use multiple projects for your query jobs.

Benefits of reservations

The benefits of using BigQuery reservations include:

  • Predictability. Capacity-based pricing offers predictable and consistent costs. You specify your maximum cost budget up front and can also take advantage of slot commitments, which offer dedicated ongoing capacity at a discounted rate.

  • Flexibility. You choose how much dedicated capacity to allocate to a workload or let BigQuery automatically scale capacity based on your workload requirements. You are billed for the slots consumed at a minimum of one-second increments.

  • Workload management. Each workload has a specified pool of BigQuery computational resources available for use. At the same time, if a workload doesn't use all of its dedicated slots, any unused slots are shared automatically across your other workloads.

  • Centralized purchasing: You can purchase and allocate slots for your entire organization. You don't need to purchase slots for each project that uses BigQuery.

Reservations

BigQuery capacity commitments are purchased in reservations and measured in slots. Slots represent virtual CPUs that are used to query data. In general, more slots let you run more concurrent queries, and your complex queries can run faster.

Slots are allocated in pools called reservations. Reservations let you allocate the slots in ways that make sense for your particular organization. For example, you might create a reservation named prod for production workloads, and a separate reservation named test for testing, so that test jobs don't compete for production workloads resources. Or, you might create reservations for different departments in your organization.

Reservations can include baseline slots, which are always allocated, and autoscaled slots, which are added or removed dynamically based on the demands of your workload.

If you purchase slot commitments before creating a reservation, then a reservation named default is created automatically as a convenience. The default reservation has no special behavior. You can create additional reservations if needed or use the default reservation.

To use the slots that you allocate, you must assign one or more projects to a reservation, as described in the next section. A reservation is the lowest level at which you can specify slot allocation. Slot allocation within a reservation is handled by the BigQuery scheduler.

Reservation assignments

To use the slot capacity that you've purchased, you must assign projects, folders, or organizations to a reservation. When a job in a project runs, it uses slots from its assigned reservation. Resources can inherit assignments from their parents in the resources hierarchy. Even if a project is not assigned to a reservation, it inherits the assignment of the parent folder or organization, if any. If a project doesn't have an assigned or inherited reservation, then the job uses on-demand pricing. For more information about the resource hierarchy, see Organizing BigQuery resources.

Resources can be assigned to None to represent an absence of an assignment. Projects that are assigned to None always use on-demand pricing. A common use case for None assignments is to assign an organization to a reservation and then use None to opt certain projects or folders out of that reservation. For more information, see Assign a project to None.

When you create an assignment, you specify the job type for that assignment:

  • QUERY: Use this reservation for query jobs, including SQL, DDL, DML, and BigQuery ML queries.

  • PIPELINE: Use this reservation for load and extract jobs.

    By default, load and extract jobs are free and use a shared pool of slots. BigQuery does not guarantee capacity availability for this shared pool or the throughput you see. If you are loading large amounts of data, your job might wait for slots to become available. In that case, you might want to purchase dedicated slots and assign PIPELINE jobs to them. As a best practice, create an additional, dedicated reservation with Ignore idle slots enabled. For more information about idle slots, see Idle slots.

    When load and extract jobs are assigned to a reservation, they lose access to the free pool. You should Monitor resource utilization and jobs to ensure your reservations have enough capacity to perform better than using the free pool.

  • BACKGROUND: Use this reservation when you choose to use your own reservation to run your BigQuery search index management jobs or BigQuery change data capture (CDC) background jobs. Also use this reservation when you replicate source databases to BigQuery with Datastream's background apply operations. BACKGROUND reservations are not available in Standard edition.

  • ML_EXTERNAL: Use this reservation for BigQuery ML queries that use services that are external to BigQuery. For more information, see Assign slots to BigQuery ML workloads. ML_EXTERNAL reservations are not available in Standard edition.

You can't allocate slots to specific assignments. The BigQuery scheduler handles slot allocation for assignments in a reservation.

Commitments

Reservations concepts.

A capacity commitment is a purchase of slots in 100-slot increments for a minimum duration of time, up to your slot quota. Capacity commitments are optional for reservations created with an edition, but can save on costs for steady-state workloads. There is no limit on the number of commitments that you can create. You are charged from the moment that your commitment purchase is successful. For current pricing information, see capacity commitment pricing.

  • Three-year commitment. You purchase a three year commitment. You can choose whether to renew or convert to a different type of commitment plan after 3*365 days.

  • Annual commitment. You purchase a 365-day commitment. You can choose whether to renew or convert to a different type of commitment plan after 365 days.

Whichever plan you select, your slots don't expire at the end of the commitment period. You keep the slots and are billed for them until you delete the commitment. You can also change the plan type after the minimum duration. After the expiration of the plan, you can delete the commitment at any time, or continue to pay based on the conditions of the renewal plan. For more information about deleting commitments, see Delete a commitment.

You are billed monthly for annual or three-year commitment plans. However, your financial commitment is for the entire commitment period and can't be canceled on a monthly basis. Your usage is updated in the billing report daily, and can be viewed at any time.

Slots are subject to capacity availability. When you attempt to purchase slot commitments, success of your purchase is not guaranteed. However, once your commitment purchase is successful, your capacity is guaranteed until you delete the commitment.

Renew commitments

You select a renewal plan when you purchase a commitment plan. You can change the renewal plan for a commitment at any time until it expires, and the following available rewal plans are available:

  • None. After the end of the commitment, the commitment is removed. Reservations are not affected.
  • Annual. After the end of the commitment, your commitment renews for another year.
  • Three-Year. After the end of the commitment, your commitment renews for another three years.

If you use the autoscaling capacity model, you can renew the annual or three-year commitment plan to either one year or three years. For information about purchasing and renewing commitments, see Create a capacity commitment.

For example, if you purchased an annual commitment at 6:00 PM on October 5, 2019, then you started being charged at that second. You can delete or renew the commitment after 6:00 PM on October 4, 2020, noting that 2020 is a leap year. You can change the renewal plans prior to 6:00 PM on October 4, 2020 as follows:

  • If you choose to renew annually, then at 6:00 PM on October 4, 2020, your commitment renews for another year.
  • If you choose to renew to a three-year commitment, then at 6:00 PM on October 4, 2020, your commitment renews for three years.

Delete commitments

After you create a commitment, you can delete it only after the commitment expires. To delete an annual commitment, set its renewal plan to NONE. After the annual commitment expires, it is removed automatically. To learn more about how to delete a commitment, see Delete a commitment.

If you accidentally purchase a commitment or made a mistake when you configured your commitment, contact Cloud Billing Support for help.

Slot allocation within reservations

BigQuery allocates slot capacity within a single reservation using an algorithm called fair scheduling.

The BigQuery scheduler enforces the equal sharing of slots among projects with running queries within a reservation, and then within jobs of a given project. The scheduler provides eventual fairness. During short periods, some jobs might get a disproportionate share of slots, but the scheduler eventually corrects this. The goal of the scheduler is to find a balance between aggressively evicting running tasks (which results in wasting slot time) and being too lenient (which results in jobs with long running tasks getting a disproportionate share of the slot time).

If an important job consistently needs more slots than it receives from the scheduler, consider creating an additional reservation with a guaranteed number of slots and assigning the job to that reservation. For more information, see Workload management.

Idle slots

At any given time, some slots might be idle. This can include:

  • Slot commitments that are not allocated to any reservation.
  • Slots that are allocated to a reservation baseline but aren't in use.

By default, queries running in a reservation automatically use idle slots from other reservations within the same administration project. That means a job can always run as long as there's capacity. Idle capacity is immediately preemptible back to the original assigned reservation as needed, regardless of the priority of the query that needs the resources. This happens automatically in real time.

To prevent this behavior and ensure a reservation only uses its provisioned slots provisioned, set ignore_idle_slots to true. Reservations with ignore_idle_slots set to true can, however, share their idle slots with other reservations.

You cannot share idle slots between reservations of different editions. You can share only the baseline slots or committed slots. Autoscaled slots might be temporarily available but are not shareable because they might scale down.

As long as ignore_idle_slots is false, a reservation can have a slot count of 0 and still have access to unused slots. If you use only the default reservation, toggle off ignore_idle_slots as a best practice. You can then assign a project or folder to that reservation and it will only use idle slots.

Assignments of type ML_EXTERNAL are an exception in that slots used by BigQuery ML external model creation jobs are not preemptible. The slots in a reservation with both ML_EXTERNAL and QUERY assignment types are only available for other query jobs when the slots are not occupied by the ML_EXTERNAL jobs. Moreover, these jobs cannot use idle slots from other reservations.

Limitations

  • Reservations that you buy cannot be shared with other organizations.
  • You must create a separate reservation and a separate administration project for each organization.
  • Each organization can have a maximum of 5 administration projects with active commitments in a single location.
  • Idle capacity cannot be shared between organizations or between different administration projects within a single organization.
  • Commitments are a regional resource. Commitments purchased in one region or multi-region cannot be used in any other regions or multi-regions. Commitments cannot be moved between regions or between regions and multi-regions.
  • Commitments purchased in one administration project cannot be moved to a different administration project.
  • Commitments purchased with one edition cannot be used with reservations of another edition.
  • Idle slots are not shared between reservations of different editions.
  • Autoscaled slots are not shareable as they will scale down when no longer required.

Quotas

Your slot quota is the maximum number of slots you can purchase in a location. You are not billed for quotas; you are only billed for reservations and commitments. For more information, see reservations quotas and limits. For information about increasing your slot quota, see Requesting a quota increase.

Pricing

For information about pricing for reservations, see capacity-based pricing.

What's next