Cost management in BigQuery: how to control spending with budgets and custom quotas

In this post, we'll be showing how to manage BigQuery costs with budgets and custom quota - keep reading, or jump directly into tutorials for creating budgets or setting custom quota!

Early in your journey to build or modernize on the cloud, you'll learn that cloud services are often pay-as-you-go; and running analytics on BigQuery is no exception. While BigQuery does offer several pricing models, the default on-demand pricing model (the one most new users start with) charges for queries by the number of bytes processed.

This pricing structure has some major benefits: you only pay for the services you use, and avoid termination charges and up-front fees. However, the elastic nature of BigQuery means that it's important to understand and take advantage of the tools available to help you stay on top of your spending and prevent surprises on your cloud bill.

Budgets and custom quotas are two powerful tools provided by Google Cloud that you can (and I'd argue you should!) use to manage BigQuery costs. So let's dive into how each of these work and help you get started.

Budgets

As your cloud usage grows and changes over time, your costs will change too. Budgets allow you to monitor all of your Google Cloud charges in one place, including BigQuery. They can track both your actual and forecasted spend, and alert you when you're exceeding your defined budgets, which helps you to both avoid unexpected expenses and plan for growth.

Creating a budget

The easiest way to get started is to follow this interactive walkthrough for creating budgets which will take you step-by-step in creating your first budget!

Budgets can be configured for a Cloud Billing account (that can include more than one project linked to the billing account), or for individual projects. To manage budgets for a Cloud Billing account, you need the Billing Account Administrator or Billing Account Costs Manager role on the Cloud Billing account. To manage budgets for an individual project, you need the Project Owner or Project Editor role on the project.

Budgets can be created within the Billing Console in the Budgets & alerts page. At a high-level, you will define the following areas when creating a budget:

• Scope
  • Budgets are a tool that span Google Cloud, and you can scope budgets to apply to the spend in an entire Cloud Billing account, or narrow the scope by filtering on projects, services, or labels. To create a budget focused on BigQuery spend, you can scope it to the BigQuery service. Note that this scope includes both BigQuery on-demand query usage and BigQuery storage.
• Budget amount
  • The budget amount can be a total that you specify, or you can base the budget amount on the previous calendar period's spend.
• Actions
  • After setting the budget amount, you can set multiple threshold rules to trigger email notifications. Each threshold can be customized as a percentage of the total budget amount, and can be based on actual costs (as you're charged for using services) or forecasted costs (as Google Cloud forecasts that you're going to be spending a certain amount). Using forecasted costs can alert you before you actually spend and stay ahead of any issues!

You have several options for creating a budget: you can use the Cloud Console (as shown in the above screenshot), the gcloud command-line tool, or the Cloud Billing API.

Once your budget is in place, email alert notifications will be sent when you hit (or are forecasted to hit) your budget!

Monitoring spending with budgets

In addition to any of the alert actions you choose when setting up a budget, you can monitor your spending against your budgets using the Cloud Billing dashboard or the Budget API. You can see how much of your budget has been consumed, which resources are contributing the most to your costs, and where you might be able to optimize your usage.

Tips for using budgets

• Budget email alerts are sent to users who have Billing Account Administrator and Billing Account User roles by default. While this is a great first step, you can also configure budgets to notify users through Cloud Monitoring, or you can have regular budget updates sent to Pub/Sub for full customization over how you want to respond to budget updates such as sending messages to Slack.
• With the Cloud Billing Budget API, you can view, create, and manage budgets programmatically at scale. This is especially useful if you're creating a large number of budgets across your organization.
• While this blog post focuses on using budgets for BigQuery usage, budgets are a tool that can be used across Google Cloud, so you can use this tool to manage Cloud spend as a whole or target budgets for particular services or projects.

Custom quota

Custom quotas are a powerful feature that allow you to set hard limits on specific resource usage. In the case of BigQuery, quotas allow you to control query usage (number of bytes processed) at a project- or user-level. Project-level custom quotas limit the aggregate usage of all users in that project, while user-level custom quotas are separately applied to each user or service account within a project.

Custom quotas are relevant when you are using BigQuery’s on-demand pricing model, which charges for the number of bytes processed by each query. When you are using the capacity pricing model, you are charged for compute capacity (measured in slots) used to run queries, so limiting the number of bytes processed is less useful.

By setting custom quotas, you can control the amount of query usage by different teams, applications, or users within your organization, preventing unexpected spikes in usage and costs.

Setting custom quotas

Again, the best way to get started with quotas is to follow this interactive walkthrough for setting custom quota which will take you step-by-step!

Note that quotas are set within a project, and you must have the Owner, Editor, or Quota Administrator role on that project in order to set quotas.

Custom quota can be set by heading to the IAM & Admin page of the Cloud console, and then choosing Quotas. This page contains hundreds of various quota, so use the filter functionality with Metric: bigquery.googleapis.com/quota/query/usage to help you zero in on the two quota options for BigQuery query usage:

• Query usage per day <- this is the project-level quota
• Query usage per user per day <- this is the user-level quota

After selecting one or both quotas, click to Edit Quotas. Here you will define your daily limits for each quota in tebibytes (TiB), so be sure to make any necessary conversions.

To set custom quotas for BigQuery, you can use the Cloud Console (as described above), the gcloud command-line tool, or the Service Usage API. You can also monitor your quotas and usage within the Quotas page or using the Service Usage API.

Tips for using custom quotas

• You may use either project-level or user-level of these quota options, or both in tandem. Used in tandem, usage will count against both quotas, and adhere to the stricter of the two limits.
• Once quota is exceeded, the user will receive a usageQuotaExceeded error and the query will not execute. Quotas are proactive, meaning, for example, you can't run an 11 TB query if you have a 10 TB quota.
• Daily quotas reset at midnight Pacific Time.
• Separate from setting a custom quota, you can also set a maximum bytes billed for a specific query (say, one you run on a schedule) to limit query costs.

Differences between budgets and custom quotas

Now that you've learned more about budgets and custom quota, let's look at them side-by-side and note some of their differences:

• Their scope: Budgets are tied to a billing account (which can be shared across projects), while quotas are set for individual projects.
• What they track: Budgets are set for a specific cost amount, while quotas are set for specific resource or service usage.
• How they are enforced: Budgets track your costs and alert you when you're exceeding your budget, while quotas enforce a hard limit on the amount of resources that can be used in a project and will return an error when a user/service tries to exceed the limit.

Next steps

Tracking and analyzing your BigQuery costs will make you feel more at ease when running queries within the on-demand pricing model, and it can help you make informed decisions, optimize your costs, and maximize the value of your cloud spend.

As you scale your BigQuery environment, you may want to move your workloads to the BigQuery editions pricing model, which charges by the amount of capacity allocated to your workload, measured in slots (a unit of measure for BigQuery compute power) rather than per each query. This model also can provide discounted capacity for long term commitments. One of BigQuery’s unique features is the ability to combine the two different pricing models (on-demand and capacity) to optimize your costs.

You can get started with budgets and quota using the in-console walkthroughs I mentioned earlier for creating budgets or setting custom quota. And head to our Cost Management page to learn about more features to help you monitor, control, and optimize your cloud costs.