Using Cached Query Results

This document describes how to use cached results in BigQuery.

Using cached query results

BigQuery writes all query results to a table. The table is either explicitly identified by the user (a destination table), or it is a temporary, cached results table. Temporary, cached results tables are maintained per-user, per- project.

When you run a duplicate query, BigQuery attempts to reuse cached results. To retrieve data from the cache, the duplicate query text must be exactly the same as the original query.

When query results are retrieved from a cached results table, the job statistics property statistics.query.cacheHit returns as true, and you are not charged for the query. Though you are not charged for queries that use cached results, the queries are subject to the BigQuery quota policies. In addition to reducing costs, queries that use cached results are significantly faster because BigQuery does not need to compute the result set.

All query results, including both interactive and batch queries, are cached in temporary tables for approximately 24 hours with some exceptions. Query results are not cached:

  • When a destination table is specified in the job configuration, the web UI, the command line, or the API

  • If any of the referenced tables or logical views have changed since the results were previously cached

  • When any of the tables referenced by the query have recently received streaming inserts (a streaming buffer is attached to the table) even if no new rows have arrived

  • If the query uses non-deterministic functions; for example, date and time functions such as CURRENT_TIMESTAMP() and NOW(), and other functions such as CURRENT_USER() return different values depending on when a query is executed

  • If you are querying multiple tables using a wildcard

  • If the cached results have expired; typical cache lifetime is 24 hours, but the cached results are best-effort and may be invalidated sooner

For query results to persist in a cached results table, the result set must be smaller than the maximum response size. For more information about managing large result sets, see Returning large query results.

Cached result tables have additional constraints on their use, given their special status. You cannot target cached result tables with DML statements. Although current semantics allow it, the use of cached results as input for dependendent jobs is discouraged. For example, you should not submit query jobs that retrieve results from the cache table. Instead, write your results to a named destination table. To enable easy cleanup, features such as the dataset level defaultTableExpirationMs property can expire the data automatically after a given duration.

Disabling retrieval of cached results

The Use cached results option reuses results from a previous run of the same query unless the tables being queried have changed. Using cached results is only beneficial for repeated queries. For new queries, the Use cached results option has no effect, though it is enabled by default.

When you repeat a query with the Use cached results option disabled, the existing cached result is overwritten. This requires BigQuery to compute the query result, and you are charged for the query. This is particularly useful in benchmarking scenarios.

If you want to disable retrieving cached results and force live evaluation of a query job, you can set the configuration.query.useQueryCache property of your query job to false.

To disable the Use cached results option:

Web UI

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose query button.

  3. Enter a valid BigQuery SQL query in the New Query text area.

  4. Click Show Options.

  5. Uncheck Use Cached Results.

Command-line

Use the nouse_cache flag to overwrite the query cache. The following example forces BigQuery to process the query without using the existing cached results:

 bq query --nouse_cache --batch "SELECT name,count FROM mydataset.names_2013 WHERE gender = 'M' ORDER BY count DESC LIMIT 6"

API

To process a query without using the existing cached results, set the useQueryCache property to false.

Java

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

To process a query without using the existing chached results, set use query cache to false when creating a QueryJobConfiguration.

public static void runUncachedQuery(String queryString)
    throws TimeoutException, InterruptedException {
  QueryJobConfiguration queryConfig =
      QueryJobConfiguration.newBuilder(queryString)
          // Do not use the query cache. Force live query evaluation. See:
          // https://cloud.google.com/bigquery/docs/cached-results#disabling_retrieval_of_cached_results
          .setUseQueryCache(false)
          .build();

  runQuery(queryConfig);
}
public static void runQuery(QueryJobConfiguration queryConfig)
    throws TimeoutException, InterruptedException {
  BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

  // Create a job ID so that we can safely retry.
  JobId jobId = JobId.of(UUID.randomUUID().toString());
  Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());

  // Wait for the query to complete.
  queryJob = queryJob.waitFor();

  // Check for errors
  if (queryJob == null) {
    throw new RuntimeException("Job no longer exists");
  } else if (queryJob.getStatus().getError() != null) {
    // You can also look at queryJob.getStatus().getExecutionErrors() for all
    // errors, not just the latest one.
    throw new RuntimeException(queryJob.getStatus().getError().toString());
  }

  // Get the results.
  QueryResponse response = bigquery.getQueryResults(jobId);
  QueryResult result = response.getResult();

  // Print all pages of the results.
  while (result != null) {
    for (List<FieldValue> row : result.iterateAll()) {
      for (FieldValue val : row) {
        System.out.printf("%s,", val.toString());
      }
      System.out.printf("\n");
    }

    result = result.getNextPage();
  }
}

Ensuring use of the cache

If you use the jobs.insert() function to run a query, you can force a query job to fail unless cached results can be used by setting the createDisposition property of the job configuration to CREATE_NEVER.

If the query result does not exist in the cache, a NOT_FOUND error is returned.

Verifying use of the cache

There are two ways to determine if BigQuery returned a result using the cache:

  • If you are using the BigQuery web UI, the result string does not contain information about the number of processed bytes, and displays the word "cached".

  • If you are using the BigQuery API, the cacheHit property in the query result is set to true.

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...