Writing Query Results

This document describes how to write query results.

Temporary and permanent tables

BigQuery saves all query results to a table, which can be either permanent or temporary:

  • A temporary table is a randomly named table saved in a special dataset. Temporary tables are used to cache query results. A temporary table has a lifetime of approximately 24 hours. Temporary tables are not available for sharing, and are not visible using any of the standard list or other table manipulation methods. You are not charged for storing temporary tables.

  • A permanent table can be a new or existing table in any dataset to which you have access. If you write query results to a new table, you are charged for storing the data. When you write query results to a permanent table, the tables you're querying must be in the same location as the dataset that contains the destination table.

Writing query results to a permanent table

When you write query results to a permanent table, you can create a new table, append the results to an existing table, or overwrite an existing table. You can write query results to a permanent table using the BigQuery web UI, the command-line tool's bq query command, or by calling the jobs.insert method and configuring a query job.

Required permissions

The permissions required for writing query results to a permanent table depend on the write disposition of the data.

Permissions for writing query results to a new table

If you are writing query results to a new table, you must have WRITER access at the dataset level, or you must be assigned a project-level IAM role that includes bigquery.tables.create permissions. The following predefined, project-level IAM roles include bigquery.tables.create permissions:

In addition, because the bigquery.user role has bigquery.datasets.create permissions, a user assigned to the bigquery.user role can create tables in any dataset that user creates. When a user assigned to the bigquery.user role creates a dataset, that user is given OWNER access to the dataset. OWNER access to a dataset gives the user full control over it and all the tables in it.

For more information on IAM roles and permissions in BigQuery, see Access Control. For more information on dataset-level roles, see Primitive roles for datasets.

Permissions for overwriting or appending data

If you are using query results to overwrite an existing table or to append data to an existing table, you must have WRITER access at the dataset level, or you must be assigned a project-level IAM role that includes bigquery.tables.updateData permissions. The following predefined, project-level IAM roles include bigquery.tables.updateData permissions:

In addition, because the bigquery.user role has bigquery.datasets.create permissions, a user assigned to the bigquery.user role can overwrite or append data in any table that user creates in the dataset. When a user assigned to the bigquery.user role creates a dataset, that user is given OWNER access to the dataset. OWNER access to a dataset gives the user full control over it and all the tables in it.

For more information on IAM roles and permissions in BigQuery, see Access Control. For more information on dataset-level roles, see Primitive roles for datasets.

Writing query results

To write your query results to a permanent table:

Web UI

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose query button.

  3. Enter a valid BigQuery SQL query in the New Query text area.

  4. Click Show Options.

  5. In the Destination Table section, click Select Table.

  6. In the Select Destination Table dialog:

    1. For Project, choose the project where the destination table will be created.

    2. For Dataset, choose the dataset that will store the table.

    3. In the Table ID field, enter a table name. The name must be unique in the destination dataset. The table name can be up to 1024 characters long and can contain only a-z, A-Z, 0-9, or _ (the underscore character).

    4. Click OK.

  7. In the Destination Table section, for Write Preference, choose one of the following:

    • Write if empty — Writes the query results to the table only if the table is empty.
    • Append to table — Appends the query results to an existing table.
    • Overwrite table — Overwrites an existing table with the same name using the query results.
  8. For Processing Location, click Unspecified and choose your data's location. You can leave processing location set to unspecified if your data is in the US or EU multi-region location. When your data is in the US or the EU, the processing location is automatically detected.

  9. Click Run query. This creates a query job that writes the query results to the table you specified.

Alternately, if you forget to specify a destination table before running your query, you can copy the temporary table to a permanent table by clicking the Save as Table button in the results window.

CLI

Enter the bq query command and specify the --destination_table flag to create a permanent table based on the query results. Specify the use_legacy_sql=false flag to use standard SQL syntax. To write the query results to a table that is not in your default project, add the project ID to the dataset name in the following format: [PROJECT_ID]:[DATASET].

Supply the --location flag and set the value to your location.

To control the write disposition for an existing destination table, specify one of the following optional flags:

  • --append_table — If the destination table exists, the query results are appended to it.
  • --replace — If the destination table exists, it is overwritten with the query results.

    bq --location=[LOCATION] query --destination_table [PROJECT_ID]:[DATASET].[TABLE] --use_legacy_sql=false '[QUERY]'
    

Where:

  • [LOCATION] is the name of the location used to process the query. The --location flag is optional if your data is in the US or the EU multi-region location. For example, if you are using BigQuery in the Tokyo region, set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  • [PROJECT_ID] is your project ID.
  • [DATASET] is the name of the dataset that contains the table to which you are writing the query results.
  • [TABLE] is the name of the table to which you're writing the query results.
  • [QUERY] is a query in standard SQL syntax.

If no write disposition flag is specified, the default behavior is to write the results to the table only if it is empty. If the table exists and it is not empty, the following error is returned: BigQuery error in query operation: Error processing job '[PROJECT_ID]:bqjob_123abc456789_00000e1234f_1': Already Exists: Table [PROJECT_ID]:[DATASET].[TABLE].

Examples:

Enter the following command to write query results to a destination table named mytable in mydataset. The dataset is in your default project. Since no write disposition flag is specified in the command, the table must be new or empty. Otherwise, an Already exists error is returned. The query retrieves data from the USA Name Data public dataset.

bq --location=US query --destination_table mydataset.mytable --use_legacy_sql=false 'SELECT name,number FROM `bigquery-public-data.usa_names.usa_1910_current` WHERE gender = "M" ORDER BY number DESC'

Enter the following command to use query results to overwrite a destination table named mytable in mydataset. The dataset is in your default project. The command uses the --replace flag to overwrite the destination table.

bq --location=US query --destination_table mydataset.mytable --replace --use_legacy_sql=false 'SELECT name,number FROM `bigquery-public-data.usa_names.usa_1910_current` WHERE gender = "M" ORDER BY number DESC'

Enter the following command to append query results to a destination table named mytable in mydataset. The dataset is in myotherproject, not your default project. The command uses the --append flag to append the query results to the destination table.

bq --location=US query --destination_table myotherproject:mydataset.mytable --append --use_legacy_sql=false 'SELECT name,number FROM `bigquery-public-data.usa_names.usa_1910_current` WHERE gender = "M" ORDER BY number DESC'

API

To save query results to a permanent table, call the jobs.insert method, configure a query job, and include a value for the configuration.query.destinationTable property. To control the write disposition for an existing destination table, configure the configuration.query.writeDisposition property.

Specify your location in the location property in the jobReference section of the job resource.

Java

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

To save query results to a permanent table, set the destination table to the desired TableId in a QueryJobConfiguration.

public static void runQueryPermanentTable(
    String queryString,
    String destinationDataset,
    String destinationTable,
    boolean allowLargeResults) throws TimeoutException, InterruptedException {
  QueryJobConfiguration queryConfig =
      QueryJobConfiguration.newBuilder(queryString)
          // Save the results of the query to a permanent table. See:
          // https://cloud.google.com/bigquery/docs/writing-results#permanent-table
          .setDestinationTable(TableId.of(destinationDataset, destinationTable))
          // Allow results larger than the maximum response size.
          // If true, a destination table must be set. See: 
          // https://cloud.google.com/bigquery/docs/writing-results#large-results
          .setAllowLargeResults(allowLargeResults)
          .build();

  runQuery(queryConfig);
}
public static void runQuery(QueryJobConfiguration queryConfig)
    throws TimeoutException, InterruptedException {
  BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

  // Create a job ID so that we can safely retry.
  JobId jobId = JobId.of(UUID.randomUUID().toString());
  Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());

  // Wait for the query to complete.
  queryJob = queryJob.waitFor();

  // Check for errors
  if (queryJob == null) {
    throw new RuntimeException("Job no longer exists");
  } else if (queryJob.getStatus().getError() != null) {
    // You can also look at queryJob.getStatus().getExecutionErrors() for all
    // errors, not just the latest one.
    throw new RuntimeException(queryJob.getStatus().getError().toString());
  }

  // Get the results.
  TableResult result = queryJob.getQueryResults();

  // Print all pages of the results.
  while (result != null) {
    for (List<FieldValue> row : result.iterateAll()) {
      for (FieldValue val : row) {
        System.out.printf("%s,", val.toString());
      }
      System.out.printf("\n");
    }

    result = result.getNextPage();
  }
}

Python

To save query results to a permanent table, create a QueryJobConfig and set the destination to the desired TableReference. Pass the job configuration to the query method.

# client = bigquery.Client()
job_config = bigquery.QueryJobConfig()

# Set the destination table. Here, dataset_id is a string, such as:
# dataset_id = 'your_dataset_id'
table_ref = client.dataset(dataset_id).table('your_table_id')
job_config.destination = table_ref

# The write_disposition specifies the behavior when writing query results
# to a table that already exists. With WRITE_TRUNCATE, any existing rows
# in the table are overwritten by the query results.
job_config.write_disposition = 'WRITE_TRUNCATE'

# Start the query, passing in the extra configuration.
query_job = client.query(
    'SELECT 17 AS my_col;', job_config=job_config)

rows = list(query_job)  # Waits for the query to finish
assert len(rows) == 1
row = rows[0]
assert row[0] == row.my_col == 17

# In addition to using the results from the query, you can read the rows
# from the destination table directly.
iterator = client.list_rows(
    table_ref, selected_fields=[bigquery.SchemaField('my_col', 'INT64')])

rows = list(iterator)
assert len(rows) == 1
row = rows[0]
assert row[0] == row.my_col == 17

Writing large query results

Normally, queries have a maximum response size. If you plan to run a query that might return larger results, you can:

  • In standard SQL, specify a destination table for the query results.
  • In legacy SQL, specify a destination table and set the allowLargeResults option.

When you specify a destination table for large query results, you are charged for storing the data.

Limitations

In legacy SQL, writing large results is subject to these limitations:

  • You must specify a destination table.
  • You cannot specify a top-level ORDER BY, TOP or LIMIT clause. Doing so negates the benefit of using allowLargeResults, because the query output can no longer be computed in parallel.
  • Window functions can return large query results only if used in conjunction with a PARTITION BY clause.

Writing large results using legacy SQL

To write large result sets using legacy SQL:

Web UI

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose query button.

  3. Enter a valid BigQuery SQL query in the New Query text area. Use the #legacySQL prefix or be sure you have Use Legacy SQL checked in the query options.

  4. Click Show Options.

  5. For Destination Table, click Select Table.

  6. In the Select Destination Table dialog:

    1. For Project, choose the project where the destination table will be created.

    2. For Dataset, choose the dataset that will store the table.

    3. In the Table ID field, enter a table name.

    4. Click OK.

  7. If you are writing a large results set to an existing table, you can use the Write Preference option to control the write disposition of the destination table:

    • Write if empty — Writes the query results to the table only if the table is empty.
    • Append to table — Appends the query results to an existing table.
    • Overwrite table — Overwrites an existing table with the same name using the query results.
  8. For Results Size, check Allow Large Results.

    Allow large results option

  9. For Processing Location, click Unspecified and choose your data's location. You can leave processing location set to unspecified if your data is in the US or EU multi-region location. When your data is in the US or the EU, the processing location is automatically detected.

  10. Click Run Query. This creates a query job that writes the large results set to the table you specified.

Command-line

Use the --allow_large_results flag with the --destination_table flag to create a destination table to hold the large results set. Because the --allow_large_results option only applies to legacy SQL, you must also specify the --use_legacy_sql=true flag. To write the query results to a table that is not in your default project, add the project ID to the dataset name in the following format: [PROJECT_ID]:[DATASET]. Supply the --location flag and set the value to your location.

To control the write disposition for an existing destination table, specify one of the following optional flags:

  • --append_table — If the destination table exists, the query results are appended to it.
  • --replace — If the destination table exists, it is overwritten with the query results.

    bq --location=[LOCATION] query --destination_table [PROJECT_ID]:[DATASET].[TABLE_NAME] --use_legacy_sql=true --allow_large_results "[QUERY]"
    

Where:

  • [LOCATION] is the name of the location used to process the query. The --location flag is optional if your data is in the US or the EU multi-region location. For example, if you are using BigQuery in the Tokyo region, set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  • [PROJECT_ID] is your project ID.
  • [DATASET] is the name of the dataset that contains the table to which you are writing the query results.
  • [TABLE] is the name of the table to which you're writing the query results.
  • [QUERY] is a query in legacy SQL syntax.

Examples:

Enter the following command to write large query results to a destination table named mytable in mydataset. The dataset is in your default project. Since no write disposition flag is specified in the command, the table must be new or empty. Otherwise, an Already exists error is returned. The query retrieves data from the USA Name Data public dataset. This query is used for example purposes only. The results set returned does not exceed the maximum response size.

bq --location=US query --destination_table mydataset.mytable --use_legacy_sql=true --allow_large_results "SELECT name,number FROM [bigquery-public-data:usa_names.usa_1910_current] WHERE gender = 'M' ORDER BY number DESC"

Enter the following command to use large query results to overwrite a destination table named mytable in mydataset. The dataset is in myotherproject, not your default project. The command uses the --replace flag to overwrite the destination table.

bq --location=US query --destination_table mydataset.mytable --replace --use_legacy_sql=true --allow_large_results "SELECT name,number FROM [bigquery-public-data:usa_names.usa_1910_current] WHERE gender = 'M' ORDER BY number DESC"

Enter the following command to append large query results to a destination table named mytable in mydataset. The dataset is in myotherproject, not your default project. The command uses the --append flag to append the query results to the destination table.

bq --location=US query --destination_table myotherproject:mydataset.mytable --append --use_legacy_sql=true --allow_large_results "SELECT name,number FROM [bigquery-public-data:usa_names.usa_1910_current] WHERE gender = 'M' ORDER BY number DESC"

API

To write large results to a destination table, call the jobs.insert method, configure a query job, and set the configuration.query.allowLargeResults property to true. Specify the destination table using the configuration.query.destinationTable property. To control the write disposition for an existing destination table, configure the configuration.query.writeDisposition property.

Specify your location in the location property in the jobReference section of the job resource.

Java

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

To enable large results, set allow large results to true and set the destination table to the desired TableId in a QueryJobConfiguration.

public static void runQueryPermanentTable(
    String queryString,
    String destinationDataset,
    String destinationTable,
    boolean allowLargeResults) throws TimeoutException, InterruptedException {
  QueryJobConfiguration queryConfig =
      QueryJobConfiguration.newBuilder(queryString)
          // Save the results of the query to a permanent table. See:
          // https://cloud.google.com/bigquery/docs/writing-results#permanent-table
          .setDestinationTable(TableId.of(destinationDataset, destinationTable))
          // Allow results larger than the maximum response size.
          // If true, a destination table must be set. See: 
          // https://cloud.google.com/bigquery/docs/writing-results#large-results
          .setAllowLargeResults(allowLargeResults)
          .build();

  runQuery(queryConfig);
}
public static void runQuery(QueryJobConfiguration queryConfig)
    throws TimeoutException, InterruptedException {
  BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

  // Create a job ID so that we can safely retry.
  JobId jobId = JobId.of(UUID.randomUUID().toString());
  Job queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());

  // Wait for the query to complete.
  queryJob = queryJob.waitFor();

  // Check for errors
  if (queryJob == null) {
    throw new RuntimeException("Job no longer exists");
  } else if (queryJob.getStatus().getError() != null) {
    // You can also look at queryJob.getStatus().getExecutionErrors() for all
    // errors, not just the latest one.
    throw new RuntimeException(queryJob.getStatus().getError().toString());
  }

  // Get the results.
  TableResult result = queryJob.getQueryResults();

  // Print all pages of the results.
  while (result != null) {
    for (List<FieldValue> row : result.iterateAll()) {
      for (FieldValue val : row) {
        System.out.printf("%s,", val.toString());
      }
      System.out.printf("\n");
    }

    result = result.getNextPage();
  }
}

Downloading and saving query results

After you run a SQL query, you can download the results to a file on your local machine, you can save the results to Google Sheets, or you can save the results to a permanent table in BigQuery.

Limitations

Downloading and saving query results are subject to the following limitations:

  • You can download query results only to a local file.
  • To download query results, the results set must contain fewer than 16,000 rows, and it must be 10 MB or smaller. If your results are larger than 10 MB or 16,000 rows you can save them to a table.
  • You can download query results only in CSV or newline-delimited JSON format.
  • You cannot download query results containing nested and repeated data in CSV format.
  • You cannot save query results containing nested and repeated data to Google Sheets.
  • When you save query results to Google Sheets, the results set must contain fewer than 16,000 rows, and it must be 10 MB or smaller. If your results are larger than 10 MB or 16,000 rows you can save them to a table instead.

Downloading query results

To download query results as a CSV or newline-delimited JSON file:

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose Query button.

  3. Enter a valid BigQuery SQL query in the New Query text area.

  4. Click Show Options.

  5. For Processing Location, click Unspecified and choose your data's location. You can leave processing location set to unspecified if your data is in the US or EU multi-region location. When your data is in the US or the EU, the processing location is automatically detected.

  6. Click Run Query.

  7. When the results are returned, click the Download as CSV or Download as JSON button above the query results.

    screenshot of download and save buttons

    The file is downloaded to your browser's default download location.

Saving query results to a table

To save query results as a table:

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose Query button.

  3. Enter a valid BigQuery SQL query in the New Query text area.

  4. Click Show Options.

  5. For Processing Location, click Unspecified and choose your data's location. You can leave processing location set to unspecified if your data is in the US or EU multi-region location. When your data is in the US or the EU, the processing location is automatically detected.

  6. Click Run Query.

  7. When the results are returned, click the Save as Table button above the query results.

    screenshot of download and save buttons

  8. In the Copy Table dialog:

    • For Destination project, choose the project that will store the query results.
    • For Destination dataset, select the dataset where you want to store the query results. The source and destination datasets must be in the same location.
    • For Destination table, enter a name for the new table. The name must be unique in the destination dataset. The table name can be up to 1024 characters long and can contain only a-z, A-Z, 0-9, or _ (the underscore character). You cannot overwrite an existing table in the destination dataset using the BigQuery web UI.

      Table copy

    • Click OK.

  9. After clicking OK, a copy job is automatically generated to create the new table by copying the temporary (cache) table. When they job completes, you can see the new table in the UI's navigation pane.

Saving query results to Google Sheets

To save query results to Google Sheets:

  1. Go to the BigQuery web UI.
    Go to the BigQuery web UI

  2. Click the Compose Query button.

  3. Enter a valid BigQuery SQL query in the New Query text area.

  4. Click Show Options.

  5. For Processing Location, click Unspecified and choose your data's location. You can leave processing location set to unspecified if your data is in the US or EU multi-region location. When your data is in the US or the EU, the processing location is automatically detected.

  6. Click Run Query.

  7. When the results are returned, click the Save to Google Sheets button above the query results.

    screenshot of download and save buttons

  8. If necessary, follow the prompts to log into your Google account and click Allow to give BigQuery permission to write the data to your Google Drive MY Drive folder.

    After following the prompts, you should receive an email with the subject "BigQuery Client Tools connected to your Google Account". The email contains information on the permissions you granted along with steps to remove the permissions.

  9. When the results are saved, a message like the following appears above the query results in the BigQuery web UI: Results saved to Google Sheets. Click to view. Click the link in the message to view your results in Google Sheets, or navigate to your MY Drive folder and open the file manually.

    When you save query results to Google Sheets, the file name begins with results-[DATE] where [DATE] is today's date in the format YYYYMMDD.

Send feedback about...