Spanner import and export overview

Overview

Spanner Import and Export functionality helps you move data in bulk into or out of Spanner databases using Dataflow. You can export any Spanner database into a Cloud Storage bucket using either Avro or CSV file formats. You can also import data from Avro or CSV files into a new Spanner database.

Use cases

Use cases of Import and Export functionality include:

  • Bulk loading: You can use import functionality to load data in bulk into Spanner.

  • Long-term backup and archiving: You can export your database at any time and store it in a Cloud Storage bucket location of your choice for long-term backup or archiving. In addition, you can use point-in-time recovery to export a database from a specific past timestamp.

    Note that Spanner also offers managed backup and restore for faster disaster-recovery use cases.

  • Copying databases to development or test projects: You can export a database from a production project and then import it into your development or test project to use for integration tests or other experiments.

  • Ingesting for analytics: You can use database export to ingest your operational data in bulk to analytics services such as BigQuery. BigQuery can automatically ingest data in Avro format from a Cloud Storage bucket, making it easier for you to run analytics on your operational data.

    Note that if you want to use BigQuery for real-time analysis of Spanner data without the need to copy or move that data, you can use Spanner federated queries instead.

File format comparison

The following table compares the capability differences between Avro and CSV file formats when importing and exporting Spanner data.

For detailed instructions on importing from or exporting to these formats, including information on limitations that may apply, please see What's Next, below.

Capability Avro format CSV format
Import or Export an entire database Yes No
Ability to export only selected tables in a database Yes* Yes†
Ability to import previously exported table/tables Yes* Yes†
Export at a past timestamp Yes Yes
Import or Export using Google Cloud CLI Yes Yes
Import or Export using the Dataflow page of Google Cloud console Yes Yes
Import or Export using the Spanner page of Google Cloud console Yes* No

* See "Notes about Avro import and export", below.
† See "Notes about CSV import and export", below.

Notes about Avro import and export

When exporting to Avro format, you can specify a list of tables to export. Any child tables exported this way need to be accompanied by their parent tables. Spanner maintains the entire database schema in the exported files.

When importing from Avro format, Spanner recreates the exported database's whole schema, including all tables. Tables included in the original export receive all their exported data; all other tables remain empty.

The Spanner page of the Google Cloud console offers limited Avro-format import and export options. For example, it does not make network and subnetwork options available. For a wider set of options, use the Dataflow page instead.

Notes about CSV import and export

Spanner limits you to exporting only a single table at a time into CSV format. When you export, the schema is not exported, only the data is exported.

Before importing from CSV files, you need to create a JSON Manifest manually.

Pricing

There are no additional charges from Spanner for use of the export or import tools; you pay the standard rates for data storage when you import a database to Spanner. However, there are other potential charges associated with importing and exporting databases. Refer to the Spanner pricing page for more details.

What's next