This page describes how to export and import Cloud Firestore in Datastore mode entities using
the managed export and import service. The managed export and import service
is available through the gcloud command-line tool and the
Cloud Datastore Admin API (REST, RPC).
With the managed export and import service, you can recover from accidental deletion of data and export data for offline processing. You can export all entities or just specific kinds of entities. Likewise, you can import all data from an export or only specific kinds. As you use the managed export and import service, consider the following:
The export service uses eventually consistent reads. You cannot assume an export happens at a single point in time. The export might include entities written after the export begins and exclude entities written before the export begins.
An export does not contain any indexes. When you import data, the required indexes are automatically rebuilt using your database's current index definitions. Per-entity property value index settings are exported and honored during import.
Imports do not assign new IDs to entities. Imports use the IDs that existed at the time of the export and overwrite any existing entity with the same ID. During an import, the IDs are reserved during the time that the entities are being imported. This feature prevents ID collisions with new entities if writes are enabled while an import is running.
If an entity in your database is not affected by an import, it will remain in your database after the import.
Data exported from one Datastore mode database can be imported into another Datastore mode database, even one in another project.
The managed export and import service limits the number of concurrent exports and imports to 50 and allows a maximum of 20 export and import requests per minute for a project.
The output of a managed export uses the LevelDB log format.
To import only a subset of entities or to import data into BigQuery, you must specify an entity filter in your export.
Before you begin
Before you can use the managed export and import service, you must complete the following tasks.
Enable billing for your Google Cloud Platform project. Only GCP projects with billing enabled can use the export and import functionality.
Create a Cloud Storage bucket in the same location as your Cloud Firestore in Datastore mode database location. All exports and imports rely on Cloud Storage and you must use the same location for your Cloud Storage bucket and Cloud Firestore in Datastore mode database. You cannot use a Requester Pays bucket for export and import operations.
Assign an IAM role to your user account that grants the
datastore.databases.exportpermission, if you are exporting data, or thedatastore.databases.importpermission, if you are importing data. TheCloud Datastore Import Export Adminrole, for example, grants both permissions.If the Cloud Storage bucket is in another project, give your project's default services account access to the bucket.
Set up your environment
Before you export or import data, you must set up environment variables for the
gcloud tool and authenticate using your user account.
Set an environment variable for your GCP project ID.
PROJECT_ID="YOUR_PROJECT_ID"Use this variable to set your project as the active configuration for the
gcloudtool.gcloud config set project ${PROJECT_ID}Authenticate using the
gcloudtool.gcloud auth loginSet an environment variable for your Cloud Storage bucket ID.
BUCKET="YOUR_BUCKET_NAME[/NAMESPACE_PATH]"where
YOUR_BUCKET_NAMEis the name of the Cloud Storage bucket andNAMESPACE_PATHis an optional Cloud Storage namespace path (this is not a Datastore mode namespace). For more information about Cloud Storage namespace paths, see Object name considerations.
Starting managed export and import operations
This section describes how to start a managed export or import operation and how to check on its progress.
Exporting all entities
Use the gcloud datastore export
command to export all entities in your database. You can add
the --async flag to prevent the gcloud tool from waiting for
the operation to complete.
gcloud
gcloud datastore export gs://${BUCKET} --async
Protocol
curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://datastore.googleapis.com/v1/projects/${PROJECT_ID}:export \
-d '{
"outputUrlPrefix": "gs://'${BUCKET}'",
}'
Exporting specific kinds or namespaces
To export a specific subset of kinds and/or namespaces, provide an entity filter with values for kinds and namespace IDs.
gcloud
gcloud datastore export --kinds="KIND1,KIND2" --namespaces="(default),NAMESPACE2" gs://${BUCKET} --async
Protocol
curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://datastore.googleapis.com/v1/projects/${PROJECT_ID}:export \
-d '{
"outputUrlPrefix": "gs://'${BUCKET}'",
"entityFilter": {
"kinds": ["KIND1", "KIND2"],
"namespaceIds": ["", "NAMESPACE2"],
},
}'
For the default namespace, use the empty string element "" for the namespace ID.Export metadata files
An export operation creates a metadata file for each namespace-kind pair
specified. Metadata files are typically named
[NAMESPACE_NAME]_[KIND_NAME].export_metadata. However, if a namespace or kind
would create an invalid Cloud Storage object name, the file will be named
export[NUM].export_metadata.
The metadata files are protocol buffers and can be decoded with the
protoc protocol compiler.
For example, you can decode a metadata file to determine the namespace
and kinds the export files contain:
protoc --decode_raw < export0.export_metadata
Importing all entities
Use the gcloud datastore import
command to import all entities previously exported with the managed
export service. You can add the --async flag to
prevent the gcloud tool from waiting for the operation to complete.
gcloud
gcloud datastore import gs://${BUCKET}/[PATH]/[FILE].overall_export_metadata --async
Protocol
curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://datastore.googleapis.com/v1/projects/${PROJECT_ID}:import \
-d '{
"inputUrl": "gs://'${BUCKET}'/[PATH]/[FILE].overall_export_metadata",
}'
You can determine the value to use for the import location by using the
Cloud Storage UI in the Google Cloud Platform Console to view the bucket, or by
examining the gcloud datastore export output or ExportEntitiesResponse
after your export is complete. Here's an example value of an import location:
gcloud
gs://${BUCKET}/2017-05-25T23:54:39_76544/2017-05-25T23:54:39_76544.overall_export_metadata --async
Protocol
"outputUrl": "gs://'${BUCKET}'/2017-05-25T23:54:39_76544/2017-05-25T23:54:39_76544.overall_export_metadata",
Importing specific kinds or namespaces
To import a specific subset of kinds and/or namespaces, provide an entity filter with values for kinds and namespace IDs.
You can specify kinds and namespaces only if the export files were created with an entity filter. You can not import a subset of kinds and namespaces from an export of all entities.
gcloud
gcloud datastore import --kinds="KIND1,KIND2" --namespaces="(default),NAMESPACE2" gs://${BUCKET}/[PATH]/[FILE].overall_export_metadata --async
Protocol
curl \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://datastore.googleapis.com/v1/projects/${PROJECT_ID}:import \
-d '{
"inputUrl": "gs://'${BUCKET}'/[PATH]/[FILE].overall_export_metadata",
"entityFilter": {
"kinds": ["KIND1", "KIND2"],
"namespaceIds": ["", "NAMESPACE2"],
},
}'
For the default namespace, use the empty string element "" for the namespace ID.Asynchronous exports or imports
Exports and imports can take a long time. When you perform an export or import,
you can provide the --async flag to prevent the gcloud tool from waiting for
the operation to complete.
After you start an export or import operation, you can use the identifier
returned by the gcloud tool to check the status of the operation. For example:
gcloud datastore operations describe ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI
If you forget the --async flag, you can also use Ctrl+c to stop waiting on
an operation. Typing Ctrl+c will not cancel the operation.
Managing long-running operations
Long-running operations are method calls that may take a substantial amount of time to complete. Datastore mode databases create long-running operations when you export or import data.
For example, when you start an export, your Datastore mode database creates a long-running operation to track the export status. Here's the output from the start of an export:
{
"name": "projects/[YOUR_PROJECT_ID]/operations/ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI",
"metadata": {
"@type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata",
"common": {
"startTime": "2017-05-25T23:54:39.583780Z",
"operationType": "EXPORT_ENTITIES"
},
"progressEntities": {},
"progressBytes": {},
"entityFilter": {
"namespaceIds": [
""
]
},
"outputUrlPrefix": "gs://[YOUR_BUCKET_NAME]"
}
}
The value of the name field is the ID of a long-running operation.
Cloud Firestore in Datastore mode provides an operations Admin API to check on the status of long-running operations, as well as cancel, delete, or list long-running operations:
| Method | Description |
|---|---|
| projects.operations.cancel | Cancel a long-running operation. |
| projects.operations.delete | Delete a long-running operation. Note: Deleting an operation does not cancel it. |
| projects.operations.get | Get the status of a long-running operation. |
| projects.operations.list | List long-running operations. |
Listing long-running operations
To list long-running operations, use the gcloud datastore operations list command:
gcloud
gcloud datastore operations list
Protocol
curl
-H "Authorization: Bearer $(gcloud auth print-access-token)"
https://datastore.googleapis.com/v1/projects/${PROJECT_ID}/operations
This example output shows a recently completed export operation. Operations are accessible for a few days after completion:
{
"operations": [
{
"name": "projects/[YOUR_PROJECT_ID]/operations/ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI",
"metadata": {
"@type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata",
"common": {
"startTime": "2017-12-05T23:01:39.583780Z",
"endTime": "2017-12-05T23:54:58.474750Z",
"operationType": "EXPORT_ENTITIES"
},
"progressEntities": {
"workCompleted": "21933027",
"workEstimated": "21898182"
},
"progressBytes": {
"workCompleted": "12421451292",
"workEstimated": "9759724245"
},
"entityFilter": {
"namespaceIds": [
""
]
},
"outputUrlPrefix": "gs://[YOUR_BUCKET_NAME]"
},
"done": true,
"response": {
"@type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesResponse",
"outputUrl": "gs://[YOUR_BUCKET_NAME]/2017-05-25T23:54:39_76544/2017-05-25T23:54:39_76544.overall_export_metadata"
}
}
]
}
Use the input_url value when you import the entities.
Estimating the completion time
As your operation runs, see the value of the state field
for the overall status of the operation.
A request for the status of a long-running operation also returns the metrics
workEstimated and workCompleted. Each of these metrics is returned in both
number of bytes and number of entities. workEstimated shows the estimated
total number of bytes and entities an operation will process, based on
database statistics. workCompleted shows the number of
bytes and entities processed so far. After the operation completes,
workCompleted reflects the total number of bytes and entities that were
actually processed, which might be larger than the value of workEstimated.
Divide workCompleted by workEstimated for a rough progress estimate. The
estimate might be inaccurate because it depends on delayed statistics
collection.
For example, here is the progress status of an export operation:
{
"operations": [
{
"name": "projects/[YOUR_PROJECT_ID]/operations/ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI",
"metadata": {
"@type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata",
...
"progressEntities": {
"workCompleted": "1",
"workEstimated": "3"
},
"progressBytes": {
"workCompleted": "85",
"workEstimated": "257"
},
...
When an operation is done, the operation description will contain "done":
true. See the value of the state field for
the result of the operation. If the done field is not set in the response,
then its value is false. Do not depend on the existence of the done value
for in-progress operations.
Billing and pricing for managed exports and imports
You are required to enable billing for your Google Cloud Platform project before you use the managed export and import service. Export and import operations contribute to your Google Cloud Platform costs in the following ways:
- Entity reads and writes performed by export and import operations count towards your Cloud Firestore in Datastore mode costs.
- Output files stored in Cloud Storage count towards your Cloud Storage data storage costs.
The costs of export and import operations do not count towards the App Engine spending limit. Export or import operations will not trigger any Google Cloud Platform budget alerts until after completion. Similarly, reads and writes performed during an export or import operation are applied to your daily quota after the operation is complete.
Permissions
To run export and import operations, your user account and your project's default service account require the Cloud Identity and Access Management permissions described below.
User account permissions
The user account or service account initiating the operation
requires the datastore.databases.export and datastore.databases.import
Cloud IAM permissions. If you are the project owner, your account has
the required permissions. Otherwise, the following Cloud IAM roles
grant the necessary permissions:
- Cloud Datastore Owner
- Cloud Datastore Import Export Admin
A project owner can grant one of these roles by following the steps in Grant access.
Default service account permissions
Each GCP project automatically creates a default service account
named PROJECT_ID@appspot.gserviceaccount.com. Export and
import operations use this service account to authorize Cloud Storage
operations.
Your project's default service account requires access to the Cloud Storage bucket used in an export or import operation. If your Cloud Storage bucket is in the same project as your Datastore mode database, then the default service account has access to the bucket by default.
If the Cloud Storage bucket is in another project, then you must give the default service account access to the Cloud Storage bucket.
Assign roles to the default service account
You can use the gsutil command-line tool to assign one of the roles below. For example, to assign the Storage Admin role to the default service account run:
gsutil iam ch serviceAccount:[PROJECT_ID]@appspot.gserviceaccount.com:roles/storage.admin \
gs://[BUCKET_NAME]
Alternatively, you can assign this role using the GCP Console.
Export operations
For export operations involving a bucket in another project, modify the permissions of the bucket to assign one of the following Cloud Storage roles to the default service account of the project containing your Datastore mode database:
- Storage Admin
- Storage Object Admin
- Storage Legacy Bucket Writer
You can also create a Cloud IAM custom role with the following permissions:
storage.buckets.getstorage.objects.createstorage.objects.list
Import operations
For import operations involving a Cloud Storage bucket in another project, modify the permissions of the bucket to assign one of the following Cloud Storage roles to the default service account of the project containing your Datastore mode database:
- Storage Admin
- Both Storage Object Viewer and Storage Legacy Bucket Reader
You can also create a Cloud IAM custom role with the following permissions:
storage.buckets.getstorage.objects.get
Differences from Cloud Datastore Admin backups
If you previously used the Cloud Datastore Admin console for backups, you should note the following differences:
There is no GUI for the managed export and import service.
Exports created by a managed export do not appear in the Cloud Datastore Admin console. Managed exports and imports are a new service that does not share data with App Engine's backup and restore functionality, which is administered through the GCP Console.
The managed export and import service does not support the same metadata as the Cloud Datastore Admin backup and does not store progress status in your database. For information on checking the progress of export and import operations, see Managing long-running operations
You cannot view service logs of managed export and import operations.
The managed import service is backwards compatible with Cloud Datastore Admin backup files. You can import a Cloud Datastore Admin backup file using the managed import service, but you cannot import the output of a managed export using the Cloud Datastore Admin console.
Importing into BigQuery
To import data from a managed export into BigQuery, see Loading Cloud Datastore export service data.
Limitations
- Data exported without specifying an entity filter cannot be loaded into BigQuery. If you want to import data into BigQuery, your export request must include one or more kind names in the entity filter.
