Google Cloud Platform
Python

Managing Datastore from the Google Cloud Platform Console

Python |Java |PHP |Go

You can manage your project's datastore from the Cloud Platform Console.

Datastore dashboard

You can view data for the entities in your application's Datastore, as well as statistics for the built-in and composite indexes from the Cloud Datastore Dashboard. The Statistics page displays data in various ways:

  • A pie chart that shows datastore space used by each property type (string, double, blob, etc.).

  • A pie chart showing datastore space by entity kind.

  • A table with the total space used by each property type. The "Metadata" property type represents space consumed by storing properties inside an entry that is not used by the properties directly. The "Datastore Stats" entity, if any, shows the space consumed by the statistics data itself in your datastore.

  • A table showing total size, average size, entry count, and the size of all entities, and the built-in and composite indexes.

By default, the pie charts display statistics for all entities. You can restrict the pie charts to a particular entity kind by choosing from the drop- down menu.

The statistics data is stored in your app's datastore. To make sure there's room for your app's data, Google App Engine will only store statistics if they consume less than 10 megabytes if namespaces are not used, or 20 megabytes if namespaces are used. If your app's stats go over the limit, kind-specific stats are not emitted. If subsequently these "non-kind" stats exceed the limits, then no stats are updated until stats storage drops below the size limitation. (Any previously reported statistics will remain.) You can determine whether this is happening by looking at the timestamps on stat records. If the timestamp gets older than 2 days, and stats in other applications are being updated regularly then this can indicate that you have run into stat storage size limits for your app.

The space consumed by the statistics data increases in proportion to the number of different entity kinds and property types used by your app. The more different entities and properties used by your app, the more likely you are to reach the stat storage limit. Also, if you use namespaces, remember that each namespace contains a complete copy of the stats for that namespace.

Indexes

The Indexes page shows a table of all indexes. You can create a new entity in this page.

Entities

The Entities page lets you select an entity kind and construct a query by applying filters. You can also create a new entity in this page.

Backup/restore, copy, and delete data

The Admin page is used to access the Datastore Admin screen. From there you can perform bulk operations on the entities in your datastore.

Enabling and disabling datastore writes

On the Admin page you'll find a button to either disable or enable datastore writes, depending on the state of your project.

Enabling Datastore Admin for an application

In order to use the Datastore Admin features for your project, you must click Enable Datastore Admin to enable it. (If Datastore Admin is already enabled, the button will read Disable Datastore Admin.)

Caveats on using data admin features

  • For copy, delete, and backups, recent updates may not be considered.
  • All Datastore Admin operations occur within your applications, and thus count against your quota.
  • We strongly recommend that you set your application to read-only mode during a backup or restore.
  • When copying entities to a remote target or restoring from backup all entities with the same keys will be overridden. Operations can be performed multiple times without the risk of creating duplicates. Be aware that the copy/restore operations do not delete extra data.
  • If a non-default queue is chosen for backup/restore, it must not have any target other than ah-builtin-python-bundle specified in queue.yaml.

Very frequent backups often lead to higher costs. When you run a Datastore Admin job, you are actually running an underlying MapReduce job. MapReduce jobs cause frontend instance hours to increase on top of Storage operations and Storage usage. To keep an eye on your resource usage:

  1. Visit the Cloud Platform Console Dashboard page
  2. Use the pulldown menus at the top of the page to select the default module and the ah-builtin-python-bundle version

Backup and restore data

You can back up all entities or just the selected kinds of entities, and you can restore from one of these backups when you need to. The backup and restore feature is intended to help you recover from accidental deletes of data or to enable you to export data.

Note that the backup does not contain any indexes. When you restore, the required indexes are automatically rebuilt using the index definitions you uploaded with your application.

You can also use backup files to export your data to other Cloud Platform services, such as BigQuery. Use Google Cloud Storage for backups.

When you restore from a backup, any new entities added since the backup are retained, and entities that existed at backup-time and that were modified after the backup are overwritten with values from the backup. You can restore all data from a backup or you can restore specific entity kinds from the backup. In addition, you can also use this feature to restore a backup of one app's data to some other app, provided that you use Google Cloud Storage for your backups.

Limits affecting backups and restores

By default, you will be limited to no more than 100 GB of backups and 100 GB of restores per day subject to retries.

Backing up data

To create a backup file for future data restores for exporting:

  1. If you haven't already, you'll need to do the following to set up a Cloud Storage bucket for your backups:
    1. Enable billing for your project.
    2. Click Create bucket in the Storage Browser for your project.
  2. From the Admin page, click Open Datastore Admin.
  3. Select the entity kind(s) that you wish to backup.
  4. Click Backup Entities to display the backup form.
  5. Notice that the default queue is used for the backup job; you can use this in most cases. If you need to change this to another queue, make sure the queue used does not have any target specified in queue.yaml other than ah-builtin-python-bundle.
  6. Notice that a backup name is supplied and that it includes a datestamp. You must change this value if you make more than one backup per day because a backup will not be made if a backup of the same name already exists.
  7. Select Google Cloud Storage as the backup storage location.
  8. When you choose Cloud Storage, you are prompted for the bucket name where the backups are to be stored, in the format /my_bucket_name. You can optionally specify the bucket name suffixed with a directory structure (e.g., /bucket_name/backups/backup1): If those folders don't already exist, they will be created.

  9. Start the backup jobs by clicking Backup Entities. Notice that a job status page is displayed.

  10. If you disabled datastore writes, re-enable them.

Aborting a backup

If Backup jobs are currently running, they appear in a Pending Backups list in the Datastore Admin screen. You can stop these running backups by selecting the backup in the list and clicking Abort. When you abort a backup job, App Engine attempts to delete backup data that has been saved up to that point. However, in some cases, some files may remain after the abort. You can locate these files in the location you chose for your backups (Blobstore or Google Cloud Storage) and safely delete them after the abort completes. The names of such files will start with the following pattern: datastore_backup__your_backup_name_.

Finding information about a backup

You may want to find out details about a backup, such as which entity kinds it contains, where it was saved (e.g., Blobstore or Google Cloud Storage), and its starting and ending time. To display this backup information:

  1. Select one or more backups in the Backups or Pending Backups list.
  2. Click Info to display information for those backups.
  3. Click Back to return to the main Datastore Admin screen

Scheduled backups

You can run scheduled backups using the App Engine Cron service. For details, see Scheduled Backups.

Restoring data

To restore from a backup:

  1. Optionally, disable Datastore writes for your app. (It's normally a good idea to do this to avoid conflicts between the restore and any new data written to the Datastore.)
  2. Go to the Admin page and click Open Datastore Admin.
  3. In the list of available backups, select the backup that you want to restore from.
  4. Click Restore.
  5. In the advisory page that is displayed, notice the list of entities with checkboxes. By default, all of the entities will be restored. Uncheck the checkbox next to each entity that you don't want to restore.
  6. Also in the advisory page, notice that the default queue, with its pre-configured performance settings, is used for the restore job. Change this to another queue that you have configured differently if you need different queue performance characteristics, making sure the queue chosen does not have any target specified in queue.yaml other than ah-builtin-python-bundle.
  7. Start the restore by clicking Restore. Notice that a job status page is displayed.
  8. If you disabled writes, re-enable Datastore writes for your application.

Restoring data to another app

If you back up your data using Google Cloud Storage, you can restore backups to apps other than the one used to create the backup.

To restore backup data from one app to a different app:

  1. Using the Google Cloud Platform Console, locate the project that has the bucket used for your backups and add the target app (the app you are restoring to) to the project team with Edit permissions.
  2. Make a new backup in your applications whose data is to be copied. The permissions set in the previous step are not retroactive to existing backups, so the target app will not be able to access those earlier backups. The target app can access only backups made after it was given permissions.
  3. Optionally, disable Datastore writes for your target app. (This is normally a good idea, to avoid conflicts between the restore and any new data written to the Datastore.)
  4. Go to the Admin page for the target app and click Open Datastore Admin.
  5. In the textbox next to the button labelled Import Backup Information specify the bucket containing the backup, in the format /gs/my_bucket. This will result in a displayed list of all the backups in that bucket. Alternatively, supply the file handle for a specific backup; the handle can be obtained from the source application by selecting the backup and clicking Info; the file handle appears next to the label Handle.
  6. Click Import Backup Information.
  7. The resulting selection page shows the available backups for the bucket you specified, unless you specified a backup by its handle. Select the desired backup and click one of the following:
    • Add to Backup List if you want this backup to be retained in the list of available backups for your app.
    • Restore From Backup if you want to restore from this backup but do not want the backup displayed in the list of available backups for your app.
  8. In the advisory page that is displayed, notice the list of entities with checkboxes. By default, all of the entities will be restored. Uncheck the checkbox next to each entity that you don't want to restore.
  9. Also in the advisory page, notice that the default queue, with its pre-configured performance settings, is used for the restore job. Change this to another queue that you have configured differently if you need different queue performance characteristics.
  10. Start the restore by clicking Restore. Notice that a job status page is displayed.
  11. If you disabled writes, re-enable Datastore writes for your application.

Copying entities to another application

You can use the Datastore Admin page to copy all entities of a kind, or all entities of all kinds, to another application. The default queue, with its pre-configured performance settings, is used for the copying job. Change this to another queue that you have configured differently if you need different queue performance characteristics.

From the Datastore Admin page, you can select and copy entity kind(s) with the click of a button:

Procedure for copying a Datastore

To copy a Datastore:

  1. Make sure Datastore Admin is enabled for your application.
  2. Enable the remote_api builtin for the target application so it can receive data from the source. To do this, add the following to app.yaml:

    builtins:
    - remote_api: on
    
  3. Grant permission for the source app to write to the target app by doing the following:

    1. Add the following to the appengine_config.py file in the root directory of your application:

      remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION = ('HTTP_X_APPENGINE_INBOUND_APPID',['source appid here'])
      

      If you do not have a appengine_config.py file, you can create a new one or copy the sample located in google/appengine/ext/appstats/sample_appengine_config.py.

    2. Upload the modified version of your application using appcfg.py update.

  4. Set the source application to read-only mode.

    Although this step is not required, it is strongly recommended. Copying entities into a new Datastore takes time and writes done during the copy might not be transferred during the copy. If the source application is not in read-only mode, you can still copy data, but you'll see a notreadonly warning in the console. The destination Datastore is not guaranteed to receive a complete copy of the new data unless writes are disabled.

  5. Select the entity kind(s) to copy individually or in bulk, and copy them using Copy To Other App. On the confirmation screen, enter the remote endpoint of the target app:

    • The typical remote endpoint is http://_your_target_app_id_.appspot.com/_ah/remote_api.
    • If you used the sample app to copy your data, the remote endpoint is http://datastore-admin._your_target_app_id_.appspot.com/_ah/remote_api.
    • If you used an alternate major version, the remote endpoint of your target application is http://app_version._your_target_app_id_.appspot.com/_ah/remote_api.

    After you confirm, the system validates the request. If the remote_api connection can be established, one or more mapreduce operations begins to copy the data. You can follow the link to see the status of the initial set of mapreduce operations. If the application uses the namespace feature, a mapreduce runs for each namespace for each entity kind.

    You can view a summary of the copy status from the Datastore Admin page. To view the individual mapreduce status, you can visit http://_your_target_app_id_.appspot.com/_ah/mapreduce/

  6. If you disabled Datastore writes as recommended prior to the copy, re-enable writes.

Deleting entities in bulk

On the Datastore Admin page, you can delete all entities of a kind, or all entities of all kinds, in all namespaces. To use this feature, you must enable Datastore Admin for your project. Then open the Datastore Admin page, select one or more entity kinds, and press Delete Entities.