App Engine

Migrating to the High Replication Datastore

You can migrate your application from the (deprecated) Master/Slave Datastore to the (now standard) High Replication Datastore (HRD) using the migration tools found in the Application Settings tab of the App Engine Administration Console. You may need to make some changes to your application to ensure an optimal migration. For more information, see the following:


Before you start

Java 6 apps must be upgraded to Java 7 prior to the migration to HRD. To upgrade:

  1. Update your app to the latest App Engine Java SDK, which requires your app to be a Java 7 app.
  2. Redeploy your app to production App Engine.
  3. Test it for correct behavior. There are a small number of potential compatibility issues between Java 6 and Java 7.

During migration of a Python Master/Slave app to HRD, you may see a warning about the requirement to migrate from Python 2.5 to Python 2.7. IGNORE this warning. Migrating from Master/Slave to HRD does not require migrating to Python 2.7; in fact, such a migration is not possible until after you migrate to HRD.

The migration process

To complete the migration, you must:

  1. Create a duplicate of your Master/Slave application using the HRD.
  2. Deploy the new HRD application.
  3. Migrate data using the Migration Tool.

If you are using serialized Datastore keys or need to pause or revert your migration, be aware of potential Datastore migration issues.

Duplicating Your Application

You should create a new HRD application and copy the settings you wish to keep from your existing application.

To make a copy of your existing application,

  1. Open the Google Developer Console.
  2. Click Create Project.
  3. Enter a project name.
  4. Click Create.
  5. Add any users from the existing application to the new project on the Permissions tab.
  6. Adjust other settings to match your existing application, such as the log retention policy in the Settings tab. If you do not see the App Engine menu item, add "/appengine" to the end of the URL. You should only need to manually adjust the URL once.
  7. You will also need to configure Billing tab and any custom quotas.

Deploying Your New HRD Application

You must deploy your HRD application before copying data to it. If you start copying data before deploying your application, all of the indexes must be rebuilt.

After you duplicate your application and finish making changes to your new application settings, deploy your new application by following the instructions below.

For Python, run the appcfg.py command with the update action and the name of your application's root directory. The root directory should contain the app.yaml file for the application:

    appcfg.py update myapp/

For more details, see Uploading, Downloading, and Managing a Python App

For Java, if you are using the Google Plugin for Eclipse, you deploy the application by clicking the deploy button in the toolbar. Alternatively, you can upload (deploy) the application from the command line using the appcfg utility in the appengine-java-sdk/bin/ directory, as follows:

For Windows:

    appengine-java-sdk\bin\appcfg.cmd update myapp/war

For Mac OS and Linux:

    ./appengine-java-sdk/bin/appcfg.sh update myapp/war

For more details, see Uploading and Managing a Java App.

Using the Migration Tool

The HRD migration tool can migrate your application while the application continues to serve requests and write data to the Master/Slave Datastore. Incoming requests continue to be served by the current application using Master/Slave during this process, until the copy process is finished and you are ready to switch over to the HRD application. There may be a brief interval during the switchover when your application may not be able to serve write requests.

To run a migration:

  1. Make sure you have duplicated your application as described under Duplicating Your Application, above. You will migrate your existing application data to this new application.
  2. Make sure you have deployed your application code and index configuration to the target HRD application, as described in Deploying Your New HRD Application.
  3. Open the App Engine Administration Console, select the application you are migrating, and select Application Settings > View Migration Tool. (Note that this button appears ONLY if your current app uses the deprecated Master/Slave datastore!)
  4. In the Destination Application dropdown, select the target HR application to which you are migrating the data.
  5. If your application has blob data in Blobstore, you need to migrate that blob data as well. The blobs will be copied and given new blob keys in the new application, and the Datastore will be updated with the new blob keys. (However, any blob keys stored in serialized form within the Datastore will not be migrated; see Migrating Serialized Blob Keys, below, for more information.)

  6. If you wish to be emailed when each phase of the migration job is done, click the Email checkbox. (This feature is useful, because the status displayed in the HRD form is not refreshed automatically.)

  7. Click Start Migration and wait while the first phase of the migration is performed. Notice that the list of tasks to be completed by the migration tool is listed in the HRD Migration form, with the currently running task showing the current status.
  8. Wait until you receive the completion email from the migration, or manually refresh the browser window after a while to display the latest status. (The Migration form does not refresh automatically.)
  9. When you see the status message "Waiting to activate read-only. See below...," click Launch Incremental Copy. Incremental copy copies all existing data to the new HRD Datastore. (This step decreases the amount of time your application spends in read-only mode.)

  10. When the copy completes, click Activate Read-only. (You'll get an email notification if you chose that option; otherwise you'll need to refresh the browser window to see the notification message "Waiting to activate read-only."

  11. You will be notified when the migration tool is ready to alias the application (by email if you chose that, or by a message when your refresh the browser window). Click Finish Migration to complete the migration. At this point, the new HRD application begins serving incoming requests.

  12. Verify that the alias to the new migrated application is properly set up by going to the App Engine Administrative Console for your Master/Slave app. You should see a message similar to the following: "This application is hidden by an alias."

  13. Once you have verified that the migration is successful and you no longer need the source application, disable billing on the source application to avoid unnecessary additional cost, then disable and delete the source application.

Potential Migration Issues

Pausing and Reverting Migration

At any point before clicking Finish Migration, you can pause the migration by clicking Pause Migration. This returns the application to read/write mode so it can continue to serve incoming requests. To resume a paused migration, click Resume Migration.

At any point before clicking Finish Migration, you can revert the migration by clicking Pause Migration followed by Revert Migration. This returns the source application to its original state, including blobs and blob keys. However, the target application retains the copied data, blobs, and new blob keys. If you want to re-migrate to that target application at a later date, you must delete the target application's Datastore data.

Migrating Serialized Blob Keys

The migration feature in the App Engine Administration Console updates only those blob keys that are native blob references created and managed by the Datastore itself; it does not migrate any blob keys that you have stored in serialized data within the Datastore, no matter where the data was serialized (for instance, in the Datastore or in URLs indexed by Google).

You can use the BlobMigrationRecord class to replace any serialized key, using get_new_blob_key() to return the new key corresponding to the supplied old key:

    from google.appengine.ext import blobstore

    def GetNewBlobKey(old_key)
      return blobstore.BlobMigrationRecord.get_new_blob_key(old_key)

Migrating Serialized Datastore Keys

The migration feature in the App Engine Administration Console updates only those Datastore keys that are created and managed by the Datastore itself. It does not migrate any Datastore keys that you have stored in serialized data, no matter where the data was serialized (for instance, in the Datastore or in URLs indexed by Google).

You can write code to convert these keys to be used by your new app ID. See the documentation for the Key class in Python and Java for more detailed information.