You can migrate your application from the (deprecated) Master/Slave Datastore to the (now standard) High Replication Datastore (HRD) using the migration tools found in the Application Settings tab of the App Engine Administration Console. You may need to make some changes to your application to ensure an optimal migration. For more information, see the following:
Before you start
Java 6 apps must be upgraded to Java 7 prior to the migration to HRD. To upgrade:
- Update your app to the latest App Engine Java SDK, which requires your app to be a Java 7 app.
- Redeploy your app to production App Engine.
- Test it for correct behavior. There are a small number of potential compatibility issues between Java 6 and Java 7.
During migration of a Python Master/Slave app to HRD, you may see a warning about the requirement to migrate from Python 2.5 to Python 2.7. IGNORE this warning. Migrating from Master/Slave to HRD does not require migrating to Python 2.7; in fact, such a migration is not possible until after you migrate to HRD.
The migration process
To complete the migration, you must:
- Create a duplicate of your Master/Slave application using the HRD.
- Deploy the new HRD application.
- Migrate data using the Migration Tool.
If you are using serialized Datastore keys or need to pause or revert your migration, be aware of potential Datastore migration issues.
Duplicating Your Application
These instructions use the Duplicate Application Settings feature inside the Application Settings tab of the App Engine Administration Console to copy an application where the copy will use HRD. You could manually accomplish the same thing using other capabilities of the Administration Console if you wish, but the method described here is the fastest, easiest way to duplicate your application.
To make a copy of your existing application,
- Open the App Engine Administration Console, select the application you want to copy, and go to Application Settings.
- In the New Application Identifier text box, notice that
your current application's application ID is shown with the
-hrdsuffix. You can use this or provide an entirely new application ID if you wish.
- Click Check Availability to verify that the application ID you specified is available. Change the ID if necessary.
- Click Duplicate Application.
Only the application title and access control settings are copied during this copy process. You must manually set any other features needed for the new application, such as the following:
- Cron configuration
- Task Queue configuration
- Backends. The backends architecture has been deprecated. You should convert your backends to modules. See the instructions for Python, Java, and Go. Remember to shut down any resident backends running under the old application ID to avoid incurring unnecessary cost.
- Any other desired settings or configurations
Deploying Your New HRD Application
You must deploy your HRD application before copying data to it. If you start copying data before deploying your application, all of the indexes must be rebuilt.
After you duplicate your application and finish making changes to your new application settings, deploy your new application by following the instructions below.
For Python, run the
appcfg.py command with the update action and
the name of your application's root directory. The root directory should contain
app.yaml file for the application:
appcfg.py update myapp/
For more details, see Uploading, Downloading, and Managing a Python App
For Java, if you are using the
Google Plugin for Eclipse,
you deploy the application by clicking the deploy button in the
toolbar. Alternatively, you can upload (deploy) the application from the command
line using the
appcfg utility in the
appengine-java-sdk/bin/ directory, as follows:
appengine-java-sdk\bin\appcfg.cmd update myapp/war
For Mac OS and Linux:
./appengine-java-sdk/bin/appcfg.sh update myapp/war
For more details, see Uploading and Managing a Java App.
Using the Migration Tool
The HRD migration tool can migrate your application while the application continues to serve requests and write data to the Master/Slave Datastore. Incoming requests continue to be served by the current application using Master/Slave during this process, until the copy process is finished and you are ready to switch over to the HRD application. There may be a brief interval during the switchover when your application may not be able to serve write requests.
To run a migration:
- Make sure you have duplicated your application as described under Duplicating Your Application, above. You will migrate your existing application data to this new application.
- Make sure you have deployed your application code and index configuration to the target HRD application, as described in Deploying Your New HRD Application.
- Open the App Engine Administration Console, select the application you are migrating, and select Application Settings > View Migration Tool. (Note that this button appears ONLY if your current app uses the deprecated Master/Slave datastore!)
- In the Destination Application dropdown, select the target HR application to which you are migrating the data.
If your application has blob data in Blobstore, you need to migrate that blob data as well. The blobs will be copied and given new blob keys in the new application, and the Datastore will be updated with the new blob keys. (However, any blob keys stored in serialized form within the Datastore will not be migrated; see Migrating Serialized Blob Keys, below, for more information.)
If you wish to be emailed when each phase of the migration job is done, click the Email checkbox. (This feature is useful, because the status displayed in the HRD form is not refreshed automatically.)
- Click Start Migration and wait while the first phase of the migration is performed. Notice that the list of tasks to be completed by the migration tool is listed in the HRD Migration form, with the currently running task showing the current status.
- Wait until you receive the completion email from the migration, or manually refresh the browser window after a while to display the latest status. (The Migration form does not refresh automatically.)
When you see the status message "Waiting to activate read-only. See below...," click Launch Incremental Copy. Incremental copy copies all existing data to the new HRD Datastore. (This step decreases the amount of time your application spends in read-only mode.)
When the copy completes, click Activate Read-only. (You'll get an email notification if you chose that option; otherwise you'll need to refresh the browser window to see the notification message "Waiting to activate read-only."
You will be notified when the migration tool is ready to alias the application (by email if you chose that, or by a message when your refresh the browser window). Click Finish Migration to complete the migration. At this point, the new HRD application begins serving incoming requests.
Verify that the alias to the new migrated application is properly set up by going to the App Engine Administrative Console for your Master/Slave app. You should see a message similar to the following: "This application is hidden by an alias."
Once you have verified that the migration is successful and you no longer need the source application, disable billing on the source application to avoid unnecessary additional cost, then disable and delete the source application.
Potential Migration Issues
Pausing and Reverting Migration
At any point before clicking Finish Migration, you can pause the migration by clicking Pause Migration. This returns the application to read/write mode so it can continue to serve incoming requests. To resume a paused migration, click Resume Migration.
At any point before clicking Finish Migration, you can revert the migration by clicking Pause Migration followed by Revert Migration. This returns the source application to its original state, including blobs and blob keys. However, the target application retains the copied data, blobs, and new blob keys. If you want to re-migrate to that target application at a later date, you must delete the target application's Datastore data.
Migrating Serialized Blob Keys
The migration feature in the App Engine Administration Console updates only those blob keys that are native blob references created and managed by the Datastore itself; it does not migrate any blob keys that you have stored in serialized data within the Datastore, no matter where the data was serialized (for instance, in the Datastore or in URLs indexed by Google).
You can use the
class to replace any serialized key, using
get_new_blob_key() to return the
new key corresponding to the supplied old key:
from google.appengine.ext import blobstore def GetNewBlobKey(old_key) return blobstore.BlobMigrationRecord.get_new_blob_key(old_key)
Migrating Serialized Datastore Keys
The migration feature in the App Engine Administration Console updates only those Datastore keys that are created and managed by the Datastore itself. It does not migrate any Datastore keys that you have stored in serialized data, no matter where the data was serialized (for instance, in the Datastore or in URLs indexed by Google).