Using a managed import to set up replication from external databases

This page describes how to set up and use a managed import for data when replicating from an external server to Cloud SQL.

You must complete all the steps on this page. When finished, you can administer and monitor the source representation instance the same way as you would any other Cloud SQL instance.

Before you begin

Before you start, you should have configured the external server, created the source representation instance, and set up the Cloud SQL replica.

Updating permissions for the replication user

The replication user on the external server is configured to accept connections from any host (%). You should update this user account so that it can only be used with the Cloud SQL replica.

If the replication is based on GTID, you need the REPLICATION SLAVE, EXECUTE, SELECT, and SHOW VIEW permissions.

If the replication is based on binlog, you need the REPLICATION SLAVE, EXECUTE, SELECT, SHOW VIEW, REPLICATION CLIENT, and RELOAD permissions.

Open a terminal on the external server and enter the following commands.

mysql Client

For GTID:

    UPDATE mysql.user
      SET Host='NEW_HOST'
      WHERE Host='OLD_HOST'
      AND User='USERNAME';
    GRANT REPLICATION SLAVE, EXECUTE, SELECT, SHOW VIEW ON . TO
    'USERNAME'@'HOST';
    FLUSH PRIVILEGES;

For binlog:

    UPDATE mysql.user
    SET Host='NEW_HOST'
    WHERE Host='OLD_HOST'
    AND User='USERNAME';
    GRANT REPLICATION SLAVE, EXECUTE, SELECT, SHOW VIEW, REPLICATION CLIENT,
    RELOAD ON . TO 'GCP_USERNAME'@'HOST';
    FLUSH PRIVILEGES;

example

UPDATE mysql.user
  SET Host='192.0.2.0'
  WHERE Host='%'
  AND User='replicationUser';
GRANT REPLICATION SLAVE, EXECUTE, SELECT, SHOW VIEW, REPLICATION CLIENT,
RELOAD ON *.* TO 'username'@'host.com';
FLUSH PRIVILEGES;
Property Description
NEW_HOST Specify the outgoing IP of the Cloud SQL replica.
OLD_HOST The current value assigned to Host that you want to change.
USERNAME The replication user account on the external server.
GCP_USERNAME The username for the GCP user account.
HOST The hostname for the GCP user account.

Verifying your replication settings

After your setup is complete, ensure that the Cloud SQL replica can replicate from the external server.

First, ensure that your external sync settings are correct. To do this, use the commands below to verify:

  • Connectivity between the Cloud SQL replica and external server
  • Replication user privileges
  • Version compatibility
  • The Cloud SQL replica is not already replicating
  • Binlogs are enabled on the external server
  • GTID is enabled if you are trying to do an external sync from an RDS external server and are using a Google Cloud bucket

Open a terminal and enter these commands to verify external sync settings are correct:

curl

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "SYNC_MODE",
         "skipVerification": "SKIP_VERIFICATION",
         "mysqlSyncConfig": {
           "syncFlags": SYNC_FLAGS"
         }
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT_ID/instances/REPLICA_INSTANCE/verifyExternalSyncSettings

example

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "online",
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/myproject/instances/myreplica/verifyExternalSyncSettings

example w/ sync flags

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "online",
         "skipVerification": false,
         "mysqlSyncConfig": {
             "syncFlags": [{"name": "max-allowed-packet", "value": "1073741824"}, {"name": "hex-blob"}, {"name": "compress"}, {"name": "databases", "value": "db1 db2"}]"
             }
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/replica-instance/verifyExternalSyncSettings
Property Description
SYNC_MODE verifyExternalSyncSettings verifies that you can keep the Cloud SQL replica and external server in sync after replication is set up. Sync modes include EXTERNAL_SYNC_MODE_UNSPECIFIED, ONLINE, and OFFLINE.
SKIP_VERIFICATION Whether or not to skip the built-in verification step before syncing your data. Only recommended if you have already verified your replication settings.
SYNC_FLAGS A list of initial sync flags to verify. Only recommended if you plan on using custom sync flags when replicating from the source. For a list of allowed flags see here.
PROJECT_ID The ID of your project in Google Cloud.
REPLICA_INSTANCE The ID of your Cloud SQL replica.

Starting replication on the external server

After you have verified that you can replicate from the external server, you are ready to perform the replication. Expect the replica to import about 25-50 GB per hour.

During the initial import process, do not perform any DDL operations on the external server. Doing so could cause inconsistencies during the import. After the import process completes, the replica uses the binary logs on the external server to catch up to the current state of the external server.

curl

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "SYNC_MODE",
         "skipVerification": "SKIP_VERIFICATION",
         "mysqlSyncConfig": {
           "syncFlags": "SYNC_FLAGS"
         }
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT_ID/instances/REPLICA_INSTANCE/startExternalSync

example

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "online",
         "skipVerification": false
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/replica-instance/startExternalSync

example w/ sync flags

gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     --header 'Content-Type: application/json' \
     --data '{
         "syncMode": "online",
         "skipVerification": false,
         "mysqlSyncConfig": {
             "syncFlags": [{"name": "max-allowed-packet", "value": "1073741824"}, {"name": "hex-blob"}, {"name": "compress"}, {"name": "databases", "value": "db1 db2"}]"
             }
       }' \
     -X POST \
     https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/replica-instance/startExternalSync
Property Description
SYNC_MODE Verifies that you can keep the Cloud SQL replica and external server in sync after replication is set up.
SKIP_VERIFICATION Whether or not to skip the built-in verification step before syncing your data. Only recommended if you have already verified your replication settings.
SYNC_FLAGS A list of initial sync flags to verify. Only recommended if you plan on using custom sync flags when replicating from the source. For a list of allowed flags see here.
PROJECT_ID The ID of your project in Google Cloud.
REPLICA_INSTANCE The ID of your Cloud SQL replica.

Allowed and default initial sync flags

When setting up replication when using managed import, mysqldump runs with the following flags by default:

  • --host=[HOST_SPECIFIED_IN_SOURCE_CONFIG]
  • --port=[PORT_SPECIFIED_IN_SOURCE_CONFIG]
  • --ssl-ca=[SSL_CA_SPECIFIED_IN_SOURCE_CONFIG]
  • --ssl-cert=[SSL_CERT_SPECIFIED_IN_SOURCE_CONFIG]
  • --ssl-key=[SSL_KEY_SPECIFIED_IN_SOURCE_CONFIG]
  • --master-data=[SPECIFIED_BY_SYNC_MODE_AND_SOURCE_GTID_MODE]
  • --set-gtid-purged=AUTO
  • --single-transaction
  • --hex-blob
  • --compress
  • --no-autocommit
  • --default-character-set=utf8mb4
  • --databases [ALL_NON_SYSTEM_DBS]
  • --add-drop-table
  • --routines

We recommend this configuration for most customers.

However, for particular cases, you might need to specify custom sync flags. The following flags are allowed:

  • --add-drop-database
  • --add-drop-table
  • --add-drop-trigger
  • --add-locks
  • --all-databases
  • --allow-keywords
  • --all-tablespaces
  • --apply-slave-statements
  • --column-statistics
  • --comments
  • --compact
  • --compatible
  • --complete-insert
  • --compress
  • --compression-algorithms
  • --create-options
  • --databases
  • --default-character-set
  • --delayed-insert
  • --disable-keys
  • --dump-date
  • --events
  • --exclude-databases
  • --extended-insert
  • --fields-enclosed-by
  • --fields-escaped-by
  • --fields-optionally-enclosed-by
  • --fields-terminated-by
  • --flush-logs
  • --flush-privileges
  • --force
  • --get-server-public-key
  • --hex-blob
  • --ignore-error
  • --ignore-read-lock-error
  • --ignore-table
  • --insert-ignore
  • --lines-terminated-by
  • --lock-all-tables
  • --lock-tables
  • --max-allowed-packet
  • --net-buffer-length
  • --network-timeout
  • --no-autocommit
  • --no-create-db
  • --no-create-info
  • --no-data
  • --no-defaults
  • --no-set-names
  • --no-tablespaces
  • --opt
  • --order-by-primary
  • --pipe
  • --quote-names
  • --quick
  • --replace
  • --routines
  • --secure-auth
  • --set-charset
  • --shared-memory-base-name
  • --show-create-skip-secondary-engine
  • --skip-opt
  • --ssl-cipher
  • --ssl-fips-mode
  • --ssl-verify-server-cert
  • --tables
  • --tls-ciphersuites
  • --tls-version
  • --triggers
  • --tz-utc
  • --verbose
  • --xml
  • --zstd-compression-level

For allowed values, see the MySQL public docs.

Global read lock permission

If you do not have permission to access the global read lock on the external server and GTID is off, as might be the case with Amazon RDS, you need to pause writes to your server, as described in the following steps:

  1. Go to the Logs Viewer, and select your Cloud SQL replica from the resource list. You should see a list of the most recent logs for your Cloud SQL replica. Ignore them for now.
  2. Open a terminal and enter the commands below these steps to replicate from the external server.
  3. Return to the Logs Viewer. When you see the log below, stop writing to the database on your external server. In most cases, this is only required for a few seconds.

       DUMP_IMPORT(START): Start importing data, please pause any write to the
       external primary database.
    
  4. When you see the log below in your Logs Viewer, enable writing to the database on your external server again.

       DUMP_IMPORT(SYNC): Consistent state on primary and replica. Writes to the
       external primary may resume.
    

Proceeding with replication

Once you start replication from the external server, you need to monitor replication and then complete your migration. To learn more, see Monitoring replication.

Troubleshooting

Click the links in the table for details:

Issue Troubleshooting
Lost connection to MySQL server during query when dumping table. The source may have become unavailable, or the dump contained packets too large.

Make sure the external primary is available to connect, or use mysqldump with the max_allowed_packet option.

To learn more about using mysqldump flags for managed import migration, see Allowed and default initial sync flags

The initial data migration was successful, but no data is being replicated. One possible root cause could be your source database has defined replication flags which result in some or all database changes not being replicated over.

Make sure the replication flags such as binlog-do-db, binlog-ignore-db, replicate-do-db or replicate-ignore-db are not set in a conflicting way.

Run the command show master status on the primary instance to see the current settings.

The initial data migration was successful but data replication stops working after a while. Things to try:

  • Check the replication metrics for your replica instance in the Cloud Monitoring section of the Google Cloud Console.
  • The errors from the MySQL IO thread or SQL thread can be found in Cloud Logging in the mysql.err log files.
  • The error can also be found when connecting to the replica instance. Run the command SHOW SLAVE STATUS, and check for the following fields in the output:
    • Slave_IO_Running
    • Slave_SQL_Running
    • Last_IO_Error
    • Last_SQL_Error
mysqld check failed: data disk is full. The data disk of the replica instance is full.

Increase the disk size of the replica instance. You can either manually increase the disk size or enable auto storage increase.

Reviewing your replication logs

When you verify your replication settings, logs are produced. You can view these logs by following these steps:

  • Go to the Logs Viewer in the Google Cloud Console.

    Go to the Logs Viewer

  • Select the Cloud SQL replica from the Instance dropdown.
  • Select the replication-setup.log log file.
  • If the Cloud SQL replica is unable to connect to the external server, confirm the following:

    • Any firewall on the external server is configured to allow connections from the Cloud SQL replica's outgoing IP address.
    • Your SSL/TLS configuration is correct.
    • Your replication user, host, and password are correct.

What's next