Troubleshooting Cloud SQL for MySQL

Topics in this page include:

If you're looking for information about specific error messages, check the error messages page.

Backup and recovery

Issue Troubleshooting
You can't see the current operation's status. The Google Cloud Console reports only success or failure when the operation is done. It isn't designed to show warnings or other updates.

Run the gcloud sql operations list command to list all operations for the given Cloud SQL instance.

You want to find out who issued an on-demand backup operation. The user interface doesn't show the user who started an operation.

Look in the logs and filter by text to find the user. You may need to use audit logs for private information. Relevant log files include:

  • cloudsql.googlapis.com/mysql-general.log
  • cloudsql.googleapis.com/mysql.err
  • If Cloud Audit Logs is enabled, and you have the required permissions to view them, cloudaudit.googleapis.com/activity may also be available.
You can't do a backup after an instance was deleted. The grace period for a Cloud SQL instance purge is four days, with the exception of read replicas, which are purged immediately. During this time, customer support can recreate the instance. After instances are purged, no data recovery is possible.

If you have done an export operation, you can create a new instance and then do an import operation to recreate the database. Exports are written to Cloud Storage and imports are read from there.

An automated backup is stuck for many hours and can't be canceled. Backups can take a long time depending on the database size.

If you really need to cancel the operation, you can ask customer support to force restart the instance.

A restore operation can fail when one or more users referenced in the SQL dump file don't exist. Before restoring a SQL dump, all the database users who own objects or were granted permissions on objects in the dumped database must exist in the target database. If they don't, the restore operation fails to recreate the objects with the original ownership or permissions.

Create the database users before restoring the SQL dump.

You want to increase the number of days that you can keep automatic backups from seven to 30 days, or longer. Only seven backups are retained. Backups get pruned regularly due to the cost and size of retaining backups. Unfortunately, this means that the currently visible backups are the only automated backups you can restore from.

To keep backups indefinitely, you can create an on-demand backup, as they are not deleted in the same way as automated backups. On-demand backups remain indefinitely. That is, they remain until they're deleted or the instance they belong to is deleted. Because that type of backup is not deleted automatically, it can affect billing.

A backup failed and you see an Unknown error message. The backup operation might have timed out.

There are two flags that influence the backup creation: checkpoint_timeout and checkpoint_completion_target. At the start of the backup, a slow checkpoint is run and it takes checkpoint_completion_target multiplied by checkpoint_timeout.

For example, 900 sec * 0.9 sec = 810 sec = 13.5 min.

For this reason, a timeout occurs. Decreasing the value of the checkpoint_completion_target fixes the issue in this case.

An automated backup failed and you didn't receive an email notification. Notifications aren't supported for backup failures.

When an automated backup fails, an Operation error message appears in the Cloud SQL instance's Details page.

You can find the status of a backup through either the REST API or gcloud commands. For example, first list the backups for an instance, and then describe a specific backup by its ID:

gcloud sql backups list \
--project=PROJECT_ID \
--instance=INSTANCE_ID
   
gcloud sql backups describe BACKUP-ID \
--instance=INSTANCE_ID
    

Cloning

Issue Troubleshooting
Cloning fails with constraints/sql.restrictAuthorizedNetworks error. The cloning operation is blocked by the Authorized Networks configuration. Authorized Networks are configured for public IP addresses in the Connectivity section of the Google Cloud Console, and cloning is not permitted due to security considerations.

Remove all Authorized Networks entries from the Cloud SQL instance if you can. Otherwise, create a replica without any Authorized Networks entries.

Connectivity

Issue Troubleshooting
Aborted connection. The issue might be:
  • Networking instability.
  • No response to TCP keep-alive commands (either the client or the server isn't responsive, possibly overloaded)
  • The database engine connection lifetime was exceeded and the server ends the connection.

Applications must tolerate network failures and follow best practices such as connection pooling and retrying. Most connection poolers catch these errors where possible. Otherwise the application must either retry or fail gracefully.

For connection retry, we recommend the following methods:

  1. Exponential backoff. Increase the time interval between each retry, exponentially.
  2. Add randomized backoff also.

Combining these methods helps reduce throttling.

Creating instances

Issue Troubleshooting
You get the error message: Failed to create subnetwork. Couldn't find free blocks in allocated IP ranges. Please allocate new ranges for this service provider. There are no more available addresses in the allocated IP range. There can be several possible scenarios:

  • The size of the allocated IP range for the private service connection is smaller than /24.
  • The size of the allocated IP range for the private service connection is too small for the number of Cloud SQL instances.
  • You're attempting to create both MySQL or SQL Server and PostgreSQL instances on the same private service connection in the VPC host project. MySQL and SQL Server can share the same service connection. PostgreSQL requires its own service connection.
  • You're attempting to create instances on the same private service connection in different regions, which is not supported.

For each of the above scenarios you can elect to either expand the existing allocated IP range or allocate an additional IP range to the private service connection.

If you used the --allocated-ip-range-name flag while creating the Cloud SQL instance, you may only expand the specified IP range.

If you're allocating a new range, take care that the allocation doesn't overlap with any existing allocations.

After creating a new IP range, update the vpc peering with the following command:

gcloud services vpc-peerings update \
--service=servicenetworking.googleapis.com \
--ranges=OLD_RESERVED_RANGE_NAME,NEW_RESERVED_RANGE_NAME \
--network=VPC_NETWORK \
--project=PROJECT_ID \
--force
    

If you're expanding an existing allocation, take care to increase only the allocation range and not decrease it. For example, if the original allocation was 10.0.10.0/24, then make the new allocation at least 10.0.10.0/23.

In general, if starting from a /24 allocation, decrementing the /mask by 1 for each condition (additional instance type group, additional region) is a good rule of thumb. For example, if trying to create both instance type groups on the same allocation, going from /24 to /23 is enough.

After expanding an existing IP range, update the vpc peering with following command:

gcloud services vpc-peerings update \
--service=servicenetworking.googleapis.com \
--ranges=RESERVED_RANGE_NAME \
--network=VPC_NETWORK \
--project=PROJECT_ID
    

Export

Issue Troubleshooting
CSV export worked but SQL export failed. CSV and SQL formats do export differently. The SQL format exports the entire database, and likely takes longer to complete. The CSV format lets you define which elements of the database to include in the export.

Use CSV exports to export only what you need.

Export is taking too long. Cloud SQL does not support concurrent synchronous operations.

Use export offloading. At a high level, in export offloading, instead of issuing an export on the source instance, Cloud SQL spins up an offload instance to perform the export. Export offloading has several advantages, including increased performance on the source instance and the unblocking of administrative operations while the export is running. With export offloading, total latency can increase by the amount of time it takes to bring up the offload instance. Generally, for reasonably sized exports, latency is not significant. However, if your export is small enough, then you may notice the increase in latency.

You want exports to be automated. Cloud SQL does not provide a way to automate exports.

You could build your own automated export system using Google Cloud products such as Cloud Scheduler, Pub/Sub, and Cloud Functions, similar to this article on automating backups.

External primary

Click the links in the table for details:

Issue Troubleshooting
Lost connection to MySQL server during query when dumping table. The source may have become unavailable, or the dump contained packets too large.

Make sure the external primary is available to connect, or use mysqldump with the max_allowed_packet option.

To learn more about using mysqldump flags for managed import migration, see Allowed and default initial sync flags

The initial data migration was successful, but no data is being replicated. One possible root cause could be your source database has defined replication flags which result in some or all database changes not being replicated over.

Make sure the replication flags such as binlog-do-db, binlog-ignore-db, replicate-do-db or replicate-ignore-db are not set in a conflicting way.

Run the command show master status on the primary instance to see the current settings.

The initial data migration was successful but data replication stops working after a while. Things to try:

  • Check the replication metrics for your replica instance in the Cloud Monitoring section of the Google Cloud Console.
  • The errors from the MySQL IO thread or SQL thread can be found in Cloud Logging in the mysql.err log files.
  • The error can also be found when connecting to the replica instance. Run the command SHOW SLAVE STATUS, and check for the following fields in the output:
    • Slave_IO_Running
    • Slave_SQL_Running
    • Last_IO_Error
    • Last_SQL_Error
mysqld check failed: data disk is full. The data disk of the replica instance is full.

Increase the disk size of the replica instance. You can either manually increase the disk size or enable auto storage increase.

External replica

Issue Troubleshooting
Error message: The slave is connecting ... master has purged binary logs containing GTIDs that the slave requires. The primary Cloud SQL instance has automatic backups and binary logs and point-in-time recovery is enabled, so it should have enough logs for the replica to be able to catch up. However, in this case although the binary logs exist, the replica doesn't know which row to start reading from.

Create a new dump file using the correct flag settings, and configure the external replica using that file

  1. Connect to your mysql client through a Compute Engine instance.
  2. Run mysqldump and use the --master-data=1 and --flush-privileges flags.

    Important: Do not include the --set-gtid-purged=OFF flag.

    Learn more.

  3. Ensure that the dump file just created contains the SET @@GLOBAL.GTID_PURGED='...' line.
  4. Upload the dump file to a Cloud Storage bucket and configure the replica using the dump file.

Flags

Issue Troubleshooting
After enabling a flag the instance loops between panicking and crashing. Contact customer support to request flag removal followed by a hard drain. This forces the instance t restart on a different host with a fresh configuration without the undesired flag or setting.
You see the error message Bad syntax for dict arg when trying to set a flag. Complex parameter values, such as comma-separated lists, require special treatment when used with gcloud commands.
The time zone doesn't automatically change. Automated time zone changes aren't supported in Cloud SQL for MySQL and must be done manually, and not by string but by time zone offset value.

Edit the instance to change the default_time_zone flag. Name areas are not supported. For example:

Europe/LondonLondon is in the UTC time zone, which would be a supported value of +00:00 for the default_time_zone flag.

High availability

Issue Troubleshooting
You can't find the metrics for a manual failover. Only automatic failovers go into the metrics.
Cloud SQL instance resources (CPU and RAM) are near 100% usage, causing the high availability instance to go down. The instance machine size is too small for the load.

Edit the instance to upgrade to a larger machine size to get more CPUs and memory.

Import

Issue Troubleshooting
The import operation is taking too long. Too many active connections can interfere with import operations.

Close unused operations. Check the CPU and memory usage of your Cloud SQL instance to make sure there are plenty of resources available. The best way to ensure maximum resources for the import is to restart the instance before beginning the operation.

A restart:

  • Closes all connections.
  • Ends any tasks that may be consuming resources.
An import operation can fail when one or more users referenced in the dump file don't exist. Before importing a dump file, all the database users who own objects or were granted permissions on objects in the dumped database must exist in the target database. If they don't, the import operation fails to recreate the objects with the original ownership or permissions.

Create the database users before importing.

An import operation fails with an error that a table doesn't exist. Tables can have foreign key dependencies on other tables, and depending on the order of operations, one or more of those tables might not yet exist during the import operation.

Things to try:

Add the following line at the start of the dump file:

SET FOREIGN_KEY_CHECKS=0;
  

Additionally, add this line at the end of the dump file:

SET FOREIGN_KEY_CHECKS=1;
  

These settings deactivate data integrity checks while the import operation is in progress, and reactivate them after the data is loaded. This doesn't affect the integrity of the data on the database, because the data was already validated during the creation of the dump file.

Logging

Issue Troubleshooting
Logging uses a lot of CPU and memory on your Cloud SQL instance. Logging needs to be tuned.

The log_statement flag can be set to none and the logging_collector flag can be set to off. If logging is still occurring, there may be other log-related flags that can be tuned. You can edit the instance to modify these flags.

Audit logs are not found. Data-Access logs are only written if the operation is an authenticated user-driven API call that creates, modifies, or reads user-created data, or if the operation accesses configuration files or metadata of resources.
Operations information is not found in logs. You want to find more information about an operation.

For example, a user was deleted but you can't find out who did it. The logs show the operation started but don't provide any more information. You must enable audit logging for detailed and personal identifying information (PII) like this to be logged.

Some logs are filtered from the error.log log of a Cloud SQL for SQL Server instance. Filtered logs include AD logs without timestamps, and include: Login failed for user 'x'. Reason: Token-based server access validation failed with an infrastructure error. Login lacks connect endpoint permission. [CLIENT: 127.0.0.1]. These logs are filtered because they potentially can cause confusion.
Logging is using a lot of disk space. There are three kinds of log files that use disk space: redo logs, general logs and binary logs.

Connect to the database and run these commands for details on each type:

SHOW VARIABLES LIKE 'innodb_log_file%';

SELECT ROUND(SUM(LENGTH(argument)/POW(1024,2),2)
AS GB from mysql.general_log;

SHOW BINARY LOGS;
    
Log files are hard to read. You'd rather view the logs as json or text.You can use the gcloud logging read command along with linux post-processing commands to download the logs.

To download the logs as JSON:

gcloud logging read \
"resource.type=cloudsql_database \
AND logName=projects/PROJECT_ID \
/logs/cloudsql.googleapis.com%2FLOG_NAME" \
--format json \
--project=PROJECT_ID \
--freshness="1d" \
> downloaded-log.json
    

To download the logs as TEXT:

gcloud logging read \
"resource.type=cloudsql_database \
AND logName=projects/PROJECT_ID \
/logs/cloudsql.googleapis.com%2FLOG_NAME" \
--format text \
--project=PROJECT_ID \
--freshness="1d"| jq -rnc --stream 'fromstream(1|truncate_stream(inputs)) \
| .textPayload' \
> downloaded-log.txt
   

Managing instances

Issue Troubleshooting
Slow performance after restarting MySQL. Cloud SQL allows caching of data in the InnoDB buffer pool. However, after a restart, this cache is always empty, and all reads require a round trip to the backend to get data. As a result, queries can be slower than expected until the cache is filled.
Slow crash recovery. A large general_log may have accumulated. You can reduce crash recovery time by preventing a large general_log from accumulating. If you have general_log on, truncate the table and only enable general_log for short periods of time.

You can find out the size of the general logs by connecting to the database and running this query:

SELECT ROUND(SUM(LENGTH(argument)/POW(1024,2)),2) from mysql.general_log;
You want to find out what is using up storage. For example, you notice that your database is using only three GB, but storage says that 14 GB is being used. Most of the space not used by tables is used by binary logs and/or temporary files.

Things to try:

  • You can check the storage occupied by binary logs using the following command in the MySQL command line interface: SHOW BINARY LOGS;
  • Temporary tables may also be occupying a significant amount of storage space. To check the temporary space usage, use the following command: SELECT * FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME='innodb_temporary'\G.
  • The following command lets you check the redo log size: SHOW VARIABLES LIKE 'innodb_log_file%';
  • You can check the size of general_log, if it is enabled, with the help of this command: SELECT ROUND(SUM(LENGTH(argument)/POW(1024,2)),2) AS GB from mysql.general_log;
  • If needed, you can truncate your log tables by using the API. For more information, see the instances.truncateLog reference page.
  • Learn more about setting and configuring slow-query logs.
Queries are blocked. It's possible for queries to lock the MySQL database causing all subsequent queries to block/timeout.

Connect to the database and execute this query:

SHOW PROCESSLIST.

The first item in the list may be the one holding the lock, which the subsequent items are waiting on.

The SHOW INNODB STATUS query can also be helpful.

You are unable to manually delete binary logs. Binary logs cannot be manually deleted. Binary logs are automatically deleted with their associated automatic backup, which generally happens after about seven days.
You want to find information about temporary files. A file named ibtmp1 is used for storing temporary data. This file is reset upon database restart. To find information about temporary file usage, connect to the database and execute the following query:

SELECT * FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME='innodb_temporary'\G

You want to find out about table sizes. This information is available in the database.

Connect to the database and execute the following query:

SELECT TABLE_SCHEMA, TABLE_NAME, sum(DATA_LENGTH+INDEX_LENGTH)/pow(1024,2) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT IN ('PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','SYS','MYSQL') GROUP BY TABLE_SCHEMA, TABLE_NAME;

mysqld got a signal 11. The instance probably crashed because queries are creating too many connections.

Refactor the queries so that they don't create so many connections.

InnoDB: page_cleaner: 1000ms intended loop took 5215ms. The settings might not be optimal. The page cleaner can't keep up with the rate of change on the instance. Once per second, the page cleaner scans the buffer pool for dirty pages to flush from the buffer pool to disk. The warning you see shows it has lots of dirty pages to flush, and it's taking more than one second to flush a batch of them to disk.

Shard the instance if possible. Using many smaller Cloud SQL instances is better than one large instance.

Temporary storage increased automatic storage. Automatic storage is enabled.

Restart deletes the temporary files but not reduce the storage. Only customer support can reset the instance size.

Data is being automatically deleted. Most likely a script is running somewhere in your environment.

Look in the logs around the time of the deletion and see if there's a rogue script running from a dashboard or another automated process.

The instance cannot be deleted. You might see the error message ERROR: (gcloud.sql.instances.delete) HTTP Error 409: The instance or operation is not in an appropriate state to handle the request, or the instance may have a INSTANCE_RISKY_FLAG_CONFIG flag status.

Some possible explanations include:

  • Another operation is in progress. Cloud SQL operations do not run concurrently. Wait for the other operation to complete.
  • The INSTANCE_RISKY_FLAG_CONFIG warning is triggered whenever at least one beta flag is being used. Remove the risky flag settings and restart the instance
The instance is stuck due to large temporary data size. The system can create many temporary tables at one time, depending on the queries and the load.

Unfortunately, you can't shrink the ibtmp1 file by any method other than restarting the service.

One mitigation option is to create the temporary table with ROW_FORMAT=COMPRESSED, so it is stored in file-per-table tablespaces in the temporary file directory. However, the downside is performance costs associated with creating and removing a file-per-table tablespace for each temporary table.

Fatal error during upgrade. Logs may reveal more, but in any case customer support may be needed to force re-create the instance.
Instance is stuck on restart after running out of disk space. Automatic storage increase capability isn't enabled.

If your instance runs out of storage, and the automatic storage increase capability isn't enabled, your instance goes offline. To avoid this issue, you can edit the instance to enable automatic storage increase.

Your on-premises primary instance is stuck. Google Cloud can't help with instances that are not in Cloud SQL.
Slow shutdown on restart. When an instance shuts down, any outstanding connections that don't end within 60 seconds make the shutdown unclean.

By having connections that last less than 60 seconds, most unclean shutdowns can be avoided, including connections from the database command prompt. If you keep these connections open for hours or days, shutdowns can be unclean.

A user cannot be deleted. The user probably has objects in the database that depend on it. You need to drop those objects or reassign them to another user.

Find out which objects are dependent on the user, then drop or reassign those objects to a different user.

This article discusses how to find the objects owned by the user.
Particular queries are running slow. Queries can be slow for many reasons, mostly due to specific database aspects. One reason that can involve Cloud SQL is network latency, when the source (writer or reader) resource and the destination (Cloud SQL) resource are in different regions.

Refer to general performance tips in particular.

For slow database inserts, updates, or deletes, consider the following actions:

  • If you enable the long_query_timeflag, you can check the logs for slow queries. Go to the Logs Explorer page for your project and run a query like this:
    resource.type="cloudsql_database"
    resource.labels.database_id="INSTANCE-ID"
    log_name="projects/PROJECT-ID/logs/cloudsql.googleapis.com%2Fmysql-slow.log"
          

    You can download the logs in JSON or TEXT format for local processing.

  • Check the locations of the writer and database; sending data a long distance introduces latency.
  • Check the location of the reader and database; latency affects read performance even more than write performance

To reduce the latency the recommendation is to locate both the source and destination resources in the same region.

Out of memory is indicated but monitoring charts don't show that. An instance can fail and report Out of memory but the Google Cloud Console or Cloud Monitoring charts seem to show there's still memory remaining.

There are other factors beside your workload that can impact memory usage, such as the number of active connections and internal overhead processes. These aren't always reflected in the monitoring charts.

Ensure that the instance has enough overhead to account for your workload plus some additional overhead.

Recovering a deleted instance. All data on an instance, including backups, is permanently lost when that instance is deleted.

To preserve your data, export it to Cloud Storage before you delete an instance.

The Cloud SQL Admin role includes the permission to delete the instance. To prevent accidental deletion, grant this role only as needed.

You want to rename an existing Cloud SQL instance. Renaming an existing instance is not supported.

There are other ways to accomplish the goal by creating a new instance.

  • You can clone the instance you want to rename and set a new name for the cloned instance. This allows you to create the new instance without having to import data manually. Just as when creating a new instance, the cloned instance has a new IP address.
  • You can export data from your instance into a Cloud Storage bucket, create a new instance with the new name you want, and then import the data into the new instance.

In both cases, you can delete your old instance after the operation is done. We recommend going with the cloning route since it has no impact on performance and doesn't require you to redo any instance configuration settings such as flags, machine type, storage size and memory.

Replication

Issue Troubleshooting
Read replica did not start replicating on creation. There's probably a more specific error in the log files. Inspect the logs in Cloud Logging to find the actual error.
Unable to create read replica - invalidFlagValue error. One of the flags in the request is invalid. It could be a flag you provided explicitly or one that was set to a default value.

First, check that the value of the max_connections flag is greater than or equal to the value on the primary.

If the max_connections flag is set appropriately, inspect the logs in Cloud Logging to find the actual error.

Unable to create read replica - unknown error. There's probably a more specific error in the log files. Inspect the logs in Cloud Logging to find the actual error.

If the error is: set Service Networking service account as servicenetworking.serviceAgent role on consumer project, then disable and re-enable the Service Networking API. This action creates the service account necessary to continue with the process.

Disk is full. The primary instance disk size can become full during replica creation. Edit the primary instance to upgrade it to a larger disk size.
The replica instance is using too much memory. The replica uses temporary memory to cache often-requested read operations, which can lead it to use more memory than the primary instance.

Restart the replica instance to reclaim the temporary memory space.

Replication stopped. The maximum storage limit was reached and automatic storage increase isn't enabled.

Edit the instance to enable automatic storage increase.

Replication lag is consistently high. The write load is too high for the replica to handle. Replication lag takes place when the SQL thread on a replica is unable to keep up with the IO thread. Some kinds of queries or workloads can cause temporary or permanent high replication lag for a given schema. Some of the typical causes of replication lag are:
  • Slow queries on the replica. Find and fix them.
  • All tables must have a unique/primary key. Every update on such a table without a unique/primary key causes full table scans on th replica.
  • Queries like DELETE ... WHERE field < 50000000 cause replication lag with row-based replication since a huge number of updates are piled up on the replica.

Some possible solutions include:

Changing parallel replication flags results in an error. An incorrect value is set for one of or more of these flags.

On the primary instance that's displaying the error message, set the parallel replication flags:

  1. Modify the binlog_transaction_dependency_tracking and transaction_write_set_extractionflags:
    • binlog_transaction_dependency_tracking=COMMIT_ORDER
    • transaction_write_set_extraction=OFF
  2. Add the slave_pending_jobs_size_max flag:

    slave_pending_jobs_size_max=33554432

  3. Modify the transaction_write_set_extraction flag:

    transaction_write_set_extraction=XXHASH64

  4. Modify the binlog_transaction_dependency_tracking flag:

    binlog_transaction_dependency_tracking=WRITESET

Replica creation fails with timeout. Long-running uncommitted transactions on the primary instance can cause read replica creation to fail.

Recreate the replica after stopping all running queries.