Troubleshooting

This page explains various error scenarios, the error messages for those scenarios, and troubleshooting steps to resolve the errors.

Connectivity error scenarios

If your instance experiences connectivity issues, check the scenarios in this section to see if one of them is causing the problem.

If not, Telnet to one of your Redis nodes and run some simple Redis commands to see if the instance is responsive or not.

Connection error caused by resources provisioned in different regions or different VPC networks

To connect to a Memorystore instance from a Google Cloud resource, such as a Compute Engine VM, the resources must be provisioned in the same region and in the same VPC network.

Attempting to Telnet to a Memorystore instance from a resource in a different region or VPC network results in the following error message:

telnet: Unable to connect to remote host: Connection timed out

Connection error caused by deleted VPC network peering

Creating a Memorystore for Redis instance creates a VPC peering between your VPC network and a Google internal VPC network.

The network peering uses the following format:

redis-peer-############

If this network peering is deleted, attempting to Telnet to the Redis instance results in the following error message:

telnet: Unable to connect to remote host: Connection timed out

The easiest way to re-establish the deleted network peering is to create a new Memorystore for Redis instance. Creating a new Redis instance re-establishes the deleted network peering, so you can delete it and your original Redis instance has the network peering it needs.

Firewall rules block your instance IP addresses

Connectivity issues can arise if you create egress firewall rules that block the Redis port (6379) or the instance's IP address.

Make sure not to create network firewall rules that block the IP range of your Redis instances.

High CPU usage error scenarios

Redis instance unresponsiveness caused by improper use of expensive Redis commands

If your Redis instance experiences high latency, unresponsiveness, or connectivity issues, the issues may be caused by improper use of the following expensive Redis commands:

These commands can put heavy CPU pressure on your instance. Open source Redis discourages running KEYS in production environments. Using LRANGE to query all or a large subset of your keyspace can demand high CPU resources. Using a complex Lua script with EVAL can cause high CPU usage.

If your instance experiences high latency or unresponsiveness, check your client side logs to find if any expensive command have been run. If so, note the time. Next, use Cloud Monitoring to view the redis.googleapis.com/stats/cpu_utilization metric. See if periods of high CPU utilization coincide with the same periods when the expensive commands were run.

We recommend against using the KEYS command in productions environments. For EVAL, use less complex Lua scripts. For LRANGE, reduce the number of keys in the keyset being queried in a single operation.

Networking error scenarios

Allocated IP range is exhausted

Memorystore for Redis requires that you use a private services access connection and an associated IP address range for that connection. It is possible to exhaust the IP addresses available in that range with Redis instances and other Google Cloud resources.

If the IP addresses are exhausted, creating an instance returns the following error message:

The IP ranges for the connection do not have enough available IPs. Allocate a new range or expand existing range and try again.

You can resolve this issue by allocating additional IP addresses. For more information on how to do this, see IP address range exhaustion.

You do not have a private services access connection established for your network

If your Redis instance uses the private services access connection mode and a private services access connection does not exist for your network, you may receive the following error:

Google private service access is not enabled. Enable private service access and try again

To resolve this issue, establish a private services access connection.

The network peering for private services access is deleted

Establishing a private services access connection creates a network peering connection called servicenetworking-googleapis-com, which appears in your project's VPC Network Peering page.

Deleting the network peering causes you to receive the following error for existing Redis instances:

  • telnet: Unable to connect to remote host: Connection timed out

Deleting the network peering causes you to receive the following error when creating a Redis instance:

  • Private services access is not configured correctly. For steps on how to verify the connection, check the documentation.

To resolve this issue, follow the last step of the gcloud instructions at Establishing a private services access connection.

Conflicting networking flags during Redis instance creation

If you use both the --reserved-ip-range parameter and the --connect-mode=private-service-access parameter, you receive the following error:

Reserved IP range is not supported for --connect-mode private services access

To resolve this issue, either use --reserved-ip-range with --connect-mode=direct-peering or use --connect-mode=PRIVATE_SERVICE_ACCESS.

You cannot use both at the same time, because the --reserved-ip-range parameter is not supported for the private services access connection mode.

Exceeding the subnetwork quota for your project

There is a limit on the number of subnetworks that can be created within your project. If you exceed this quota, you receive the following error message:

Internal network quota exceeded. Please request higher limit here: https://forms.gle/ZfVduUGq2iSYcYGm8

To resolve this issue, fill out the form in the error message, or contact Google Cloud Support.

Service project not attached to host project

If you are using Shared VPC, your service project is not attached to your host project if you receive the following error:

Invalid network name <network-name>. Project <project-name> referenced is not the host project for <service-project-name>.

To resolve this issue, attach your service project to your host project.

Incompatible use of direct peering connection mode and Shared VPC network during instance creation

You cannot create a Redis instance in a service project with the direct peering connection mode while designating a Shared VPC network from the host project for the instance.

The connection mode is set to direct-peering by default if you do not set a value for --connection-mode. If you attempt to use the direct peering connection mode during instance creation and also choose a Shared VPC network from the host project as the value for --authorized-network, you receive the following error:

Authorized_network must exist in the same project as redis instance

To resolve this issue, make sure you specify --connect-mode=PRIVATE_SERVICE_ACCESS in your Redis instance creation command, or choose a regular VPC network in the same project as your Redis instance.

Unsupported Compute Engine IP address ranges

You cannot access Memorystore for Redis from Compute Engine VMs that have an IP address within the range 172.17.0.0/16, because that range is reserved for an internal component.

Errors connecting to your Redis instance from other GCP resources

Errors connecting to your instance from serverless environments requiring a Serverless VPC Access connector

If you cannot connect to a Redis instance using one of the serverless environments that require a Serverless VPC Access connector, it is possible that you did not set up a Serverless VPC Access connector for your environment.

See Serverless VPC Access connector requirement for more details.

Errors when connecting to your instance using a Google Kubernetes Engine cluster

You cannot connect to a Memorystore for Redis instance from a GKE cluster without VPC-native/IP aliasing enabled on your cluster. It is easiest to enable VPC-native/IP aliasing during GKE cluster creation. When creating your cluster, select VPC Native under advanced options. For more details, see Creating VPC-native clusters.

Cloud Identity and Access Management ( Cloud IAM ) error scenarios

Restoring a deleted policy binding for a service account

Memorystore for Redis uses the following service accounts to manage your Redis instances:

  • service-project-number@service-networking.iam.gserviceaccount.com
  • service-project-number@cloud-redis.iam.gserviceaccount.com

Deleting the policy bindings for these service accounts prevents you from being able to create new instances.

If you attempt to create a Redis instance using gcloud in this scenario, you may receive the following error message:

(gcloud.redis.instances.create) FAILED_PRECONDITION: A required IAM policy might be missing. Please run this command:"gcloud projects add-iam-policy-binding <YOUR-PROJECT-ID> --member='serviceAccount:service-<YOUR-PROJECT-NUMBER>@cloud-redis.iam.gserviceaccount.com' --role='roles/redis.serviceAgent'" and try again. com.google.apps.framework.request.StatusException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: A required IAM policy might be missing. Please run this command:"gcloud projects add-iam-policy-binding <YOUR-PROJECT-ID> --member='serviceAccount:service-<YOUR-PROJECT-NUMBER>@cloud-redis.iam.gserviceaccount.com' --role='roles/redis.serviceAgent'" and try again.

To re-establish the policy binding for these service accounts, run one of the following commands, replacing variables with appropriate values. Run the command associated with the deleted service account.

gcloud projects add-iam-policy-binding project-id --member='serviceAccount:service-project-number@service-networking.iam.gserviceaccount.com' --role='roles/servicenetworking.serviceAgent'
gcloud projects add-iam-policy-binding project-id --member='serviceAccount:service-project-number@cloud-redis.iam.gserviceaccount.com' --role='roles/redis.serviceAgent'

Operation timeout errors

The following error scenarios result in an unresponsive Redis instance and/or instance/node operation timeouts.

Network partition error

Sometimes Google Cloud resources cannot communicate across zones within a region due to a network partition error in Google Cloud servers. This can cause your instance to lose connection, resulting in a timeout error.

After Google Cloud resolves the network partition error for the region or zone where your instance is provisioned, connectivity should resume normally.

In this scenario, you may see a connectivity error message such as the following:

telnet: Unable to connect to remote host: Connection timed out

If you are not able to identify the cause of the timeout error, reach out to Google Cloud Support.

Service project and host project not in the same VPC service control perimeter

If you are using Shared VPC and a VPC service control perimeter, and your Redis instance creation operation times out, this may indicate that your service project and host project are not in the same service perimeter. Your service project and host project must be in the same perimeter in order for your Redis instance to communicate with connecting clients over the Shared VPC network.

To see if you are experiencing this issue, check your Redis instance's audit logs for the following error:

violationReason: "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"

To resolve this issue, put your host network and your service network in the same service perimeter.

Troubleshooting import and export issues

This section outlines some common issues you may run into when using import and export for Memorystore for Redis.

Import and export buttons are disabled in the Cloud Console

Issue: the user logged into the console does not have the redis.instances.import and/or redis.instances.export permissions required to import and/or export RDB files.

Solution: grant the permissions to the user, and refresh the instance details page.

The import operation completed but the data was not restored

If an import operation completes but the data is not restored, first check either the Cloud Console or the command line for an error message, and resolve any issues described by the error message.

If there is a failure during the import process, the instance is recovered using an empty RDB file. You can attempt to restore the data by importing the same RDB file again, or using a different RDB file.

The import failed because the RDB file was too big

If you received the error message "Import RDB file gs://bucket/object.rdb size exceeds max memory 10GB", you should scale up your instance and retry the import. You can also try to import a smaller RDB file into your instance.

Troubleshooting gcloud command-line tool issues

If you run into an issue where a gcloud tool command is unavailable, or if the command behaves differently from how it is documented, try updating the gcloud SDK:

gcloud components update