Troubleshoot issues

This page explains various error scenarios, the error messages for those scenarios, and troubleshooting steps to resolve the errors.

Connectivity error scenarios

If your instance experiences connectivity issues, check the scenarios in this section to see if one of them is causing the problem.

If not, Telnet to one of your Redis nodes and run some simple Redis commands to see if the instance is responsive or not.

• If the node is unresponsive, see if any of the troubleshooting issues from Networking error scenarios are blocking your node's network connection. If not, contact Google Cloud Support.

Connection error caused by resources provisioned in different VPC networks

To connect to a Memorystore instance from a Google Cloud resource, such as a Compute Engine VM, the resources must be provisioned on the same authorized VPC network as the Redis instance.

Attempting to Telnet to a Memorystore instance from a resource in a different region or VPC network results in the following error message:

telnet: Unable to connect to remote host: Connection timed out

Connection error caused by deleted VPC network peering

Creating a Memorystore for Redis instance creates a VPC peering between your VPC network and a Google internal VPC network.

The network peering uses the following format:

redis-peer-############

If this network peering is deleted, attempting to Telnet to the Redis instance results in the following error message:

telnet: Unable to connect to remote host: Connection timed out

The easiest way to re-establish the deleted network peering is to create a new Memorystore for Redis instance. Creating a new Redis instance re-establishes the deleted network peering, so you can delete it and your original Redis instance has the network peering it needs.

Firewall rules block your instance IP addresses

Connectivity issues can arise if you create egress firewall rules that block the Redis port (6379) or the instance's IP address.

Make sure not to create network firewall rules that block the IP range of your Redis instances.

High CPU usage error scenarios

Redis instance unresponsiveness caused by improper use of expensive Redis commands

If your Redis instance experiences high latency, unresponsiveness, or connectivity issues, the issues may be caused by improper use of the following expensive Redis commands:

• KEYS
• LRANGE
• EVAL
• HGETALL
• ZRANGE

These commands can put heavy CPU pressure on your instance. Open source Redis discourages running KEYS in production environments, however you can use SCAN, which is a safer alternative to the KEYS command. Using LRANGE to query all or a large subset of your keyspace can demand high CPU resources. Using a complex Lua script with EVAL can cause high CPU usage.

HGETALL and ZRANGE can also return a very large number of keys, adversely affecting the performance of your server.

Before running expensive commands, you should check the size of the data structures that the command queries to ensure it will not cause latency.

If your instance experiences high latency or unresponsiveness, check your client side logs to find if any expensive command have been run. If so, note the time. Next, use Cloud Monitoring to view the redis.googleapis.com/stats/cpu_utilization metric. See if periods of high CPU utilization coincide with the same periods when the expensive commands were run.

We recommend against using the KEYS command in productions environments. For EVAL, use less complex Lua scripts. For LRANGE, reduce the number of keys in the keyset being queried in a single operation.

Networking error scenarios

Allocated IP range is exhausted or a conflicting route exists

When creating resources within the IP address range dedicated for Memorystore for Redis, you can exhaust all of the addresses, causing the error message below. Alternatively, there may be an route that conflicts with the IP address of the Redis instance you are trying to create.

These scenarios cause the following error message:

The IP ranges for the connection do not have enough available IPs. Allocate a new range or expand existing range and try again.

You can resolve this issue by allocating additional IP addresses, or removing the route conflict discrepancy. For more information on how to do this, see IP address range exhaustion.

You do not have a private services access connection established for your network

If your Redis instance uses the private services access connection mode and a private services access connection does not exist for your network, you may receive the following error:

Google private service access is not enabled. Enable private service access and try again

To resolve this issue, establish a private services access connection.

The network peering for private services access is deleted

Establishing a private services access connection creates a network peering connection called servicenetworking-googleapis-com, which appears in your project's VPC Network Peering page.

Deleting the network peering causes you to receive the following error for existing Redis instances:

• telnet: Unable to connect to remote host: Connection timed out

Deleting the network peering causes you to receive the following error when creating a Redis instance:

• Private services access is not configured correctly. For steps on how to verify the connection, check the documentation.

To resolve this issue, follow the last step of the gcloud instructions at Establishing a private services access connection.

Conflicting networking flags during Redis instance creation

If you use both the --reserved-ip-range parameter and the --connect-mode=private-service-access parameter, you receive the following error:

Reserved IP range is not supported for --connect-mode private services access

To resolve this issue, either use --reserved-ip-range with --connect-mode=direct-peering or use --connect-mode=PRIVATE_SERVICE_ACCESS.

You cannot use both at the same time, because the --reserved-ip-range parameter is not supported for the private services access connection mode.

Exceeding the subnetwork quota for your project

There is a limit on the number of subnetworks that can be created within your project. If you exceed this quota, you receive the following error message:

Internal network quota exceeded. Please request higher limit here: https://forms.gle/ZfVduUGq2iSYcYGm8 or

Unable to create instance. Network quota limit has been reached. Please request higher limit here: https://forms.gle/ZfVduUGq2iSYcYGm8

To resolve this issue, fill out the form in the error message, or contact Google Cloud Support.

Service project not attached to host project

If you are using Shared VPC, your service project is not attached to your host project if you receive the following error:

Invalid network name <network-name>. Project <project-name> referenced is not the host project for <service-project-name>.

To resolve this issue, attach your service project to your host project.

Incompatible use of direct peering connection mode and Shared VPC network during instance creation

You cannot create a Redis instance in a service project with the direct peering connection mode while designating a Shared VPC network from the host project for the instance.

The connection mode is set to direct-peering by default if you do not set a value for --connection-mode. If you attempt to use the direct peering connection mode during instance creation and also choose a Shared VPC network from the host project as the value for --network, you receive the following error:

Authorized_network must exist in the same project as redis instance

To resolve this issue, make sure you specify --connect-mode=PRIVATE_SERVICE_ACCESS in your Redis instance creation command, or choose an authorized VPC network in the same project as your Redis instance.

Unsupported Compute Engine IP address ranges

You cannot access Memorystore for Redis from Compute Engine VMs that have an IP address within the range 172.17.0.0/16, because that range is reserved for an internal component.

Errors connecting to your Redis instance from other Google Cloud resources

Errors connecting to your instance from serverless environments requiring a Serverless VPC Access connector

If you cannot connect to a Redis instance using one of the serverless environments that require a Serverless VPC Access connector, it is possible that you did not set up a Serverless VPC Access connector for your environment.

See Serverless VPC Access connector requirement for more details.

Errors when connecting to your instance using a Google Kubernetes Engine cluster

You cannot connect to a Memorystore for Redis instance from a GKE cluster without VPC-native/IP aliasing enabled on your cluster. It is easiest to enable VPC-native/IP aliasing during GKE cluster creation. When creating your cluster, select VPC Native under advanced options. For more details, see Creating VPC-native clusters.

Identity and Access Management ( IAM ) error scenarios

Restoring a deleted policy binding for a service account

Memorystore for Redis uses the following service accounts to manage your Redis instances:

• [e-mail]
• [e-mail]

Deleting the policy bindings for these service accounts prevents you from being able to create new instances.

If you attempt to create a Redis instance using gcloud in this scenario, you may receive the following error message:

(gcloud.redis.instances.create) FAILED_PRECONDITION: A required IAM policy might be missing. Please run this command:"gcloud projects add-iam-policy-binding <YOUR-PROJECT-ID> --member='serviceAccount:service-<YOUR-PROJECT-NUMBER>@cloud-redis.iam.gserviceaccount.com' --role='roles/redis.serviceAgent'" and try again.

To re-establish the policy binding for these service accounts, run one of the following commands, replacing variables with appropriate values. Run the command associated with the deleted service account.

gcloud projects add-iam-policy-binding project-id --member='serviceAccount:[e-mail]' --role='roles/servicenetworking.serviceAgent'

gcloud projects add-iam-policy-binding project-id --member='serviceAccount:[e-mail]' --role='roles/redis.serviceAgent'

Operation timeout errors

The following error scenarios result in an unresponsive Redis instance and/or instance/node operation timeouts.

Network partition error

Sometimes Google Cloud resources cannot communicate across zones within a region due to a network partition error in Google Cloud servers. This can cause your instance to lose connection, resulting in a timeout error.

After Google Cloud resolves the network partition error for the region or zone where your instance is provisioned, connectivity should resume normally.

In this scenario, you may see a connectivity error message such as the following:

telnet: Unable to connect to remote host: Connection timed out

If you are not able to identify the cause of the timeout error, reach out to Google Cloud Support.

Service project and host project not in the same VPC service control perimeter

If you are using Shared VPC and a VPC service control perimeter, and your Redis instance creation operation times out, this may indicate that your service project and host project are not in the same service perimeter. Your service project and host project must be in the same perimeter in order for your Redis instance to communicate with connecting clients over the Shared VPC network.

To see if you are experiencing this issue, check your Redis instance's audit logs for the following error:

violationReason: "NETWORK_NOT_IN_SAME_SERVICE_PERIMETER"

To resolve this issue, put your host network and your service network in the same service perimeter.

Troubleshooting import and export issues

This section outlines some common issues you may run into when using import and export for Memorystore for Redis.

Import and export buttons are disabled in the Google Cloud console

Issue: the user logged into the console does not have the redis.instances.import and/or redis.instances.export permissions required to import and/or export RDB files.

Solution: grant the permissions to the user, and refresh the instance details page.

The import operation completed but the data was not restored

If an import operation completes but the data is not restored, first check either the Google Cloud console or the command line for an error message, and resolve any issues described by the error message.

If there is a failure during the import process, the instance is recovered using an empty RDB file. You can attempt to restore the data by importing the same RDB file again, or using a different RDB file.

The import failed because the RDB file was too big

If you received the error message "Import RDB file gs://bucket/object.rdb size exceeds max memory 10GB", you should scale up your instance and retry the import. You can also try to import a smaller RDB file into your instance.

Troubleshooting Google Cloud CLI issues

If you run into an issue where a gcloud CLI command is unavailable, or if the command behaves differently from how it is documented, try updating the gcloud CLI:

gcloud components update

Stopping all ongoing commands and connections for a Redis instance

As Memorystore for Redis is a Google managed product, there are some commands that are blocked in your Redis instance in order to provide a safe and reliable environment. One of the restricted commands is CLIENT, which includes CLIENT KILL, used to stop commands.

If a Redis command is consuming a lot of CPU/RAM utilisation and is affecting your production environment, you need to restart the instance (for Basic Tier configurations), or to failover into a replica (for Standard Tier configurations). This restart/failover operation stops all the commands running on the Redis server, and ends all the ongoing connections.

Below you'll find commands to perform restarts or failovers for each Memorystore for Redis configuration.

Stopping commands in Standard tier Memorystore for Redis instances

gcloud redis instances failover INSTANCE_NAME --data-protection-mode=limited-data-loss

Warning: Any failover from the primary to the replica risks some data loss. The limited-data-loss mode keeps data loss to a minimum by verifying that the difference in synchronization between your primary and replica is below 30MB before initiating the failover. See manual failover for additional information.

Stopping commands in Basic tier Memorystore for Redis instances

The only way to perform a restart in a Memorystore for Redis instances is changing its configuration, like scaling up the instance. An example of a command that you can run to restart your instance can be found below.

gcloud redis instances update INSTANCE_NAME --region REGION_ID --size NUMBER_GB

After you scale your instance to a different size, you can run another scaling operation to return it to the original size.

Issues with the domain restricted sharing organization policy

Depending on when your instance was created, Memorystore for Redis uses one of two different service account formats. To identify which service account format your instance uses, see Memorystore for Redis service account format.

There is a known issue where the iam.allowedPolicyMemberDomains organization policy causes errors when used with Memorystore for Redis instances that use the [PROJECT_NUMBER][e-mail] service account format.

In these scenarios, you may encounter this error: One or more users named in the policy do not belong to a permitted customer.

There are two options to resolve this issue.

Option 1

You can create a new instance. Newly created instances have the appropriate service account format which is compatible with the organization policy. If it is critical to maintain your cache contents, you can perform an export to backup existing data and an import to the new instance. Please note that a newly created instance has a new service IP address that needs to be configured in your application.

Option 2

If it is infeasible to recreate your Memorystore instance, follow the Forcing account access workaround.