Troubleshoot issues

This page explains various error scenarios, the error messages for those scenarios, and troubleshooting steps to resolve the errors.

Networking error scenarios

Allocated IP range is exhausted

Memorystore for Memcached requires that you use a private services access connection and an associated IP address range for that connection. It is possible to exhaust the IP addresses available in that range with Memcached instances and other Google Cloud resources.

If the IP addresses are exhausted, creating an instance returns the following error message:

ERROR: (gcloud.memcache.instances.create) { "code": 9, "message": "Unable to create instance. The allocated private IP address space is exhausted. For information on expanding the allocation, see https://cloud.google.com/vpc/docs/configure-private-services-access#modify-ip-range" }

You can resolve this issue by allocating additional IP addresses. For more information on how to do this, see IP address range exhaustion.

You do not have a private services access connection established for your network

If you attempt to create a Memorystore for Memcached instance without a private services access connection established for your project's network, the creation will fail with the following error:

Google private service access is not enabled. Enable private service access and try again

In order to successfully create an instance, follow the private services access setup instructions at Establishing a private services access connection.

The network peering for private services access is deleted

Establishing a private services access connection creates a network peering connection called servicenetworking-googleapis-com, which appears in your project's VPC Network Peering page.

Deleting this network peering may cause you to receive the following errors:

telnet: Unable to connect to remote host: Connection timed out
Unable to create instance. Enable private service access for the authorized network and try again. For information on enabling private service access, see https://cloud.google.com/service-infrastructure/docs/enabling-private-services-access

To resolve this issue, follow the last step of the gcloud instructions at Establishing a private services access connection.

Firewall rules block your instance IP addresses

Connectivity issues can arise if you create firewall rules that block IP addresses from the allocated IP range associated with your private services access connection.

Make sure not to create network firewall rules that block the IP range of your Memcached instances.

Errors connecting to your Memcached instance from other Google Cloud resources

Errors connecting to your instance from the App Engine standard environment or Cloud Run functions

If you cannot connect to a Memcached instance using the App Engine standard environment or Cloud Run functions, it is possible that you did not set up a Serverless VPC Access connector for your environment.

For instructions on setting up Serverless VPC Access for the App Engine standard environment, see Connecting to a VPC network.

For instructions on setting up Serverless VPC Access for Cloud Run functions, see Connecting to a VPC network.

Errors when connecting to your instance using a Google Kubernetes Engine cluster

You cannot connect to a Memorystore for Memcached instance from a GKE cluster without VPC-native/IP aliasing enabled on your cluster. It is easiest to enable VPC-native/IP aliasing during GKE cluster creation. When creating your cluster, select VPC Native under advanced options. For more details, see Creating VPC-native clusters.

Errors when connecting to your instance using the App Engine flexible environment or Compute Engine VMs

If your instance experiences connectivity issues, follow these troubleshooting steps:

Ensure that your Memcached instance is provisioned in the same region and network as the client trying to access it.
Check that your project does not have any network firewall rules blocking your Memcached instance's node IPs.
- To view the IP addresses of your nodes, view the nodes tab of your Memcached instance in the Google Cloud console.
Telnet to one of your memcached nodes, and run some simple Memcached set and get commands to see if the instance is responsive or not.
- If the node is unresponsive, see if any of the troubleshooting issues from Networking error scenarios are blocking your node's network connection. If not, contact Google Cloud Support.

Identity and Access Management ( IAM ) error scenarios

Restoring a deleted service account

Memorystore for Memcached uses the following service accounts to manage your Memcached instances:

service-project-number@service-networking.iam.gserviceaccount.com
service-project-number@cloud-memcache-sa.iam.gserviceaccount.com

Deleting the policy bindings for these service accounts prevents you from being able to create new instances.

If you attempt to create a Memcached instance using gcloud in this scenario, you may receive the following error message:

ERROR: (gcloud.memcache.instances.create) { "code": 13, "message": "an internal error has occurred" }

To re-establish the policy binding for these service accounts, run one of the following commands, replacing variables with appropriate values. Run the command associated with the deleted service account.

gcloud projects add-iam-policy-binding project-id --member='serviceAccount:service-project-number@service-networking.iam.gserviceaccount.com' --role='roles/servicenetworking.serviceAgent'

gcloud projects add-iam-policy-binding project-id --member='serviceAccount:service-project-number@cloud-memcache-sa.iam.gserviceaccount.com' --role='roles/memcache.serviceAgent'

Operation timeout errors

The following error scenarios result in an unresponsive Memcached instance and/or instance/node operation timeouts.

Network partition error

Sometimes Google Cloud resources cannot communicate across zones within a region due to a network partition error in Google Cloud servers. This can cause your instance to lose connection, resulting in a timeout error.

After Google Cloud resolves the network partition error for the region or zone where your instance is provisioned, connectivity should resume normally.

In this scenario, you may see a connectivity error message such as the following:

telnet: Unable to connect to remote host: Connection timed out

If you are not able to identify the cause of the timeout error, reach out to Google Cloud Support.

Improper application reference to node IPs

If your client is timing out when attempting to connect to a node(s), one potential issue is that your application may not be specifying the proper IP address of the node(s) to which you are trying to connect. Verify that your application is pointing to the IP addresses of the nodes in your cluster.

In this scenario, you may see a connectivity error message, such as the following message:

telnet: Unable to connect to remote host: Connection timed out

To resolve this issue, look up the IP addresses of your nodes, and make sure that your application points accurately to the node addresses.

Additional error scenarios

Unsupported Compute Engine IP address ranges

You cannot access Memorystore for Memcached from Compute Engine VMs that have an IP address within the range 172.17.0.0/16, because that range is reserved for an internal component.

Unbalanced node distribution across zones in a region

If a zone in a Google Cloud region is unavailable at the time you provision a Memcached instance, nodes are created in the available zones. This can lead to an imbalanced distribution of Memached nodes across zones in a region.

This should not have a noticeable impact on your application performance. However, ideally, your nodes should be distributed across all zones in a region to protect your data from zonal outages.

If you are unsatisfied with the distribution of nodes across zones when you create an instance, you can either wait until that zone is back online to create your instance, or you can use the following strategy to attempt to re-distribute nodes across zones at a later time:

To re-distribute Memcached nodes among the available nodes in a zone, scale up your instance, then scale back down to the desired number of nodes. If the desired zone(s) is back online, the scaling process will re-distribute nodes evenly across available zones. Also, note that scaling causes a cache flush for the data in your Memcached instance.