No network connectivity between runtime plane and control plane
Stay organized with collections
Save and categorize content based on your preferences.
You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent
Apigee Edge documentation for this topic.
Symptoms
API products, Developers, Apps do not get populated on the Apigee UI.
API proxy deployments do not complete.
Management API requests executed for reading and writing API products, Developers, and Apps fail.
Error messages
This section describes the possible error messages displayed when there is
no network connectivity.
ApigeeIssue
If this issue was reported as an ApigeeIssue, when the following command
is run:
kubectl -n APIGEE_NAMESPACE get apigeeissues
where APIGEE_NAMESPACE is the name for
a grouping of Kubernetes resources.
the following error code is displayed:
NAME SEVERITY AGE URL
control-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity
API products, Developers, and Apps Apigee UI pages
The API products, Developers, and Apps Apigee UI pages display the
following error:
The following error message can be seen on apigee-watcher logs:
{"level":"error","ts":1675232549.5093117,"caller":"watcher/watcher.go:60","msg":"error during watch","name":"mpstatus","error":"INTERNAL: error sending mp status to mgmt: INTERNAL: failed to send runtime status Post \"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\": dial tcp 172.217.24.42:443: i/o timeout","stacktrace":"edge-internal.git.corp.google.com/apigee-watcher.git/watcher. (*Watcher).Start.func1.1\n\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60"}
If there is an unknown failure in the cluster network on connecting to
the internet, this issue could occur.
Cause: No network route available to the internet
Diagnosis
Depending on the platform used, verify whether there is a network route
available from the cluster network to the internet. For example, for more
information on networking on GKE, see
Network overview.
Check with your infrastructure and networking team to see if the cluster
network uses a forward proxy server for connecting to the internet.
Resolution
Check with the network administrator and add a network route to the
internet if possible.
If there is a forward proxy server used for communicating between the
cluster network and internet,
configure forward proxy server settings in Apigee using the
overrides.yaml file and apply that change to the runtime
plane using the
Helm command:
Make sure to include all of the settings shown, including --atomic
so that the action rolls back on failure.
Your installation may or may not have been set up using an
ENV_GROUP_RELEASE_NAME that differs from ENV_GROUP.
For information on the settings, see
Install Apigee hybrid using Helm.
If the above list of Apigee API endpoints are not allow-listed, contact the
network administrator and complete that requirement.
Cause: Private Google Access is not enabled
Diagnosis
If Apigee is deployed on a
private GKE cluster that does not have access to the internet,
Private Google Access needs to be enabled to allow the Apigee
runtime plane components access to Google APIs internally.
In the Google Cloud console, go to the VPC networks page.
Click Save. Once enabled, this issue will be resolved.
Cause: Unknown network failure
Diagnosis
Contact your network administrator and verify whether there are any
unknown failures in the cluster network.
Resolution
Work with your network administrator and resolve the issues in the cluster
network. Once the network issues are fixed this problem will get resolved.
Must gather diagnostic information
If the problem persists even after following the above instructions, gather
the following diagnostic information and then contact Google Cloud Customer Care:
The Google Cloud Project ID.
The name of the Apigee hybrid organization.
The outputs of the following commands executed on all Kubernetes cluster nodes:
ping apigee.googleapis.com
traceroute apigee.googleapis.com
telnet apigee.googleapis.com 443
The overrides.yaml file, masking any sensitive information.
The Kubernetes pod status in all namespaces:
kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt
The Kubernetes cluster-info dump:
Generate Kubernetes cluster-info dump:
kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump
Zip Kubernetes cluster-info dump:
zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eThis document troubleshoots network connectivity issues between Apigee hybrid runtime plane components and the Apigee control plane, applicable only to Apigee hybrid users.\u003c/p\u003e\n"],["\u003cp\u003eSymptoms of network connectivity issues include non-populated data on the Apigee UI, failed API proxy deployments, and Management API request failures.\u003c/p\u003e\n"],["\u003cp\u003ePotential causes for these network issues include a lack of a network route to the internet, unallow-listed Apigee API endpoints, disabled Private Google Access on GKE, or other unknown network failures.\u003c/p\u003e\n"],["\u003cp\u003eResolutions involve establishing network routes, allow-listing Apigee API endpoints, enabling Private Google Access, or working with network administrators to resolve other network issues, depending on the identified root cause.\u003c/p\u003e\n"],["\u003cp\u003eIf issues persist, gather diagnostic information such as Google Cloud Project ID, organization name, network test outputs, \u003ccode\u003eoverrides.yaml\u003c/code\u003e, pod status, and cluster-info dumps, before contacting Google Cloud Customer Care.\u003c/p\u003e\n"]]],[],null,["# No network connectivity between runtime plane and control plane\n\n*You're viewing **Apigee** and **Apigee hybrid** documentation.\nThere is no equivalent\n[Apigee Edge](https://docs.apigee.com/api-platform/get-started/get-started) documentation for this topic.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptoms\n--------\n\n- API products, Developers, Apps do not get populated on the Apigee UI.\n- API proxy deployments do not complete.\n- Management API requests executed for reading and writing API products, Developers, and Apps fail.\n\nError messages\n--------------\n\n\nThis section describes the possible error messages displayed when there is\nno network connectivity.\n\n### ApigeeIssue\n\n\nIf this issue was reported as an ApigeeIssue, when the following command\nis run: \n\n```\nkubectl -n APIGEE_NAMESPACE get apigeeissues\n```\n\n\nwhere \u003cvar translate=\"no\"\u003eAPIGEE_NAMESPACE\u003c/var\u003e is the name for\na grouping of Kubernetes resources.\n\n\nthe following error code is displayed: \n\n```text\nNAME SEVERITY AGE URL\ncontrol-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity\n```\n\n### API products, Developers, and Apps Apigee UI pages\n\n\nThe API products, Developers, and Apps Apigee UI pages display the\nfollowing error: \n\n```actionscript-3\nError: no connections available from the Apigee connect agent(s):\nrefer to documentation to triage further.\n```\n\n### apigee-synchronizer logs\n\n\nThe following error message can be seen on `apigee-synchronizer`\nlogs: \n\n```gdscript\n{\"level\":\"SEVERE\",\"thread\":\"Apigee-Timer-3\",\"mdc\":{\"action\":\"SYNC\",\"env\":\"dev\",\n \"org\":\"example-hybrid-dev\"},\"className\":\"com.apigee.httpclient.adaptor.RequestAdaptor\",\n \"method\":\"doSend\",\"severity\":\"SEVERE\",\"message\":\"request failed\n [CONTEXT ratelimit_period\\u003d\\\"1 MINUTES [skipped: 1]\\\" ]\",\n \"formattedDate\":\"2023-02-01T06:27:48.700Z\",\"logger\":\"SERVICES.HTTPCLIENTSERVICE\",\n \"exceptionStackTrace\":\"java.net.SocketTimeoutException: connect timed out\n... com.apigee.httpclient.adaptor.RequestAdaptor.send(RequestAdaptor.java:251)\nat com.apigee.hybrid.runtime.contract.sync.context.HttpContractDownloader.download(HttpContractDownloader.java:84)\nat com.apigee.hybrid.runtime.contract.sync.context.ControlPlaneReplicationContext.getContract(ControlPlaneReplicationContext.java:66)\n```\n\n### apigee-watcher logs\n\n\nThe following error message can be seen on `apigee-watcher` logs: \n\n```transact-sql\n{\"level\":\"error\",\"ts\":1675232549.5093117,\"caller\":\"watcher/watcher.go:60\",\n\"msg\":\"error during watch\",\"name\":\"mpstatus\",\"error\":\"INTERNAL: error sending\n mp status to mgmt: INTERNAL: failed to send runtime status\n Post \\\"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\\\":\n dial tcp 172.217.24.42:443: i/o timeout\",\"stacktrace\":\n \"edge-internal.git.corp.google.com/apigee-watcher.git/watcher.\n (*Watcher).Start.func1.1\\n\\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60\"}\n```\n\nPossible Causes\n---------------\n\n### Cause: No network route available to the internet\n\n#### Diagnosis\n\n1. Depending on the platform used, verify whether there is a network route available from the cluster network to the internet. For example, for more information on networking on GKE, see [Network overview](/kubernetes-engine/docs/concepts/network-overview).\n2. Check with your infrastructure and networking team to see if the cluster network uses a forward proxy server for connecting to the internet.\n\n#### Resolution\n\n1. Check with the network administrator and add a network route to the internet if possible.\n2. If there is a forward proxy server used for communicating between the cluster network and internet, [configure forward proxy server settings](/apigee/docs/hybrid/latest/config-prop-ref#httpproxy) in Apigee using the `overrides.yaml` file and apply that change to the runtime plane using the [Helm command](/apigee/docs/hybrid/latest/helm-reference):\n\n Dry run: \n\n ```\n helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n --set envgroup=ENV_GROUP \\\n -f OVERRIDES_FILE \\\n --dry-run=server\n ```\n\n Make sure to include all of the settings shown, including `--atomic`\n so that the action rolls back on failure.\n\n Your installation may or may not have been set up using an\n \u003cvar translate=\"no\"\u003eENV_GROUP_RELEASE_NAME\u003c/var\u003e that differs from \u003cvar translate=\"no\"\u003eENV_GROUP\u003c/var\u003e.\n For information on the settings, see\n [Install Apigee hybrid using Helm](/apigee/docs/hybrid/latest/install-helm-charts).\n\n Install the chart: \n\n ```\n helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n --set envgroup=ENV_GROUP \\\n -f OVERRIDES_FILE\n ```\n\n### Cause: Apigee API endpoints are not allow-listed\n\n#### Diagnosis\n\nCheck with the network administrator and verify whether\n[the list of Apigee API endpoints](https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/firewall-rules) are allow-listed in the firewall on\nthe platform where Apigee is installed. For GKE, this could be\n[Cloud Next Generation Firewall](https://console.cloud.google.com/networking/firewalls/list).\n\n#### Resolution\n\n\nIf the above list of Apigee API endpoints are not allow-listed, contact the\nnetwork administrator and complete that requirement.\n\n### Cause: Private Google Access is not enabled\n\n#### Diagnosis\n\n1. If Apigee is deployed on a [private GKE cluster](/kubernetes-engine/docs/how-to/private-clusters) that does not have access to the internet, [Private Google Access](https://cloud.google.com/vpc/docs/private-google-access) needs to be enabled to allow the Apigee runtime plane components access to Google APIs internally.\n2. In the Google Cloud console, go to the **VPC networks** page.\n\n [Go to VPC networks](https://console.cloud.google.com/networking/networks/list)\n3. Click the name of a network.\n4. On the **Subnets** tab, In the **Private Google Access** column, verify that the relevant subnet is enabled. If it is not, that's the reason for this failure.\n\n#### Resolution\n\n1. In the Google Cloud console, go to the **VPC networks** page.\n\n [Go to VPC networks](https://console.cloud.google.com/networking/networks/list)\n2. Click the name of a network.\n3. Click the name of a subnet.\n4. Click edit **Edit**.\n5. Select **Private Google Access On**.\n6. Click **Save**. Once enabled, this issue will be resolved.\n\n### Cause: Unknown network failure\n\n#### Diagnosis\n\n\nContact your network administrator and verify whether there are any\nunknown failures in the cluster network.\n\n#### Resolution\n\n\nWork with your network administrator and resolve the issues in the cluster\nnetwork. Once the network issues are fixed this problem will get resolved.\n\nMust gather diagnostic information\n----------------------------------\n\n\nIf the problem persists even after following the above instructions, gather\nthe following diagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):\n\n1. The Google Cloud Project ID.\n2. The name of the Apigee hybrid organization.\n3. The outputs of the following commands executed on all Kubernetes cluster nodes: \n\n ```\n ping apigee.googleapis.com\n ``` \n\n ```\n traceroute apigee.googleapis.com\n ``` \n\n ```\n telnet apigee.googleapis.com 443\n ```\n4. The `overrides.yaml` file, masking any sensitive information.\n5. The Kubernetes pod status in all namespaces: \n\n ```\n kubectl get pods -A \u003e kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt\n ```\n6. The Kubernetes cluster-info dump: Generate Kubernetes cluster-info dump:\n\n ```\n kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump\n ```\n\n Zip Kubernetes cluster-info dump: \n\n ```\n zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*\n ```"]]
