No network connectivity between runtime plane and control plane

You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Symptoms

API products, Developers, Apps do not get populated on the Apigee UI.
API proxy deployments do not complete.
Management API requests executed for reading and writing API products, Developers, and Apps fail./li>

Error messages

This section describes the possible error messages displayed when there is no network connectivity.

ApigeeIssue

If this issue was reported as an ApigeeIssue, when the following command is run:

kubectl -n APIGEE_NAMESPACE get apigeeissues

where APIGEE_NAMESPACE is the name for a grouping of Kubernetes resources.

the following error code is displayed:

NAME                               SEVERITY    AGE URL
control-plane-connectivity-failure Error       1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity

API products, Developers, and Apps Apigee UI pages

The API products, Developers, and Apps Apigee UI pages display the following error:

Error: no connections available from the Apigee connect agent(s):
refer to documentation to triage further.

apigee-synchronizer logs

The following error message can be seen on apigee-synchronizer logs:

{"level":"SEVERE","thread":"Apigee-Timer-3","mdc":{"action":"SYNC","env":"dev",
  "org":"example-hybrid-dev"},"className":"com.apigee.httpclient.adaptor.RequestAdaptor",
  "method":"doSend","severity":"SEVERE","message":"request failed
  [CONTEXT ratelimit_period\u003d\"1 MINUTES [skipped: 1]\" ]",
  "formattedDate":"2023-02-01T06:27:48.700Z","logger":"SERVICES.HTTPCLIENTSERVICE",
  "exceptionStackTrace":"java.net.SocketTimeoutException: connect timed out
... com.apigee.httpclient.adaptor.RequestAdaptor.send(RequestAdaptor.java:251)
at com.apigee.hybrid.runtime.contract.sync.context.HttpContractDownloader.download(HttpContractDownloader.java:84)
at com.apigee.hybrid.runtime.contract.sync.context.ControlPlaneReplicationContext.getContract(ControlPlaneReplicationContext.java:66)

apigee-watcher logs

The following error message can be seen on apigee-watcher logs:

{"level":"error","ts":1675232549.5093117,"caller":"watcher/watcher.go:60",
"msg":"error during watch","name":"mpstatus","error":"INTERNAL: error sending
  mp status to mgmt: INTERNAL: failed to send runtime status
  Post \"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\":
  dial tcp 172.217.24.42:443: i/o timeout","stacktrace":
  "edge-internal.git.corp.google.com/apigee-watcher.git/watcher.
  (*Watcher).Start.func1.1\n\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60"}

Possible Causes

Platform	Cause	Description
All	No network route available to the internet	If no network route is available to the internet, the Apigee runtime plane components will not be able to communicate with Apigee control plane APIs.
All	Apigee API endpoints are not allow-listed	If there is a firewall in place, Apigee API endpoints may have not been allow-listed.
GKE	Private Google Access is not enabled	If the deployment is on GKE, Private Google Access may have not been enabled in the subnet.
All	Unknown network failure	If there is an unknown failure in the cluster network on connecting to the internet, this issue could occur.

Cause: No network route available to the internet

Diagnosis

Depending on the platform used, verify whether there is a network route available from the cluster network to the internet. For example, for more information on networking on GKE, see Network overview.
Check with your infrastructure and networking team to see if the cluster network uses a forward proxy server for connecting to the internet.

Resolution

Check with the network administrator and add a network route to the internet if possible.
If there is a forward proxy server used for communicating between the cluster network and internet, configure forward proxy server settings in Apigee using the overrides.yaml file and apply that change to the runtime plane.
```
apigeectl apply --settings virtualhosts -f overrides/overrides.yaml
```

Cause: Apigee API endpoints are not allow-listed

Diagnosis

Check with the network administrator and verify whether the list of Apigee API endpoints are allow-listed in the firewall on the platform where Apigee is installed. For GKE, this could be Cloud Next Generation Firewall.

Resolution

If the above list of Apigee API endpoints are not allow-listed, contact the network administrator and complete that requirement.

Cause: Private Google Access is not enabled

Diagnosis

If Apigee is deployed on a private GKE cluster that does not have access to the internet, Private Google Access needs to be enabled to allow the Apigee runtime plane components access to Google APIs internally.
In the Google Cloud console, go to the VPC networks page.

Go to VPC networks
Click the name of a network.
On the Subnets tab, In the Private Google Access column, verify that the relevant subnet is enabled. If it is not, that's the reason for this failure.

Resolution

In the Google Cloud console, go to the VPC networks page.

Go to VPC networks
Click the name of a network.
Click the name of a subnet.
Click Edit.
Select Private Google Access On.
Click Save. Once enabled, this issue will be resolved.

Cause: Unknown network failure

Diagnosis

Contact your network administrator and verify whether there are any unknown failures in the cluster network.

Resolution

Work with your network administrator and resolve the issues in the cluster network. Once the network issues are fixed this problem will get resolved.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

The Google Cloud Project ID.
The name of the Apigee hybrid organization.

The outputs of the following commands executed on all Kubernetes cluster nodes:

ping apigee.googleapis.com

traceroute apigee.googleapis.com

telnet apigee.googleapis.com 443

The overrides.yaml file, masking any sensitive information.

The Kubernetes pod status in all namespaces:

kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt

The Kubernetes cluster-info dump:

Generate Kubernetes cluster-info dump:

kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump

Zip Kubernetes cluster-info dump:

zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*