You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent
Apigee Edge documentation for this topic.
Symptoms
- API products, Developers, Apps do not get populated on the Apigee UI.
- API proxy deployments do not complete.
- Management API requests executed for reading and writing API products, Developers, and Apps fail./li>
Error messages
This section describes the possible error messages displayed when there is no network connectivity.
ApigeeIssue
If this issue was reported as an ApigeeIssue, when the following command is run:
kubectl -n APIGEE_NAMESPACE get apigeeissues
where APIGEE_NAMESPACE is the name for a grouping of Kubernetes resources.
the following error code is displayed:
NAME SEVERITY AGE URL control-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity
API products, Developers, and Apps Apigee UI pages
The API products, Developers, and Apps Apigee UI pages display the following error:
Error: no connections available from the Apigee connect agent(s): refer to documentation to triage further.
apigee-synchronizer logs
The following error message can be seen on apigee-synchronizer
logs:
{"level":"SEVERE","thread":"Apigee-Timer-3","mdc":{"action":"SYNC","env":"dev", "org":"example-hybrid-dev"},"className":"com.apigee.httpclient.adaptor.RequestAdaptor", "method":"doSend","severity":"SEVERE","message":"request failed [CONTEXT ratelimit_period\u003d\"1 MINUTES [skipped: 1]\" ]", "formattedDate":"2023-02-01T06:27:48.700Z","logger":"SERVICES.HTTPCLIENTSERVICE", "exceptionStackTrace":"java.net.SocketTimeoutException: connect timed out ... com.apigee.httpclient.adaptor.RequestAdaptor.send(RequestAdaptor.java:251) at com.apigee.hybrid.runtime.contract.sync.context.HttpContractDownloader.download(HttpContractDownloader.java:84) at com.apigee.hybrid.runtime.contract.sync.context.ControlPlaneReplicationContext.getContract(ControlPlaneReplicationContext.java:66)
apigee-watcher logs
The following error message can be seen on apigee-watcher
logs:
{"level":"error","ts":1675232549.5093117,"caller":"watcher/watcher.go:60", "msg":"error during watch","name":"mpstatus","error":"INTERNAL: error sending mp status to mgmt: INTERNAL: failed to send runtime status Post \"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\": dial tcp 172.217.24.42:443: i/o timeout","stacktrace": "edge-internal.git.corp.google.com/apigee-watcher.git/watcher. (*Watcher).Start.func1.1\n\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60"}
Possible Causes
Platform | Cause | Description |
---|---|---|
All | No network route available to the internet | If no network route is available to the internet, the Apigee runtime plane components will not be able to communicate with Apigee control plane APIs. |
All | Apigee API endpoints are not allow-listed | If there is a firewall in place, Apigee API endpoints may have not been allow-listed. |
GKE | Private Google Access is not enabled | If the deployment is on GKE, Private Google Access may have not been enabled in the subnet. |
All | Unknown network failure | If there is an unknown failure in the cluster network on connecting to the internet, this issue could occur. |
Cause: No network route available to the internet
Diagnosis
- Depending on the platform used, verify whether there is a network route available from the cluster network to the internet. For example, for more information on networking on GKE, see Network overview.
- Check with your infrastructure and networking team to see if the cluster network uses a forward proxy server for connecting to the internet.
Resolution
- Check with the network administrator and add a network route to the internet if possible.
-
If there is a forward proxy server used for communicating between the
cluster network and internet,
configure forward proxy server settings in Apigee using the
overrides.yaml
file and apply that change to the runtime plane.apigeectl apply --settings virtualhosts -f overrides/overrides.yaml
Cause: Apigee API endpoints are not allow-listed
Diagnosis
Check with the network administrator and verify whether the list of Apigee API endpoints are allow-listed in the firewall on the platform where Apigee is installed. For GKE, this could be Cloud Next Generation Firewall.
Resolution
If the above list of Apigee API endpoints are not allow-listed, contact the network administrator and complete that requirement.
Cause: Private Google Access is not enabled
Diagnosis
- If Apigee is deployed on a private GKE cluster that does not have access to the internet, Private Google Access needs to be enabled to allow the Apigee runtime plane components access to Google APIs internally.
-
In the Google Cloud console, go to the VPC networks page.
- Click the name of a network.
- On the Subnets tab, In the Private Google Access column, verify that the relevant subnet is enabled. If it is not, that's the reason for this failure.
Resolution
-
In the Google Cloud console, go to the VPC networks page.
- Click the name of a network.
- Click the name of a subnet.
- Click Edit.
- Select Private Google Access On.
- Click Save. Once enabled, this issue will be resolved.
Cause: Unknown network failure
Diagnosis
Contact your network administrator and verify whether there are any unknown failures in the cluster network.
Resolution
Work with your network administrator and resolve the issues in the cluster network. Once the network issues are fixed this problem will get resolved.
Must gather diagnostic information
If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:
- The Google Cloud Project ID.
- The name of the Apigee hybrid organization.
- The outputs of the following commands executed on all Kubernetes cluster nodes:
ping apigee.googleapis.com
traceroute apigee.googleapis.com
telnet apigee.googleapis.com 443
- The
overrides.yaml
file, masking any sensitive information. - The Kubernetes pod status in all namespaces:
kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt
- The Kubernetes cluster-info dump:
Generate Kubernetes cluster-info dump:
kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump
Zip Kubernetes cluster-info dump:
zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*