Troubleshooting

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

Istio 404 (Not Found) error

Debugging a 404 (Not Found) error on Istio can be frustrating. Hopefully this will give you a place to start tracking down where things may be going wrong.

Wildcard Gateway conflict

There can be only one Gateway definition that uses a wildcard "*" hosts value. If you've deployed anything else that includes a wildcard Gateway, client calls will fail with a 404 status.

Example:

$ istioctl get gateways
GATEWAY NAME         HOSTS     NAMESPACE   AGE
bookinfo-gateway     *         default     20s
httpbin-gateway      *         default     3s

If so, you'll need to delete or change one of the conflicting gateways.

Search for where the route is failing

Istio is like an onion (or, perhaps, an Ogre), it has layers. A systematic way to debug a 404 is to work outward from the target.

The backend workload

Verify you can access the workload from the sidecar:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl localhost:80/headers

The backend sidecar

Set your service address and get the IP address of the workload pod.

SERVICE=httpbin.default.svc.cluster.local:80
  POD_IP=$(kubectl get pod $WORKLOAD_POD -o jsonpath='{.status.podIP}')

Access the workload through the sidecar:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl -v http://$SERVICE/headers --resolve "$SERVICE:$POD_IP"

Or, if Istio mTLS is enabled:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl -v https://$SERVICE/headers --resolve "$SERVICE:$POD_IP" --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem --insecure

The gateway (or a frontend sidecar)

Access the service from the gateway:

kubectl -n istio-system exec $GATEWAY_POD -- curl -v http://$SERVICE/header

Or, if Istio mTLS is enabled:

kubectl -n istio-system exec $GATEWAY_POD -- curl -v https://$SERVICE/headers --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem --insecure

Missing analytics

If you aren't seeing analytics in the Analytics UI, consider these possible causes:

  • Apigee intake can be delayed a few minutes
  • Envoy gRPC Access Log not configured correctly
  • Envoy cannot reach Remote Service
  • Remote Service is failing upload

Missing or bad API key not being rejected

If API key validation is not working properly, consider these possible causes:

Direct proxy

Check the ext-authz configuration.

Sidecar
  • Be sure listener is configured for intercept.
  • Check the ext-authz configuration.

Invalid requests are being checked and allowed

  • Remote Service configured for fail open
  • Envoy not configured for RBAC checks

For information on how to address these issues, refer to the following Envoy documentation topic: External Authorization, and refer to information about the failure_mode_allow property. This property allows you to change the filter's behavior on errors.

Missing or bad JWT not being rejected

The probable cause is that the Envoy JWT filter is not configured.

Valid API key fails

Probable causes

  • Envoy cannot reach the remote service
  • Your credentials are not valid
  • Apigee API Product not configured for target and env
  • Envoy is not aware of Apigee API Product

Troubleshooting steps

Check your API Product on Apigee

  • Is it enabled for your environment (test vs prod)?

    The product must be bound to the same environment as your Remote Service.

  • Is it bound to the target you're accessing?

    Check the Apigee remote service targets section. Remember, the service name must be a fully qualified host name. If it's an Istio service, the name will be something like helloworld.default.svc.cluster.localcode> - which represents the helloworld service in the default namespace.

  • Does the Resource Path match your request?

    Remember, a path like / or /** will match any path. You may also use '*' or '**' wildcards for matching.

  • Do you have a Developer App?

    The API Product must be bound to a Developer App to check its keys.

  • Is Apigee API Product operationConfigType set to remoteservice?

    Check the API Product configuration using the Apigee Management API, and confirm that operationConfigType is set to remoteservice.

Check your request

  • Are you passing the Consumer Key in the x-api-key header

    Example:

    curl http://localhost/hello -H "x-api-key: wwTcvmHvQ7Dui2qwj43GlKJAOwmo"
  • Are you using a good Consumer Key?

    Ensure the Credentials from the App you're using are approved for your API Product.

Check the Remote Service logs

  1. Start the Remote Service with logging at the debug level. See Setting remote service log levels.

    Use -l debug option on the command line. For example:

    apigee-remote-service-envoy -l debug
  2. Attempt to access your target and check the logs

    Check the logs for a line that looks something like this:

    Resolve api: helloworld.default.svc.cluster.local, path: /hello, scopes: []
Selected: [helloworld]
Eliminated: [helloworld2 doesn't match path: /hello]

Setting remote service log levels

Using a command line flag, you can start the remote service in one of the following debug modes, in order of verbosity, where debug is most verbose and error is least:

  • debug - The most verbose logging mode.
  • info - The default.
  • warn
  • error - Least verbose logging mode.

For example, to start the service at the debug level: 

apigee-remote-service-envoy -l debug