Troubleshooting Cloud Endpoints on Compute Engine

This page presents troubleshooting techniques when the Extensible Service Proxy (ESP) was deployed on a Compute Engine virtual machine (VM).

Viewing logs on the VM instance

You can troubleshoot problems by looking at the Cloud Endpoints Runtime log on the VM instance.

To view the log:

  1. SSH into your virtual machine:

    gcloud config set project [YOUR_PROJECT_ID]
    
    gcloud compute ssh [INSTANCE_NAME]
    

    Replace [YOUR_PROJECT_ID] and [INSTANCE_NAME] with your Google Cloud Platform (GCP) project ID and virtual machine instance name, respectively.

  2. View the nginx error log:

    If you are running endpoints-runtime on a raw VM:

    tail -f /var/log/nginx/error.log
    

    If you are running endpoints-runtime within Docker:

    docker ps
    
    docker logs [CONTAINER_NAME]
    

    Replace [CONTAINER_NAME] with the name of your container.

Displaying ESP status

To display ESP status:

  1. SSH into your virtual machine:

    gcloud config set project [YOUR_PROJECT_ID]
    
    gcloud compute ssh [INSTANCE_NAME]
    

    Replace [YOUR_PROJECT_ID] and [INSTANCE_NAME] with your GCP project ID and virtual machine instance name, respectively.

  2. Run the following command to get the name of the ESP container (typically the container name is esp):

    docker ps
    
  3. Run the following command to get a bash shell in the container:

    docker exec -it [ESP_CONTAINER_NAME] /bin/bash
    

    Replace [ESP_CONTAINER_NAME] with the name of the ESP container from the previous step.

  4. Install curl.

  5. Enter the following:

    curl http://localhost:8090/endpoints_status
    

Getting the service configuration ID

If you set rollout_strategy to managed when you started ESP, and you need to find out the configuration ID that an instance of ESP is using, near the end of the output from the curl http://localhost:8090/endpoints_status command, you see something like the following:

      "serviceConfigRollouts": {
          "rolloutId": "2017-08-09r27",
          "percentages": {
               "2017-08-09r26": "100"
          }
      }

The value in the rolloutId is the service configuration ID that ESP is using. This configuration ID should match the latest deployed configuration. You can view the deployment history on the Endpoints Services page in the GCP Console and view changes made to the service configuration. See Comparing Configuration Files .

Troubleshooting issues on Compute Engine without Docker

This section applies to deployments where you installed ESP directly on a Compute Engine VM instance and not in a Docker container.

Requests are not forwarded from ESP to your backend

You can configure ESP to listen on any valid port. Whatever port you have configured, make sure that you do not have any other applications using that port.

To confirm the port that ESP is listening on:

For example purposes, this section uses port 80, which is a typical port for ESP to listen on.

  1. Make sure ESP is running by looking for the nginx process:

    pgrep --list-full nginx
    
  2. Open the file /etc/default/nginx and make sure you see a line like the following:

    PORT=80
    
  3. To confirm that ESP is listening on port 80:

    sudo netstat -tnlp
    

    You should see output like the following that shows nginx is listening on port 80:

    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13543/nginx.conf
    ...
    

    If netstat is not installed on your instance, you can run the following command to verify that nginx is running on port 80:

    sudo ss -tnlp
    

Fetching service config failed

ERROR:Fetching service config failed (status code 403, reason Forbidden, url ...

When you attempt to start ESP and you see this error, this indicates that the service configuration for your API is not in the same GCP project that you created the Compute Engine VM instance in.

  1. In the GCP Console, go to the VM Instances page.

    Go to the VM Instances page

  2. Select the GCP project that you created the VM instance in.

  3. In the list of instances, click the instance name to display the VM instance details page.
  4. Scroll down to the Custom metadata section.
  5. Write down the value specified for endpoints-service-name. You need to make sure that the configuration for the service was deployed to the same GCP project that the VM instance is in.

Next, check to see if that service name is in the same GCP project that the VM was created in.

Console

  1. In the GCP Console, go to the Endpoints dashboard for your project.

    Endpoints Dashboard

  2. If you have more than one API, click the API from the list.
  3. Click the Deployment history tab.
  4. The service name is displayed between the API name and the tabs, near the top-left side of the page. In the Service configuration deployments list, the configuration ID is displayed along with the date and the email address of the project member who deployed the configuration. The Service configuration deployments list displays the latest 100 configuration deployments. The most recent configuration is displayed at the top of the list.

Command line

  1. Enter the following to display the project IDs for your Cloud projects:
    gcloud projects list
  2. Using the applicable project ID from the previous step, set the default project to the one that your API is in:
    gcloud config set project [YOUR_PROJECT_ID]
  3. Get a list of services in your project:
    gcloud endpoints services list
  4. Using the applicable service name from the previous step, get a list of configuration IDs for the service:
    gcloud endpoints configs list --service=[YOUR_SERVICE_NAME]

For more information on the above commands, see the gcloud reference.

If there is a mismatch, your options are:

  • Create a VM instance in the GCP project that the service configuration was deployed in, and go through the steps to deploy your API and ESP to the new VM instance.
  • Change the name of the service that you have configured in the name field of your service configuration file, and deploy the configuration to the same GCP project that the VM instance is in.

Service names must be unique on GCP. You cannot have the same service name in more than one GCP project. If you are using the cloud.goog domain, which contains the GCP project ID, the easiest solution it to change the name in your service configuration and deploy the configuration in the same GCP project that the VM instance is in.

If you are using a custom domain (for example, myapi.mycompany.com) for your API, it might be easier to create a VM instance in the GCP project that the service configuration was deployed in. Although you can delete the service, Cloud Endpoints will not allow you to deploy a service configuration with the same service name for approximately 30 days. If you must use the same service name, and creating a new VM instance is not an option, you can contact GCP support and request that they purge the service from GCP.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Endpoints with gRPC