Troubleshooting the Agent

This page provides instructions for troubleshooting common issues found with installing or interacting with the Stackdriver Logging agent.

Verifying the agent installation

To check that the installation was successful, look for the agent's test log entry in the Logs Viewer.

Go to the Logs Viewer

  1. At the top of the page, choose the project containing your VM instance:

    • For Compute Engine VM instances, choose the project that contains the VM instance.
    • For Amazon EC2 VM instances, choose the AWS LINK project that Stackdriver creates when you connect your AWS account to a Workspace.
    • Do not choose a Workspace project, unless the project contains your Compute Engine VM instance.
  2. In the windows tabs, choose the resource for your VM instance:

    • For Compute Engine, choose GCE VM Instance.
    • For Amazon EC2, choose AWS EC2 Instance.
    • Select syslog (Linux), fluent.info (Windows), or All logs.

If you see a log entry, "Successfully sent gRPC to Stackdriver Logging API," then the agent installation is complete. This message generated once when the agent is installed and also each the agent is restarted.

For more information about the Logs Viewer, see Viewing Logs.

Testing the agent

If you suspect that the agent is not working, check that it is running and try to send a test message to Stackdriver Logging:

Linux instance

The following procedure works on both Compute Engine and Amazon EC2 VM instances running Linux:

  1. Verify that the Logging agent is running by executing the following commands on your VM instance:

    ps ax | grep fluentd
    

    You should see output similar to the following:

     2284 ?        Sl     0:00 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd [...]
     2287 ?        Sl    42:44 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd [...]
    
  2. Send a test log message by running the following command on your VM instance:

    logger "Some test message"
    

Windows instance

The Logging agent has two Windows service names:

  • StackdriverLogging for versions v1-5 and later
  • fluentdwinsvc for earlier versions

You should be running one agent service. Run the following commands on your VM instance using PowerShell:

  1. Ask for the status of both services. If you know which service should be running, you can use just that service name:

    Get-Service StackdriverLogging,fluentdwinsvc
    
  2. If a service is not running, you see an error message. If it is running, you see output like the following:

    Status    Name                DisplayName
    ------    ----                -----------
    Running  StackdriverLogging   Stackdriver Logging
    
  3. If you query both services, you should see one error message and one Running status:

    • If you do not see any Running status, then the Logging agent is not running.
    • If you see that StackdriverLogging is running, then you are running a recent agent version. To determine the specific version, see Getting the version.
    • If you see that fluentdwinsvc is running, then you should upgrade your agent to the latest version.
  4. Requires Administrator privileges: If any agent version is running, then send a test log message by running the following PowerShell commands:

    New-EventLog   -LogName Application -Source "Test Source"
    Write-EventLog -LogName Application -Source "Test Source" -EntryType Information -EventID 1 -Message "Testing 123 Testing."
    

Finding your test message

After sending a test message, look for it in the Logs Viewer:

Go to the Logs Viewer

  1. At the top of the page, choose the project containing your VM instance:

    • For Compute Engine VM instances, choose the project that contains the VM instance.
    • For Amazon EC2 VM instances, choose the AWS LINK project that Stackdriver creates when you connect your AWS account to a Workspace.
    • Do not choose a Workspace project, unless that is the project containing your Compute Engine VM instance.
  2. In the windows tabs, choose the resource for your VM instance:

    • For Compute Engine, choose GCE VM Instance.
    • For Amazon EC2, choose AWS EC2 Instance.
    • Select syslog (Linux), fluent.info (Windows), or All logs.
  3. You should see a log entry with your test message. If so, then the Logging agent is operating correctly.

Verifying Compute Engine credentials

For a Compute Engine VM instance to run the agent without private-key credentials, the instance must have suitable access scopes and the service account identity being used by the instance must have suitable IAM permissions.

When you create a VM instance, the default scope and service account settings are sufficient to run the agents. Very old instances, or instances for which you have changed the default settings, might not have suitable credentials.

Verifying access scopes

To verify the access scopes, do the following:

  1. Open the Compute Engine > VM Instances page:

    Open the instances page

  2. Click the name of your VM instance. The detail page for your instance appears.

  3. Look under the Cloud API access scopes heading on the page.

    1. If you see "This instance has full API access to all Google Cloud Services," then your access scopes are adequate.
    2. If you see next to Stackdriver Logging API that you have Write Only or Full permission, then your instance's access scopes are adequate for the Stackdriver Logging agent.
    3. If you see next to Stackdriver Monitoring API that you have Write Only or Full permission, then your instance's access scopes are adequate for the Stackdriver Monitoring agent.

Correcting the problem

If you do not have suitable access scopes in your Compute Engine instance, add the needed access scopes to your instance.

The following table shows the scopes relevant to the Stackdriver Logging and Monitoring agents:

Access scope Agent permissions
https://www.googleapis.com/auth/logging.write Adequate for the Logging agent
https://www.googleapis.com/auth/monitoring.write Adequate for the Monitoring agent

Verifying default service account permission

Even if your Compute Engine VM instance's access scopes are adequate, your instance's default service account might not provide the right IAM permissions for the agent.

To verify the default service account permission, start by locating the default service account:

  1. Open the Compute Engine dashboard for your project:

    Open the Compute Engine Instances page

  2. Click the name of your VM instance. The detail page for your instance appears.

  3. Look for the Service account heading on the page. The default service account for the instance is listed. It might look like the following:

    [ID]-compute@developer.gserviceaccount.com
    
  4. Open the IAM & admin > IAM page for your project. The page heading is Permissions for project [PROJECT_NAME]:

    Open the IAM page

  5. Select View By: Members. You should see a list of people, groups, and service accounts. In the Role column are the roles each member has in your project.

  6. In the row for your instance's default service account, you should see one or more roles:

    • If you see Editor, that role is adequate for all the agents. Editor is the default role assigned to service accounts for Compute Engine.
    • If you see Logs Writer, that role is sufficient for the Logging agent. For other Logging roles that include the write permission, see Access Control for Stackdriver Logging.
    • If you see Monitoring Metric Writer, that role is sufficient for the Monitoring Agent. For other Monitoring roles that include the write permission, see Access Control for Stackdriver Monitoring.

Correcting the problem

If your default service account does not have adequate roles, try editing the roles for your service account in the IAM & admin > IAM page. Add the proper Logging or Monitoring roles to authorize the agent(s): Logging > Logs Writer or Monitoring > Monitoring Metric Writer.

Verifying private-key credentials

On Compute Engine VM instances, you can configure the agent to use a non-default service account that has the proper authorization. On AWS EC2 VM Instances, you must configure the agent to use such a service account.

To configure the agent this way, you must create private-key credentials for the designated service account and give those credentials to the agent. The agent looks for the credentials in two ways:

  1. The agent looks for an environment variable, GOOGLE_APPLICATION_CREDENTIALS, that holds the name of a file which contains the private-key credentials.
  2. If the environment variable is not present, then the agent will look for credentials in a standard file:

Linux

 /etc/google/auth/application_default_credentials.json

Windows

C:\ProgramData\Google\Auth\application_default_credentials.json

The following information helps you diagnose private-key credentials problems:

  1. Is the private key in place?
  2. Is the private key still valid for the service account?
  3. Does the service account have the roles needed for the agent?

To verify that valid private-key credentials are installed on your VM instance, first verify that the credentials file exists in its expected location, and then verify that the information in the credentials file is valid. Previously-valid credentials can be revoked using the IAM & Admin > Service accounts section of the GCP Console. If valid credentials aren't present, see Adding credentials to replace the existing credentials or to add new ones.

Are the credentials present?

To see if private-key service account credentials are on your instance, run the following Linux commands on your instance:

sudo cat $GOOGLE_APPLICATION_CREDENTIALS
sudo cat /etc/google/auth/application_default_credentials.json

If either command displays a file like the one shown below, then your instance might have valid private-key credentials. If both commands display a file, then the file denoted by GOOGLE_APPLICATION_CREDENTIALS is used.

{
  "type": "service_account",
  "project_id": "[YOUR-PROJECT-ID]",
  "private_key_id": "[YOUR-PRIVATE-KEY-ID]",
  "private_key": "[YOUR-PRIVATE-KEY]",
  "client_email": "[YOUR-PROJECT-NUMBER]-[YOUR-KEY@DEVELOPER].gserviceaccount.com",
  "client_id": "[YOUR-CLIENT-ID]",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "{x509-cert-url}",
  "client_x509_cert_url": "{client-x509-cert-url}"
}

If there are no credential files present, then see Adding credentials.

Are the credentials valid?

In the credentials file, project_id is your GCP project, client_email identifies the service account in the project, and private_key_id identifies the private key in the service account. Match this information with what is shown in the IAM & Admin > Service accounts section of the GCP Console.

The credentials file is not valid if any of the following are true:

  • You are checking a Compute Engine instance, but the GCP project in the credentials file is not the project that contains your instance.
  • You are checking an Amazon EC2 instance, but the GCP project in the credentials file is not the connector project (named AWS Link...) for your AWS account.
  • The listed service account doesn't exist. It could have been deleted.
  • The listed service account doesn't have the right roles enabled: Logs Writer for the Stackdriver Logging agent and Monitoring Metric Writer for the Stackdriver Monitoring agent.
  • The private key doesn't exist. It could have been revoked.

If the service account is the correct one but the private key has been revoked, then you can create a new private key and copy it to your instance. See Creating service account keys.

Otherwise, you must create a new service account as described in the section Adding credentials.

Other common issues

The following table lists some common problems that you may encounter with the Stackdriver Logging agent and tells you how to fix them.

On Linux, the Logging agent records errors in /var/log/google-fluentd/google-fluentd.log. The error class Google::APIClient::ClientError indicates there is a problem with permissions or API access.

You may start seeing errors after the agent has been running successfully. For example, someone might have revoked the required permissions from your project or your VM instance.

Error Cause Solution
The agent's installer on Windows fails to run You might have downloaded the installer to a system directory. Move the installer to a non-system directory, such as C:\Users\[USERID]\.
Project has not enabled the API You have not enabled the Stackdriver Logging API in your project. Go to the APIs console and change the status of the Stackdriver Logging API to ON.
Request had invalid credentials
or
Unable to fetch access token (no scopes configured?)
Your VM instance does not have suitable credentials. If you are using Amazon EC2, then you must install credentials on your VM instances before installing the agent. See Authorizing the Agent to install credentials.
Authorization failed Your private-key authorization credentials for the Logging agent are not configured correctly. See Verifying private key credentials.
The caller does not have permission The service account used for authorization in your project has insufficient permissions. It might be the default service account used within Compute Engine or App Engine, or it might be a user-defined service account used for private key authorization. The account must have Can edit permission. Change the permission of the service account in your project's Permissions page.
Cannot obtain project ID The Stackdriver Logging agent failed to get the project ID from a service account's private key credentials file. To add or override a project ID for the agent, edit the agent's configuration file, /etc/google-fluentd/google-fluentd.conf, on your VM instance. In the <match **> section, add the following line:
project_id [YOUR_PROJECT_ID]
Otherwise, see Authorizing the Agent to fix or replace the credentials.
Was this page helpful? Let us know how we did:

Send feedback about...

Stackdriver Logging