Debug Logpoints

After you have deployed or started your application, you can open Cloud Debugger in the Google Cloud Console. Cloud Debugger allows you to inject logging into running services without restarting or interfering with the normal function of the service. This can be useful for debugging production issues without having to add log statements and redeploy.

Visit the Debug page of the Cloud Console to use Google Cloud Debugger.

Before you begin

Cloud Debugger can be used with or without access to your application's source code. If your source code isn't available, see Adding a debug logpoint below for instructions on entering the filename and line number manually.

If your source code is stored in Google Cloud Repositories, it's automatically displayed in the Debug view.

To access source code that's stored elsewhere, such as locally or in a git repository, you may need to select a source code location.

Logpoints

Logpoints allow you to inject logging into running services without restarting or interfering with the normal function of the service. Every time any instance executes code at the logpoint location, Cloud Debugger logs a message. Output is sent to the appropriate log for the target's environment. On App Engine, for example, output is sent to the request log in Cloud Logging.

Logpoints remain active for 24 hours after creation, or until they are deleted or the service is redeployed.

Tip: Google Cloud Shell provides you with command-line access to your cloud resources from your browser. You can run logpoints commands from Cloud Shell without having to install the Cloud SDK.

Note: Logpoints may not available during startup, while debugger initializes in the background.

Debugger agent with canarying

After you set a logpoint, the Debugger agent tests the logpoint on a subset of your instances. After the Debugger agent verifies the logpoint can execute successfully, the logpoint is applied to all your instances. This takes about 40 seconds.

After this process, when the logpoint is triggered, the Debugger agent logs the message. If the logpoint is triggered within 40 seconds after setting it, the Debugger agent logs the message from those instances that had the canary logpoint applied to it.

The Debugger agent implements several strategies for minimizing the amount of latency caused when capturing the data.

You see the following while the Debugger agent is canarying:

To learn what to do if the Debugger agent fails while in canary mode, go to the Troubleshooting section on the Debug snapshots page.

To learn what Debugger agent versions have canary functionality, see the language-specific pages.

Debugger agent without canarying

When you use the Debugger agent without canarying and set a logpoint, it applies to all running instances of your app. The first time any instance executes the code at the logpoint location, the Debugger agent logs the message. The Debugger agent implements several strategies for minimizing the amount of latency caused when capturing the data.

Note: A logpoint is only taken once during app runtime. You can manually retake the logpoint in order to capture a new set of data.

Note: logpoints might not be available during startup, while the Debugger agent initializes in the background.

Adding a debug logpoint

Console

To add a logpoint from the Cloud Console:

1. Make sure that the Logpoint tab is selected in the right panel.
2. In the left panel, select the file that contains the source code where you want to add a logpoint. The contents of the file will be displayed in the center panel.
3. Click the line number at the location to add a logpoint.
4. Fill out the message between the logpoint("") field and click the add button. You can put an expression between cury braces, such as {newScore.score} to log its value.

If no source code is available, you can manually enter the filename:line and other details in the Logpoint panel:

gcloud

To add a logpoint from the command line:

gcloud debug logpoints create LOCATION MESSAGE

Where:

• LOCATION is the filename and line at which to set the logpoint. Format is FILE:LINE, where FILE can be the filename or the filename preceded by enough path components to differentiate it from other files with the same name. It's an error to provide a filename that is not unique in the debug target.
• MESSAGE is the message you want to log.

The following example logs the value of score at the info log level (which is the default log level for logpoints):

gcloud debug logpoints create HighScoreService.java:105 \
  "User {name} scored {newScore.score}"

When line 105 of HighScoreService.java executes, the message gets logged with the values of variables name and newScore.score inserted into the output string.

Logpoint message format

The message of a logpoint determines what gets logged in the output. Expressions allow you to evaluate and log values of interest. Anything in the message between curly braces such as {myObj.myFunc()} or {a + b} will be replaced with the value of that expression in the output. The message User {name} scored {newScore.score} in the example above will log output similar to User user1 scored 99.

You can use the following language features for expressions:

Java

Most Java expressions are supported, including:

• Local variables: a == 8.
• Numerical and boolean operations: x + y < 20.
• Instance and static fields: this.counter == 20, this.myObj.isShutdown, myStatic, or com.mycompany.MyClass.staticMember.
• String comparisons with the equality operator: myString == "abc".
• Function calls. Only read-only functions can be used. For example, StringBuilder.indexOf() is supported, but StringBuilder.append() is not.
• Type casting, with fully qualified types: ((com.myprod.ClassImpl) myInterface).internalField

The following language features are not supported:

• Unboxing of numeric types, such as Integer; use myInteger.value instead.

Python

Most Python expressions are supported, including:

• Reading local and global variables.
• Reading from arrays, lists, slices, dictionaries and objects.
• Calling simple methods.

The following language features are not supported:

• Calling functions that allocate new objects or use complex constructs.
• Creating new objects inside the expression.

Go

Most Go expression syntax is supported, including:

• Reading local and global variables.
• Reading from arrays, slices, maps, and structs.

The following language features are not supported:

• Reading from interface values.
• Type conversions and composite literals.
• Function calls other than len.

Logpoint condition

A logpoint condition is a simple expression in the application language that must evaluate to true for the logpoint to be logged. Logpoint conditions are evaluated each time the line is executed, by any instance, until the logpoint expires or is deleted.

The condition is a full boolean expression that can include logical operators:

travelAccessory == “Towel”
ultimateAnswer <= 42
travelAccessory == “Towel” && ultimateAnswer <= 42

The same language features supported for expressions are available for conditions.

Console

Enter the condition inside the 'if' statement:

If no source code is available, you can specify the condition in the Logpoint panel.

gcloud

Conditions are expressed using the --condition flag of logpoints create:

gcloud debug logpoints create HighScoreService.java:105 

  --condition="newScore.score > 40" 

  --log-level="warning" 

  "Suspiciously high score ({newScore.score}) from user {name}"

In the example above, the logpoint checks the value of newScore.score when line 105 of the application executes. If the value is greater than 40, a warning message is added to the log.

Viewing output

Logpoint output is sent to the appropriate log for the target's environment.

App Engine

Logpoints set on App Engine apps send their output to the request log in Cloud Logging.

You can view the logs in the Logs Panel or in the dedicated Logs Explorer.

Compute Engine

Logpoints set on Compute Engine apps send their output to the same location as regular log statements. For example, in Python, the default logging module sends its output to stdout, but can be configured to write to a specific file.

You can set up the logging agent to forward these logs to Cloud Logging. From there, you can view the logs in the Logs Explorer.

Deleting logpoints

Logpoints become inactive and stop logging messages after 24 hours, and are automatically deleted after 30 days. You can manually delete logpoints, which will both stop the logging and remove it from the history for future reference. Note that, however, deleting a logpoint will not delete the log messages already generated.

Console

To delete a logpoint manually, use the overflow menu in the Logpoint History panel.

gcloud

To delete a logpoint manually, you can specify the logpoint by its ID, or using regular expressions on the code location:

gcloud debug logpoints delete HighScoreService.java:105

Debug target not specified. Using default target: default-1
This command will delete the following logpoints:

LOCATION                   CONDITION            LOG_LEVEL  LOG_MESSAGE_FORMAT                                             ID
HighScoreService.java:105                       INFO       User {name} scored {newScore.score}.                           53aaa3bd8d2d7-b76f-feb5d
HighScoreService.java:105  newScore.score > 40  WARNING    Suspiciously high score ({newScore.score}) from user {name}  53aaa3bsdffd7-b56f-fasfg

Do you want to continue (Y/n)?  Y
Deleted 2 logpoints.