Streaming and live tailing log entries

Overview

This page describes how to view your log entries in real time.

Cloud Logging lets you view your log entries in real time as Logging writes them by using the following features:

Stream logs. Streaming logs is available in the Logs Explorer.

Live-tailing. Live-tailing is available as the gcloud command-line tool gcloud alpha logging tail and as the Cloud Logging API method entries.tail.

When you view and analyze your logs by using the Logs Explorer, gcloud logging read, or the API method entries.list, you are viewing log entries that Cloud Logging has already stored. In contrast, when you stream or live-tail log entries, you are viewing log entries as soon as your applications write them to the Cloud Logging API.

Streaming logs in the Logs Explorer

In the Logs Explorer, you can view your logs data in real time using Stream logs. When you use Stream logs, you can add a query to the Logs Explorer to stream only those logs that match the query.

To stream logs based on a query, add a query to the Query builder pane, and then select Stream logs. As Logging writes the logs data, only those logs that match the query are shown in the Query result pane. If a query isn't provided, Logging shows each log as it's written.

Stream logs that match a query expression

To stop streaming, click Stop streaming, or scroll down in the Query results pane.

Live tailing using the gcloud command-line tool

Live tailing lets you view your log entries in real time as Cloud Logging writes them, by using either the gcloud command-line tool or the Cloud Logging API.

For information on the API method for live tailing, see the entries.tail method.

Installing gcloud alpha logging tail

To use gcloud alpha logging tail, you need to have Python 3 and the grpcio Python package installed.

For instructions on how to install Python, see the Python page. For instructions on how to install the Python package manager, pip, that is needed to install the grpcio package, see The Python Package Installer page.

Complete the following steps to install gcloud alpha logging tail:

  1. Verify that you have the gcloud command-line tool installed. For instructions on how to install the gcloud command-line tool, see Installing Cloud SDK.

  2. Verify that you're using version 302.0.0 or greater of the gcloud tool.

    gcloud version

    For instructions on updating the gcloud tool, see gcloud components update.

  3. Install the gcloud tool alpha components:

    gcloud components install alpha

  4. For MacOS, Linux, and Cloud Shell users:

    1. Install gRPC client libraries:

       sudo pip3 install grpcio

    2. Set the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to any value:

       export CLOUDSDK_PYTHON_SITEPACKAGES=1

  5. Use the following commands to set your Cloud project ID and to authenticate:

    gcloud config set project PROJECT_ID
gcloud auth login

  6. Verify that gcloud alpha logging tail is installed by running the following command:

    gcloud alpha logging tail

    You are now viewing the log entries for your project as Logging writes them.

    Log entries during a live-tail session.

For more information on using live tailing, see the gcloud alpha logging tail reference guide.

Buffering and ordering

Because Logging can receive log entries out of chronological order, live tailing provides a buffer-window setting so you can balance the tradeoff between viewing the log entries as they are being written and viewing them in ascending order. You can set the buffer window between 0 and 60 seconds.

Note the following characteristics of the buffer window:

  • The default buffer window is two seconds.

  • Logging delays ingesting the log entries for the duration of the buffer window.

  • If a log entry is written outside of the buffer window, then Logging returns the log entries as they are received.

When configuring the buffer window, you make a tradeoff between viewing logs as they are ingested and viewing the entries out of order.

Buffer window Tradeoff
0 seconds Newest log entries returned, but with more likelihood of them being out of order.
60 seconds A delay of 60 seconds before seeing the entries returned, but most of the logs are returned in ascending order.

Limits and quotas

The following table lists the limits and quotas for live tailing:

Limits and quotas Value
Entries returned per minute 60,000
When more the 60,000 entries match a filter, Logging suppresses them and returns the count of suppressed entries in the response.
Open live-tailing sessions per Cloud project 10

Client limitations

For a Cloud project that writes lots of entries quickly, your client might be unable to consume them as quickly as they're being written. In this case, Logging limits the total number of entries sent, prioritizing the most recent entries. At the end of the tail session, Logging returns the number of entries that were not displayed due to the limits of the client.

Live tailing using client libraries

Live tailing lets you view your log entries in real time as Cloud Logging writes them. For information on the API method for live tailing, see the entries.tail method.

This sample demonstrates live tailing log entries of a given logger.

Node.js

To learn how to install and use the client library for Logging, see the Logging Client Libraries. 

const {Logging} = require('@google-cloud/logging');
const logging = new Logging();

/**
 * TODO(developer): Replace logName with the name of your log.
 */
const log = logging.log(logName);
console.log('running tail log entries test');

const stream = log
  .tailEntries({
    filter: 'timestamp > "2021-01-01T23:00:00Z"',
  })
  .on('error', console.error)
  .on('data', resp => {
    console.log(resp.entries);
    console.log(resp.suppressionInfo);
    // If you anticipate many results, you can end a stream early to prevent
    // unnecessary processing and API requests.
    stream.end();
  })
  .on('end', () => {
    console.log('log entry stream has ended');
  });

// Note: to get all project logs, invoke logging.tailEntries