Monitor a stream

Overview

You can use Datastream to monitor data and events that are processed by a stream. This information appears in the following graphs:

  • Throughput: The rate at which Datastream processes data or events. This rate can be:

    • The amount of data (in MB) that Datastream transfers from the source to the destination.

    • The number of events that are associated with the data being transferred. An event is a single change in the source, such as a new row added to a table in a database.

  • Unsupported events: The number of events that can't be processed from the source to the destination.

  • Data freshness: The time difference between the data residing in the source and the data being transferred into the destination by the stream. It is calculated as the time elapsed between the source timestamp and the read timestamp for the oldest event being processed. If there are no new events to read from the source, the freshness is set to 0.

    If there are enqueued events that Datastream hasn't processed yet, they aren't taken into account when Datastream calculates the data freshness metric. For example, if there is a spike in throughput, it is reflected in the freshness metric only after Datastream processes the events causing the spike.

  • System latency: The time that Datastream takes to process an event. This interval is calculated as the time between when Datastream reads the event and when the event is written to the destination.

  • Total latency: The time difference between when data is written to the source and the corresponding events are written to the destination.

Monitor data or events processed

  1. Go to the Streams page in the Google Cloud Console.

    Go to the Streams page

  2. Click the stream that you want to monitor.

  3. Optional. Scroll until the Data freshness graph appears. For this graph, click the Create alerting policy link to create an alerting policy for it. An alerting policy describes a set of conditions that you want to monitor for the graph.

    After clicking the link, the Create alerting policy page appears in Cloud Monitoring. On this page, you can define the alerting policy for the graph. This includes specifying which criteria will trigger the policy, who'll be notified if the criteria are met, and how they'll be notified.

    For more information about alerting policies, including how to create them, see Managing metric-based alerting policies.

  4. Click the MONITORING tab.

  5. Scroll until the desired graph appears that represents the data or events that Datastream is monitoring.

  6. Optionally, if you're looking at the Throughput graph, then select (bytes/sec) to see the amount of data that Datastream transfers from the source to the destination, or (event/sec) to see how many events are associated with the data being transferred.

  7. To view how much data or how many events Datastream processed over a period of hours or days, click 1 hour, 6 hours, 12 hours, 1 day, 2 days, 4 days, 7 days, 14 days, or 30 days.

    Or, to set a custom range, select the Custom menu, and then use the Calendar picker to specify a start date and time and end date and time for the amount of data or the number of events that you want to view.

    Datastream updates the graph in the pane to reflect your selection and displays the interval that you specified.

    For example, if today is October 31, 2021, and you click 30 days, the graph will show the amount of data or the number of events Datastream processed since October 01, 2021.

  1. Hold the pointer over the line in the graph to display how much data or how many events Datastream processed for a particular date and time.

What's next