You can use Datastream to monitor data and events that are processed by a stream. This information appears in the following graphs:
- Data freshness: The time gap between data residing in the source and the data being transferred into the destination by the stream. This is calculated as the time elapsed since Datastream last checked for new data in the source.
Throughput: The rate at which Datastream processes data or events. This rate can be:
- The amount of data (in MB) that Datastream transfers from the source to the destination.
The number of events that are associated with the data being transferred. An event is a single change in the source, such as a new row added to a table in a database.
Unsupported events: The number of events that can't be processed from the source to the destination.
System latency: How long it takes Datastream to process an event. This interval is calculated as the time between when Datastream reads the event and when the event is written to the destination.
Total latency: The time difference between when data is written to the source and the corresponding events are written to the destination.
Monitor data or events procssed
Go to the Streams page in the Google Cloud Console.
Click the stream that you want to monitor.
Click the MONITORING tab.
Scroll until you see the desired graph that represents the data or events that Datastream is monitoring.
Optionally, if you're looking at the Throughput graph, then select (bytes/sec) to see the amount of data that Datastream transfers from the source to the destination, or (event/sec) to see how many events are associated with the data being transferred.
To view how much data or how many events Datastream processed over a period of hours or days, click 1 hour, 6 hours, 12 hours, 1 day, 2 days, 4 days, 7 days, 14 days, or 30 days.
Or, to set a custom range, select the Custom menu, and then use the Calendar picker to specify a start date and time and end date and time for the amount of data or the number of events that you want to view.
Datastream updates the graph in the pane to reflect your selection and displays the interval that you specified.
For example, if today is October 31, 2020, and you click 30 days, the graph will show the amount of data or the number of events Datastream processed since October 01, 2020.
Hover your mouse over the line in the graph to display how much data or how many events Datastream processed for a particular date and time.