General best practices when using Datastream

In this page, you'll find best practices for using Datastream. These include general best practices when using Datastream.

Change a stream's source database

In some cases, you may have to change the source database of a stream. For example, you may have to modify the stream to replicate from a replica instead of from the primary database instance.

1. Create a connection profile for the replica instance.
2. Create a stream, using the connection profile for the replica that you created and the existing connection profile for the destination.
3. Start the stream with historical backfill disabled. When the stream is started, it will bring only the data from the binary logs.
4. Optional. After the stream is running, modify it to enable automatic backfill.

5. Pause the stream that's reading from the primary instance.

   You don't have to stop the Dataflow job, change the Pub/Sub notifications, or modify the datasets in BigQuery.

6. Optional. Delete the stream that was streaming data from the primary instance.

7. Optional. Delete the connection profile for the primary instance.

Alert and monitor in Datastream

The Datastream dashboard contains a great deal of information. This information can be helpful for debugging purposes. Additional information can be found in the logs, which are available in Cloud Logging.

Use Google Cloud Monitoring to create a custom dashboard to suit your business needs.

Datastream alerts

There's no default alert set up for Datastream, but alerts can be created easily by clicking the "Create alerting policy" link for each metric in the Datastream UI. We recommend creating alerts for the following Datastream metrics:

• Data freshness
• Unsupported events
• Total latency

An alert on any of these metrics can indicate a problem with either the stream or the source database.

How many tables can a single stream handle?

A single stream can handle up to 10,000 tables, and there's no limit to the size of the tables. However, there may be some other business-logic considerations where you should use multiple streams for your source database. . Examples of these considerations include having a better control of user access to data, using multiple streams to make maintenance easier for different business flows, and so on.

Keep in mind the impact on the source database. Each stream will have its own limit on the number of connections and simultaneous tasks, so combining multiple streams could overwhelm the database.