General best practices when using Datastream

In this page, you'll find best practices for using Datastream. These include general best practices when using Datastream.

Change a stream's source database

In some cases, you may have to change the source database of a stream. For example, you may have to modify the stream to replicate from a replica instead of from the primary database instance.

1. Create a connection profile for the replica instance.
2. Create a stream, using the connection profile for the replica that you created and the existing connection profile for the destination.
3. Start the stream with historical backfill disabled. When the stream is started, it will bring only the data from the binary logs.
4. Optional. After the stream is running, modify it to enable automatic backfill.

5. Pause the stream that's reading from the primary instance.

   You don't have to stop the Dataflow job, change the Pub/Sub notifications, or modify the datasets in BigQuery.

6. Optional. Delete the stream that was streaming data from the primary instance.

7. Optional. Delete the connection profile for the primary instance.

Alert and monitor in Datastream

The Datastream dashboard contains a great deal of information. This information can be helpful for debugging purposes. Additional information can be found in the logs, which are available in Cloud Logging.

Use Google Cloud Monitoring to create a custom dashboard to suit your business needs.

Datastream alerts

There's no default alert set up for Datastream. For example, you can create an alerting policy for the Data freshness metric by clicking the Create alerting policy link in the Overview tab. For the remaining metrics, follow these steps:

1. In the Google Cloud console, go to the  Alerting page:

   Go to Alerting

2. Click Create policy.

3. Click the Select a metric drop-down.

4. In the filter field, enter Datastream.

5. Optional: You might need to disable the Active filter to view all available metrics.

6. Search for the metric that you want to monitor under Datastream Stream.

7. Click Apply.

8. Optional: Enter the required details in the Add filters and Transform data sections. Click Next.

9. Enter the required information in the Configure alert trigger section. Click Next.

10. Configure your notifications in the Configure notifications and finalize alert section.

11. Review your alert and click Create policy when ready.

    For detailed information about how to complete each of these steps, see Create alerting policy.

We recommend creating alerts for the following Datastream metrics:

• Data freshness
• Stream unsupported event count
• Stream total latencies

An alert on any of these metrics can indicate a problem with either the stream or the source database.

How many tables can a single stream handle?

We recommend that a single stream includes up to 10,000 tables. There's no limit to the size of the tables. If you need to create a stream with more tables, then the stream might enter an error state. To avoid this, consider splitting the source into multiple streams.

Keep in mind the impact on the source database. Each stream will have its own limit on the number of connections and simultaneous tasks, so combining multiple streams could overwhelm the database.