[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-09-04。"],[[["\u003cp\u003eEnabling the Datastream API is necessary for writing to a Cloud Storage bucket, and if the bucket is in a different project, the Datastream service account needs permissions to access it.\u003c/p\u003e\n"],["\u003cp\u003eYou can define the destination for the streamed data by specifying the project name, bucket name, and an optional file prefix in the stream configuration.\u003c/p\u003e\n"],["\u003cp\u003eData is written to the bucket at a specific path format based on bucket, prefix, object name, and source event timestamps, with file rotation occurring when maximum file size, timeout, or source schema change is met.\u003c/p\u003e\n"],["\u003cp\u003eDuplicate events may occur during the initial stream setup when backfill and CDC overlap, but these can be filtered out using the event metadata.\u003c/p\u003e\n"],["\u003cp\u003eThe maximum event size for streaming data into Cloud Storage is 100 MB, and you can adjust the maximum file size and maximum timeout values using the Datastream API.\u003c/p\u003e\n"]]],[],null,["# Configure a Cloud Storage destination\n\nThis page describes how to configure Datastream to write data to a\nCloud Storage bucket.\n\nCloud Storage is a service for storing data in Google Cloud. You store the\ndata as objects, in containers called *buckets* . When you stream to\nCloud Storage, Datastream organizes data by object name and\ntimestamp. You can then consume your data using the Cloud Storage API.\nFor more information, see [About Cloud Storage buckets](/storage/docs/buckets).\n\nEnable the API\n--------------\n\nEnabling the Datastream API is sufficient when the destination Cloud Storage bucket is in the same project as Datastream.\n\nIf you're writing to a bucket in a different project, then you must give the [service account](https://cloud.google.com/docs/authentication#service_accounts) associated with the Datastream permission to access the bucket.\n\n1. Obtain the email address used for the service account. To do so, find the **Project number** in the Google Cloud console home page. The email address of the service account will be `service-[project_number]@gcp-sa-datastream.iam.gserviceaccount.com`.\n2. Give this email address the required permissions to write to the destination bucket (typically the `roles/storage.objectViewer`, `roles/storage.objectCreator` and `roles/storage.legacyBucketReader` permissions).\n\nSpecify the destination bucket and prefix\n-----------------------------------------\n\nProvide the project name, bucket name, and optional file prefix in the stream configuration to determine where the data is written.\n\nWrite behavior\n--------------\n\n- The data for a given stream is written to the provided bucket or file prefix at: `[bucket]/[prefix]/[object name]/yyyy/mm/dd/hh/mm/[filename(idempotent)]`.\n- The object name in the path for database sources is the schema name followed by the table name (separated by an underscore '_').\n- The timestamp in the path (hh/mm) is the source timestamp from the event metadata.\n - For backfill events, the timestamp represents the time when the backfill task was started (typically when the stream was started, or when backfill was manually triggered).\n - For CDC events, the timestamp represents when the record was changed on the source.\n- Files are rotated whenever either the maximum file size or maximum timeout is reached, whichever comes first. You can adjust the maximum file size and maximum timeout values using the [Datastream API](/datastream/docs/reference/rest/v1/projects.locations.streams#gcsdestinationconfig).\n- Additionally, files are rotated each time a source schema change is detected (For example, a new column is added.).\n- There can be multiple files within the same path for the same minute.\n- Consuming the data can be done according to the [Cloud Storage API](/storage/docs/json_api/v1).\n- The maximum event size when you stream data into Cloud Storage is 100 MB.\n\nIn some cases, most commonly when a stream first starts running, there is an overlap between backfill and CDC, resulting in duplicate events. This happens when Datastream starts reading from the current database log file (which was created before the stream was started), and this log file includes some events that are also captured by the backfill. This is anticipated behavior, and duplicate events can be eliminated using the [event metadata](/datastream/docs/events-and-streams#events), if necessary.\n\nWhat's next\n-----------\n\n- Learn how to [stream changes to data in near real-time with Datastream](/datastream/docs/quickstart-change-streaming-to-gcs)."]]
