Google 提供开源 Dataflow 模板,您可以直接使用,而无需编写流水线代码。本页面列出了可用的模板。
流处理模板
用于连续处理数据的模板:
- Apache Kafka to Apache Kafka
- Apache Kafka to BigQuery
- Apache Kafka to Cloud Storage
- Change Data Capture from MySQL to BigQuery (Stream)
- Bigtable change streams to BigQuery
- Bigtable change streams to Pub/Sub
- Spanner change streams to BigQuery
- Spanner change streams to Cloud Storage
- Spanner change streams to Pub/Sub
- Datastream to BigQuery (Stream)
- Datastream to Spanner
- Datastream to MySQL 或 PostgreSQL (Stream)
- Data Masking/Tokenization from Cloud Storage to BigQuery (using Cloud DLP)
- JMS to Pub/Sub
- MongoDB to BigQuery (Stream)
- MQTT to Pub/Sub
- Pub/Sub 到 BigQuery
- 具有 Python UDF 的 Pub/Sub to BigQuery
- Pub/Sub Avro to BigQuery
- Pub/Sub Proto to BigQuery
- 具有 Python UDF 的 Pub/Sub Proto to BigQuery
- Pub/Sub Subscription to BigQuery
- Pub/Sub to Avro Files on Cloud Storage
- Pub/Sub to Datadog
- Pub/Sub to Elasticsearch
- Pub/Sub to Java Database Connectivity (JDBC)
- Pub/Sub to MongoDB
- Pub/Sub to MongoDB with Python UDF
- Pub/Sub to Pub/Sub
- Pub/Sub to Redis
- Pub/Sub to Splunk
- Pub/Sub Topic or Subscription to Text Files on Cloud Storage
- Pub/Sub Topic to Text Files on Cloud Storage
- Text Files on Cloud Storage to BigQuery (Stream)
- 具有 Python UDF 的 Cloud Storage Text Files to BigQuery
- Text files on Cloud Storage to Pub/Sub (Stream)
批处理模板
用于批量处理数据的模板:
- Any Source DB to Spanner
- Apache Cassandra to Bigtable
- AstraDB to BigQuery
- BigQuery to Bigtable
- BigQuery export to Parquet(通过 Storage API)
- BigQuery to Cloud Storage TFRecords
- BigQuery to Elasticsearch
- BigQuery to MongoDB
- Bigtable to Cloud Storage Avro
- Bigtable to Cloud Storage JSON
- Bigtable to Cloud Storage Parquet
- Bigtable to Cloud Storage SequenceFile
- Bigtable change streams to Vertex AI Vector Search
- Bigtable to Vertex AI Vector Search files on Cloud Storage
- Spanner to BigQuery
- Spanner to Cloud Storage Avro
- Spanner to Cloud Storage Text
- Spanner to Vertex AI Vector Search files on Cloud Storage
- Cloud Storage Avro to Bigtable
- Cloud Storage Parquet to Bigtable
- Cloud Storage SequenceFile to Bigtable
- Cloud Storage Avro to Spanner
- Cloud Storage CSV files to BigQuery
- Cloud Storage Text to BigQuery
- 具有 Python UDF 的 Cloud Storage Text Files to BigQuery
- Cloud Storage Text to Spanner
- Cloud Storage Text to Datastore [已弃用]
- Cloud Storage Text to Firestore (Datastore mode)
- Cloud Storage Text to Pub/Sub (Batch)
- Cloud Storage to Elasticsearch
- Datastore to Cloud Storage Text [已弃用]
- Firestore (Datastore mode) to Cloud Storage Text
- Google Ads to BigQuery
- Google Cloud to Neo4j
- JDBC to BigQuery
- JDBC to Pub/Sub
- MongoDB to BigQuery
- MySQL to BigQuery
- 从 Oracle 迁移到 BigQuery
- PostgreSQL to BigQuery
- SQL Server to BigQuery