Google은 파이프라인 코드를 작성하는 대신 사용할 수 있는 오픈소스 Dataflow 템플릿을 제공합니다. 이 페이지에는 사용 가능한 템플릿이 나열되어 있습니다.
스트리밍 템플릿
지속적인 데이터 처리를 위한 템플릿:
- Apache Kafka to BigQuery
- Change Data Capture from MySQL to BigQuery(Stream)
- Bigtable change streams to BigQuery
- Bigtable 변경 내역을 Pub/Sub로 전송
- Spanner change streams to BigQuery
- Spanner change streams to Cloud Storage
- Spanner change streams to Pub/Sub
- Datastream to BigQuery(스트림)
- Datastream to Spanner
- Datastream to MySQL 또는 PostgreSQL(스트리밍)
- Data Masking/Tokenization from Cloud Storage to BigQuery(Cloud DLP 사용)
- JMS to Pub/Sub
- MongoDB to BigQuery(CDC)
- MQTT to Pub/Sub
- Pub/Sub to BigQuery
- Pub/Sub Avro to BigQuery
- Pub/Sub Proto to BigQuery
- Pub/Sub Subscription to BigQuery
- Pub/Sub to Avro Files on Cloud Storage
- Pub/Sub to Datadog
- Pub/Sub to Elasticsearch
- Pub/Sub to Java Database Connectivity(JDBC)
- Pub/Sub to MongoDB
- Pub/Sub to Pub/Sub
- Pub/Sub to Redis
- Pub/Sub to Splunk
- Pub/Sub Topic 또는 Subscription to Text Files on Cloud Storage
- Pub/Sub Topic to Text Files on Cloud Storage
- Text Files on Cloud Storage to BigQuery(스트리밍)
- Text Files on Cloud Storage to Pub/Sub(스트리밍)
일괄 처리 템플릿
데이터 일괄 처리에 사용되는 템플릿:
- Apache Cassandra to Bigtable
- AstraDB to BigQuery
- BigQuery to Bigtable
- BigQuery export to Parquet (via Storage API)
- BigQuery to Cloud Storage TFRecords
- BigQuery to Elasticsearch
- BigQuery to MongoDB
- Bigtable to Cloud Storage Avro
- Bigtable to Cloud Storage JSON
- Bigtable to Cloud Storage Parquet
- Bigtable to Cloud Storage SequenceFile
- Bigtable to Vertex AI Vector Search files on Cloud Storage
- Spanner to BigQuery
- Spanner to Cloud Storage Avro
- Spanner to Cloud Storage Text
- Spanner to Vertex AI Vector Search files on Cloud Storage
- Cloud Storage Avro to Bigtable
- Cloud Storage Parquet to Bigtable
- Cloud Storage SequenceFile to Bigtable
- Cloud Storage Avro to Spanner
- Cloud Storage CSV files to BigQuery
- Cloud Storage Text to BigQuery
- Cloud Storage Text to Spanner
- Cloud Storage Text to Datastore[지원 중단됨]
- Cloud Storage Text to Firestore (Datastore 모드)
- Cloud Storage Text to Pub/Sub (일괄)
- Cloud Storage to Elasticsearch
- Datastore to Cloud Storage Text[지원 중단됨]
- Firestore (Datastore 모드) to Cloud Storage Text
- Google Ads to BigQuery
- Google Cloud to Neo4j
- JDBC to BigQuery
- JDBC to Pub/Sub
- MongoDB to BigQuery
- MySQL to BigQuery
- Oracle to BigQuery
- PostgreSQL to BigQuery
- SQL Server to BigQuery