File Teks di Cloud Storage ke template Pub/Sub (Stream)

Template ini membuat pipeline streaming yang terus-menerus melakukan polling untuk file teks baru yang diupload ke Cloud Storage, membaca setiap file baris demi baris, dan memublikasikan string ke topik Pub/Sub. Template ini memublikasikan data dalam file baru yang dibatasi baris baru yang berisi data JSON atau file CSV ke topik Pub/Sub untuk pemrosesan real-time. Anda dapat menggunakan template ini untuk memutar ulang data ke Pub/Sub.

Pipeline berjalan tanpa batas waktu dan harus dihentikan secara manual melalui 'cancel', bukan 'drain', karena penggunaan transformasi 'Watch', yang merupakan 'SplittableDoFn' yang tidak mendukung pengosongan.

Saat ini, interval polling tetap dan disetel ke 10 detik. Template ini tidak menetapkan stempel waktu apa pun pada masing-masing record, sehingga waktu peristiwa sama dengan waktu publikasi selama eksekusi. Jika pipeline Anda bergantung pada waktu peristiwa yang akurat untuk pemrosesan, Anda sebaiknya tidak menggunakan pipeline ini.

Persyaratan pipeline

File input harus dalam format JSON atau CSV yang dipisahkan baris baru. Kumpulan data yang mencakup beberapa baris dalam file sumber dapat menyebabkan masalah downstream, karena setiap baris dalam file dipublikasikan sebagai pesan ke Pub/Sub.
Topik Pub/Sub harus ada sebelum dieksekusi.
Pipeline berjalan tanpa batas waktu dan harus dihentikan secara manual.

Parameter template

Parameter	Deskripsi
`inputFilePattern`	Pola file input yang akan dibaca. Misalnya, `gs://bucket-name/files/.json` atau `gs://bucket-name/path/.csv`.
`outputTopic`	Topik input Pub/Sub yang akan ditulis. Nama harus dalam format `projects/<project-id>/topics/<topic-name>`.

Menjalankan template

Konsol

Buka halaman Create job from template Dataflow.

Buka Buat tugas dari template

Di kolom Job name, masukkan nama pekerjaan yang unik.
Opsional: Untuk Endpoint regional, pilih nilai dari menu drop-down. Region default-nya adalah us-central1.
Untuk daftar region tempat Anda dapat menjalankan tugas Dataflow, lihat Lokasi Dataflow.
Dari menu drop-down Dataflow template, pilih the Text Files on Cloud Storage to Pub/Sub (Stream) template.
Di kolom parameter yang disediakan, masukkan parameter value Anda.
Opsional: Untuk beralih dari pemrosesan tepat satu kali ke mode streaming minimal satu kali, pilih Minimal Sekali.
Klik Run job.

gcloud

Di shell atau terminal Anda, jalankan template:

gcloud dataflow jobs run JOB_NAME \
    --gcs-location gs://dataflow-templates-REGION_NAME/VERSION/Stream_GCS_Text_to_Cloud_PubSub \
    --region REGION_NAME\
    --staging-location STAGING_LOCATION\
    --parameters \
inputFilePattern=gs://BUCKET_NAME/FILE_PATTERN,\
outputTopic=projects/PROJECT_ID/topics/TOPIC_NAME

Ganti kode berikut:

JOB_NAME: nama pekerjaan unik pilihan Anda
REGION_NAME: region tempat Anda ingin men-deploy tugas Dataflow, misalnya us-central1
STAGING_LOCATION: lokasi untuk staging file lokal (misalnya, gs://your-bucket/staging)
TOPIC_NAME: nama topik Pub/Sub Anda
BUCKET_NAME: nama bucket Cloud Storage Anda
FILE_PATTERN: glob pola file untuk dibaca dalam bucket Cloud Storage (misalnya, path/*.csv)

API

Untuk menjalankan template menggunakan REST API, kirim permintaan HTTP POST. Untuk informasi selengkapnya tentang API dan cakupan otorisasinya, lihat projects.templates.launch.

POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/templates:launch?gcsPath=gs://dataflow-templates-LOCATION/VERSION/Stream_GCS_Text_to_Cloud_PubSub
{
   "jobName": "JOB_NAME",
   "environment": {
       "ipConfiguration": "WORKER_IP_UNSPECIFIED",
       "additionalExperiments": []
    },
   "parameters": {
       "inputFilePattern": "gs://BUCKET_NAME/FILE_PATTERN",
       "outputTopic": "projects/PROJECT_ID/topics/TOPIC_NAME"
   }
}

Ganti kode berikut:

PROJECT_ID: ID project Google Cloud tempat Anda ingin menjalankan tugas Dataflow
JOB_NAME: nama pekerjaan unik pilihan Anda
LOCATION: region tempat Anda ingin men-deploy tugas Dataflow, misalnya us-central1
STAGING_LOCATION: lokasi untuk staging file lokal (misalnya, gs://your-bucket/staging)
TOPIC_NAME: nama topik Pub/Sub Anda
BUCKET_NAME: nama bucket Cloud Storage Anda
FILE_PATTERN: glob pola file untuk dibaca dalam bucket Cloud Storage (misalnya, path/*.csv)

Kode sumber template

Java

/*
 * Copyright (C) 2018 Google LLC
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License. You may obtain a copy of
 * the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */
package com.google.cloud.teleport.templates;

import com.google.cloud.teleport.metadata.Template;
import com.google.cloud.teleport.metadata.TemplateCategory;
import com.google.cloud.teleport.templates.TextToPubsub.Options;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.Watch;
import org.joda.time.Duration;

/**
 * The {@code TextToPubsubStream} is a streaming version of {@code TextToPubsub} pipeline that
 * publishes records to Cloud Pub/Sub from a set of files. The pipeline continuously polls for new
 * files, reads them row-by-row and publishes each record as a string message. The polling interval
 * is fixed and equals to 10 seconds. At the moment, publishing messages with attributes is
 * unsupported.
 *
 * <p>Check out <a
 * href="https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/README_Stream_GCS_Text_to_Cloud_PubSub.md">README</a>
 * for instructions on how to use or modify this template.
 */
@Template(
    name = "Stream_GCS_Text_to_Cloud_PubSub",
    category = TemplateCategory.STREAMING,
    displayName = "Text Files on Cloud Storage to Pub/Sub",
    description = {
      "This template creates a streaming pipeline that continuously polls for new text files uploaded to Cloud Storage, reads each file line by line, and publishes strings to a Pub/Sub topic. "
          + "The template publishes records in a newline-delimited file containing JSON records or CSV file to a Pub/Sub topic for real-time processing. "
          + "You can use this template to replay data to Pub/Sub.\n",
      "The pipeline runs indefinitely and needs to be terminated manually via a <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#cancel\">cancel</a> and not a <a href=\"https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain\">drain</a>, due to its use of the <code>Watch</code> transform, which is a <code>SplittableDoFn</code> that does not support draining.\n",
      "Currently, the polling interval is fixed and set to 10 seconds. This template does not set any timestamp on the individual records, so the event time is equal to the publishing time during execution. "
          + "If your pipeline relies on an accurate event time for processing, you should not use this pipeline."
    },
    optionsClass = Options.class,
    documentation =
        "https://cloud.google.com/dataflow/docs/guides/templates/provided/text-to-pubsub-stream",
    contactInformation = "https://cloud.google.com/support",
    requirements = {
      "Input files must be in newline-delimited JSON or CSV format. Records that span multiple lines in the source files can cause issues downstream, because each line within the files is published as a message to Pub/Sub.",
      "The Pub/Sub topic must exist prior to execution.",
      "The pipeline runs indefinitely and needs to be terminated manually.",
    },
    streaming = true,
    supportsAtLeastOnce = true,
    supportsExactlyOnce = true)
public class TextToPubsubStream {
  private static final Duration DEFAULT_POLL_INTERVAL = Duration.standardSeconds(10);

  /**
   * Main entry-point for the pipeline. Reads in the command-line arguments, parses them, and
   * executes the pipeline.
   *
   * @param args Arguments passed in from the command-line.
   */
  public static void main(String[] args) {

    // Parse the user options passed from the command-line
    Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);

    run(options);
  }

  /**
   * Executes the pipeline with the provided execution parameters.
   *
   * @param options The execution parameters.
   */
  public static PipelineResult run(Options options) {
    // Create the pipeline.
    Pipeline pipeline = Pipeline.create(options);

    /*
     * Steps:
     *  1) Read from the text source.
     *  2) Write each text record to Pub/Sub
     */
    pipeline
        .apply(
            "Read Text Data",
            TextIO.read()
                .from(options.getInputFilePattern())
                .watchForNewFiles(DEFAULT_POLL_INTERVAL, Watch.Growth.never()))
        .apply("Write to PubSub", PubsubIO.writeStrings().to(options.getOutputTopic()));

    return pipeline.run();
  }
}

Langkah selanjutnya

Pelajari Template Dataflow.
Lihat daftar template yang disediakan Google.