AvroIO.Write.Bound (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.io

Class AvroIO.Write.Bound<T>

    • Method Detail

      • withSuffix

        public AvroIO.Write.Bound<T> withSuffix(String filenameSuffix)
        Returns a new PTransform that's like this one but that writes to the file(s) with the given filename suffix.

        See ShardNameTemplate for a description of shard templates.

        Does not modify this object.

      • withNumShards

        public AvroIO.Write.Bound<T> withNumShards(int numShards)
        Returns a new PTransform that's like this one but that uses the provided shard count.

        Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

        Does not modify this object.

        Parameters:
        numShards - the number of shards to use, or 0 to let the system decide.
        See Also:
        ShardNameTemplate
      • withoutSharding

        public AvroIO.Write.Bound<T> withoutSharding()
        Returns a new PTransform that's like this one but that forces a single file as output.

        This is a shortcut for .withNumShards(1).withShardNameTemplate("")

        Does not modify this object.

      • withSchema

        public <X> AvroIO.Write.Bound<X> withSchema(Class<X> type)
        Returns a new PTransform that's like this one but that writes to Avro file(s) containing records whose type is the specified Avro-generated class.

        Does not modify this object.

        Type Parameters:
        X - the type of the elements of the input PCollection
      • withSchema

        public AvroIO.Write.Bound<GenericRecord> withSchema(String schema)
        Returns a new PTransform that's like this one but that writes to Avro file(s) containing records of the specified schema in a JSON-encoded string form.

        Does not modify this object.

      • withoutValidation

        public AvroIO.Write.Bound<T> withoutValidation()
        Returns a new PTransform that's like this one but that has GCS output path validation on pipeline creation disabled.

        Does not modify this object.

        This can be useful in the case where the GCS output location does not exist at the pipeline creation time, but is expected to be available at execution time.

      • populateDisplayData

        public void populateDisplayData(DisplayData.Builder builder)
        Description copied from class: PTransform
        Register display data for the given transform or component.

        populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

        By default, does not register any display data. Implementors may override this method to provide their own display data.

        Specified by:
        populateDisplayData in interface HasDisplayData
        Overrides:
        populateDisplayData in class PTransform<PCollection<T>,PDone>
        Parameters:
        builder - The builder to populate with display data.
        See Also:
        HasDisplayData
      • getShardNameTemplate

        public String getShardNameTemplate()
        Returns the current shard name template string.
      • getFilenamePrefix

        public String getFilenamePrefix()
      • getShardTemplate

        public String getShardTemplate()
      • getNumShards

        public int getNumShards()
      • getFilenameSuffix

        public String getFilenameSuffix()
      • getType

        public Class<T> getType()
      • getSchema

        public Schema getSchema()
      • needsValidation

        public boolean needsValidation()


Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow
Need help? Visit our support page.