FileBasedSink (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

java.lang.Object
- com.google.cloud.dataflow.sdk.io.Sink<T>
- - com.google.cloud.dataflow.sdk.io.FileBasedSink<T>

Type Parameters:

T - the type of values written to the sink.

All Implemented Interfaces:

HasDisplayData, Serializable

Direct Known Subclasses:

XmlSink.Bound
```
public abstract class FileBasedSink<T>
extends Sink<T>
```
Abstract Sink for file-based output. An implementation of FileBasedSink writes file-based output and defines the format of output files (how values are written, headers/footers, MIME type, etc.).
At pipeline construction time, the methods of FileBasedSink are called to validate the sink and to create a Sink.WriteOperation that manages the process of writing to the sink.
The process of writing to file-based sink is as follows:
1. An optional subclass-defined initialization,
2. a parallel write of bundles to temporary files, and finally,
3. these temporary files are renamed with final output filenames.
Supported file systems are those registered with IOChannelUtils.
See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`FileBasedSink.FileBasedWriteOperation<T>` Abstract `Sink.WriteOperation` that manages the process of writing to a `FileBasedSink`.
`static class`	`FileBasedSink.FileBasedWriter<T>` Abstract `Sink.Writer` that writes a bundle to a `FileBasedSink`.
`static class`	`FileBasedSink.FileResult` Result of a single bundle write.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Sink
Sink.WriteOperation<T,WriteT>, Sink.Writer<T,WriteT>

Field Summary

Fields
Modifier and Type	Field and Description
`protected ValueProvider<String>`	`baseOutputFilename` Base filename for final output files.
`protected String`	`extension` The extension to be used for the final output files.
`protected String`	`fileNamingTemplate` Naming template for output files.

Constructor Summary

Constructors
Constructor and Description
`FileBasedSink(String baseOutputFilename, String extension)` Construct a FileBasedSink with the given base output filename and extension.
`FileBasedSink(String baseOutputFilename, String extension, String fileNamingTemplate)` Construct a FileBasedSink with the given base output filename, extension, and file naming template.
`FileBasedSink(ValueProvider<String> baseOutputFilename, String extension, String fileNamingTemplate)` Construct a FileBasedSink with the given base output filename, extension, and file naming template.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`abstract FileBasedSink.FileBasedWriteOperation<T>`	`createWriteOperation(PipelineOptions options)` Return a subclass of `FileBasedSink.FileBasedWriteOperation` that will manage the write to the sink.
`String`	`getBaseOutputFilename()` Returns the base output filename for this file based sink.
`ValueProvider<String>`	`getBaseOutputFilenameProvider()` Returns the base output filename for this file based sink.
`void`	`populateDisplayData(DisplayData.Builder builder)` Register display data for the given transform or component.
`void`	`validate(PipelineOptions options)` Perform pipeline-construction-time validation.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - baseOutputFilename
```
protected final ValueProvider<String> baseOutputFilename
```
    Base filename for final output files.
  - extension
```
protected final String extension
```
    The extension to be used for the final output files.
  - fileNamingTemplate
```
protected final String fileNamingTemplate
```
    Naming template for output files. See ShardNameTemplate for a description of possible naming templates. Default is ShardNameTemplate.INDEX_OF_MAX.
- Constructor Detail
  - FileBasedSink
```
public FileBasedSink(String baseOutputFilename,
                     String extension)
```
    Construct a FileBasedSink with the given base output filename and extension.
  - FileBasedSink
```
public FileBasedSink(String baseOutputFilename,
                     String extension,
                     String fileNamingTemplate)
```
    Construct a FileBasedSink with the given base output filename, extension, and file naming template.
    See ShardNameTemplate for a description of file naming templates.
  - FileBasedSink
```
public FileBasedSink(ValueProvider<String> baseOutputFilename,
                     String extension,
                     String fileNamingTemplate)
```
    Construct a FileBasedSink with the given base output filename, extension, and file naming template.
    See ShardNameTemplate for a description of file naming templates.
- Method Detail
  - getBaseOutputFilename
```
public String getBaseOutputFilename()
```
    Returns the base output filename for this file based sink.
  - getBaseOutputFilenameProvider
```
public ValueProvider<String> getBaseOutputFilenameProvider()
```
    Returns the base output filename for this file based sink.
  - validate
```
public void validate(PipelineOptions options)
```
    Perform pipeline-construction-time validation. The default implementation is a no-op. Subclasses should override to ensure the sink is valid and can be written to. It is recommended to use Preconditions in the implementation of this method.
    
    Specified by:
    
    validate in class Sink<T>
  - createWriteOperation
```
public abstract FileBasedSink.FileBasedWriteOperation<T> createWriteOperation(PipelineOptions options)
```
    Return a subclass of FileBasedSink.FileBasedWriteOperation that will manage the write to the sink.
    
    Specified by:
    
    createWriteOperation in class Sink<T>
  - populateDisplayData
```
public void populateDisplayData(DisplayData.Builder builder)
```
    Description copied from class: Sink
    
    Register display data for the given transform or component.
    populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.
    By default, does not register any display data. Implementors may override this method to provide their own display data.
    
    Specified by:
    
    populateDisplayData in interface HasDisplayData
    
    Overrides:
    
    populateDisplayData in class Sink<T>
    
    Parameters:
    
    builder - The builder to populate with display data.
    
    See Also:
    
    HasDisplayData

Us ha resultat útil aquesta pàgina? Feu-nos-ho saber:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Última actualització: desembre 4, 2019

FileBasedSink (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Class FileBasedSink<T>

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Sink

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

baseOutputFilename

extension

fileNamingTemplate

Constructor Detail

FileBasedSink

FileBasedSink

FileBasedSink

Method Detail

getBaseOutputFilename

getBaseOutputFilenameProvider

validate

createWriteOperation

populateDisplayData

Envia suggeriments sobre...