Google Cloud Dataflow SDK for Java, version 1.9.1
Class BigQueryIO.Write
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.io.BigQueryIO.Write
-
- Enclosing class:
- BigQueryIO
public static class BigQueryIO.Write extends Object
APTransform
that writes aPCollection
containingTableRows
to a BigQuery table.In BigQuery, each table has an enclosing dataset. The dataset being written must already exist.
By default, tables will be created if they do not exist, which corresponds to a
BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED
disposition that matches the default of BigQuery's Jobs API. A schema must be provided (viawithSchema(TableSchema)
), or else the transform may fail at runtime with anIllegalArgumentException
.By default, writes require an empty table, which corresponds to a
BigQueryIO.Write.WriteDisposition.WRITE_EMPTY
disposition that matches the default of BigQuery's Jobs API.Here is a sample transform that produces TableRow values containing "word" and "count" columns:
static class FormatCountsFn extends DoFn<KV<String, Long>, TableRow> { public void processElement(ProcessContext c) { TableRow row = new TableRow() .set("word", c.element().getKey()) .set("count", c.element().getValue().intValue()); c.output(row); } }
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
BigQueryIO.Write.Bound
APTransform
that can write either a bounded or unboundedPCollection
ofTableRows
to a BigQuery table.static class
BigQueryIO.Write.CreateDisposition
An enumeration type for the BigQuery create disposition strings.static class
BigQueryIO.Write.WriteDisposition
An enumeration type for the BigQuery write disposition strings.
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method and Description static BigQueryIO.Write.Bound
named(String name)
Creates a write transformation with the given transform name.static BigQueryIO.Write.Bound
to(SerializableFunction<BoundedWindow,String> tableSpecFunction)
Creates a write transformation from a function that maps windows to table specifications.static BigQueryIO.Write.Bound
to(String tableSpec)
Creates a write transformation for the given table specification.static BigQueryIO.Write.Bound
to(TableReference table)
Creates a write transformation for the given table.static BigQueryIO.Write.Bound
to(ValueProvider<String> tableSpec)
Creates a write transformation for the given table.static BigQueryIO.Write.Bound
toTableReference(SerializableFunction<BoundedWindow,TableReference> tableRefFunction)
Creates a write transformation from a function that maps windows toTableReference
objects.static BigQueryIO.Write.Bound
withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)
Creates a write transformation with the specified options for creating the table.static BigQueryIO.Write.Bound
withoutValidation()
Creates a write transformation with BigQuery table validation disabled.static BigQueryIO.Write.Bound
withSchema(TableSchema schema)
Creates a write transformation with the specified schema to use in table creation.static BigQueryIO.Write.Bound
withSchema(ValueProvider<TableSchema> schema)
LikewithSchema(TableSchema)
, but with aValueProvider
.static BigQueryIO.Write.Bound
withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)
Creates a write transformation with the specified options for writing to the table.
-
-
-
Method Detail
-
named
public static BigQueryIO.Write.Bound named(String name)
Creates a write transformation with the given transform name. The BigQuery table to be written has not yet been configured.
-
to
public static BigQueryIO.Write.Bound to(String tableSpec)
Creates a write transformation for the given table specification.Refer to
BigQueryIO.parseTableSpec(String)
for the specification format.
-
to
public static BigQueryIO.Write.Bound to(ValueProvider<String> tableSpec)
Creates a write transformation for the given table.
-
to
public static BigQueryIO.Write.Bound to(TableReference table)
Creates a write transformation for the given table.
-
to
public static BigQueryIO.Write.Bound to(SerializableFunction<BoundedWindow,String> tableSpecFunction)
Creates a write transformation from a function that maps windows to table specifications. Each time a new window is encountered, this function will be called and the resulting table will be created. Records within that window will be written to the associated table.See
BigQueryIO.parseTableSpec(String)
for the format thattableSpecFunction
should return.tableSpecFunction
should be deterministic. When given the same window, it should always return the same table specification.
-
toTableReference
public static BigQueryIO.Write.Bound toTableReference(SerializableFunction<BoundedWindow,TableReference> tableRefFunction)
Creates a write transformation from a function that maps windows toTableReference
objects.tableRefFunction
should be deterministic. When given the same window, it should always return the same table reference.
-
withSchema
public static BigQueryIO.Write.Bound withSchema(TableSchema schema)
Creates a write transformation with the specified schema to use in table creation.The schema is required only if writing to a table that does not already exist, and
BigQueryIO.Write.CreateDisposition
is set toBigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED
.
-
withSchema
public static BigQueryIO.Write.Bound withSchema(ValueProvider<TableSchema> schema)
LikewithSchema(TableSchema)
, but with aValueProvider
.
-
withCreateDisposition
public static BigQueryIO.Write.Bound withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)
Creates a write transformation with the specified options for creating the table.
-
withWriteDisposition
public static BigQueryIO.Write.Bound withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)
Creates a write transformation with the specified options for writing to the table.
-
withoutValidation
public static BigQueryIO.Write.Bound withoutValidation()
Creates a write transformation with BigQuery table validation disabled.
-
-