Class JsonStreamWriter (2.18.0)

public class JsonStreamWriter implements AutoCloseable

A StreamWriter that can write JSON data (JSONObjects) to BigQuery tables. The JsonStreamWriter is built on top of a StreamWriter, and it simply converts all JSON data to protobuf messages then calls StreamWriter's append() method to write to BigQuery tables. It maintains all StreamWriter functions, but also provides an additional feature: schema update support, where if the BigQuery table schema is updated, users will be able to ingest data on the new schema after some time (in order of minutes).

Inheritance

java.lang.Object > JsonStreamWriter

Implements

AutoCloseable

Static Methods

newBuilder(String streamOrTableName, TableSchema tableSchema)

public static JsonStreamWriter.Builder newBuilder(String streamOrTableName, TableSchema tableSchema)

newBuilder that constructs a JsonStreamWriter builder with BigQuery client being initialized by StreamWriter by default.

Parameters
NameDescription
streamOrTableNameString

name of the stream that must follow "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+" or table name "projects/[^/]+/datasets/[^/]+/tables/[^/]+"

tableSchemaTableSchema

The schema of the table when the stream was created, which is passed back through WriteStream

Returns
TypeDescription
JsonStreamWriter.Builder

Builder

newBuilder(String streamOrTableName, TableSchema tableSchema, BigQueryWriteClient client)

public static JsonStreamWriter.Builder newBuilder(String streamOrTableName, TableSchema tableSchema, BigQueryWriteClient client)

newBuilder that constructs a JsonStreamWriter builder.

Parameters
NameDescription
streamOrTableNameString

name of the stream that must follow "projects/[^/]+/datasets/[^/]+/tables/[^/]+/streams/[^/]+"

tableSchemaTableSchema

The schema of the table when the stream was created, which is passed back through WriteStream

clientBigQueryWriteClient
Returns
TypeDescription
JsonStreamWriter.Builder

Builder

Methods

append(JSONArray jsonArr)

public ApiFuture<AppendRowsResponse> append(JSONArray jsonArr)

Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at current end of stream. If there is a schema update, the current StreamWriter is closed. A new StreamWriter is created with the updated TableSchema.

Parameter
NameDescription
jsonArrorg.json.JSONArray

The JSON array that contains JSONObjects to be written

Returns
TypeDescription
ApiFuture<AppendRowsResponse>

ApiFuture<AppendRowsResponse> returns an AppendRowsResponse message wrapped in an ApiFuture

Exceptions
TypeDescription
IOException
DescriptorValidationException

append(JSONArray jsonArr, long offset)

public ApiFuture<AppendRowsResponse> append(JSONArray jsonArr, long offset)

Writes a JSONArray that contains JSONObjects to the BigQuery table by first converting the JSON data to protobuf messages, then using StreamWriter's append() to write the data at the specified offset. If there is a schema update, the current StreamWriter is closed. A new StreamWriter is created with the updated TableSchema.

Parameters
NameDescription
jsonArrorg.json.JSONArray

The JSON array that contains JSONObjects to be written

offsetlong

Offset for deduplication

Returns
TypeDescription
ApiFuture<AppendRowsResponse>

ApiFuture<AppendRowsResponse> returns an AppendRowsResponse message wrapped in an ApiFuture

Exceptions
TypeDescription
IOException
DescriptorValidationException

close()

public void close()

Closes the underlying StreamWriter.

getDescriptor()

public Descriptors.Descriptor getDescriptor()

Gets current descriptor

Returns
TypeDescription
Descriptor

Descriptor

getInflightWaitSeconds()

public long getInflightWaitSeconds()

Returns the wait of a request in Client side before sending to the Server. Request could wait in Client because it reached the client side inflight request limit (adjustable when constructing the Writer). The value is the wait time for the last sent request. A constant high wait value indicates a need for more throughput, you can create a new Stream for to increase the throughput in exclusive stream case, or create a new Writer in the default stream case.

Returns
TypeDescription
long

getStreamName()

public String getStreamName()

Gets streamName

Returns
TypeDescription
String

String