public class BigQueryWriteClient implements BackgroundResource
Service Description: BigQuery Write API.
The Write API can be used to write data to BigQuery.
For supplementary information about the Write API, see: https://cloud.google.com/bigquery/docs/write-api
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
TableName parent = TableName.of("[PROJECT]", "[DATASET]", "[TABLE]");
WriteStream writeStream = WriteStream.newBuilder().build();
WriteStream response = bigQueryWriteClient.createWriteStream(parent, writeStream);
}
Note: close() needs to be called on the BigQueryWriteClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of BigQueryWriteSettings to create(). For example:
To customize credentials:
BigQueryWriteSettings bigQueryWriteSettings =
BigQueryWriteSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create(bigQueryWriteSettings);
To customize the endpoint:
BigQueryWriteSettings bigQueryWriteSettings =
BigQueryWriteSettings.newBuilder().setEndpoint(myEndpoint).build();
BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create(bigQueryWriteSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
Implements
BackgroundResourceStatic Methods
create()
public static final BigQueryWriteClient create()
Constructs an instance of BigQueryWriteClient with default settings.
Type | Description |
BigQueryWriteClient |
Type | Description |
IOException |
create(BigQueryWriteSettings settings)
public static final BigQueryWriteClient create(BigQueryWriteSettings settings)
Constructs an instance of BigQueryWriteClient, using the given settings. The channels are created based on the settings passed in, or defaults for any settings that are not set.
Name | Description |
settings | BigQueryWriteSettings |
Type | Description |
BigQueryWriteClient |
Type | Description |
IOException |
create(BigQueryWriteStub stub)
public static final BigQueryWriteClient create(BigQueryWriteStub stub)
Constructs an instance of BigQueryWriteClient, using the given stub for making calls. This is for advanced usage - prefer using create(BigQueryWriteSettings).
Name | Description |
stub | BigQueryWriteStub |
Type | Description |
BigQueryWriteClient |
Constructors
BigQueryWriteClient(BigQueryWriteSettings settings)
protected BigQueryWriteClient(BigQueryWriteSettings settings)
Constructs an instance of BigQueryWriteClient, using the given settings. This is protected so that it is easy to make a subclass, but otherwise, the static factory methods should be preferred.
Name | Description |
settings | BigQueryWriteSettings |
BigQueryWriteClient(BigQueryWriteStub stub)
protected BigQueryWriteClient(BigQueryWriteStub stub)
Name | Description |
stub | BigQueryWriteStub |
Methods
appendRowsCallable()
public final BidiStreamingCallable<AppendRowsRequest,AppendRowsResponse> appendRowsCallable()
Appends data to the given stream.
If offset
is specified, the offset
is checked against the end of stream. The server
returns OUT_OF_RANGE
in AppendRowsResponse
if an attempt is made to append to an offset
beyond the current end of the stream or ALREADY_EXISTS
if user provides an offset
that has
already been written to. User can retry with adjusted offset within the same RPC connection. If
offset
is not specified, append happens at the end of the stream.
The response contains an optional offset at which the append happened. No offset information will be returned for appends to a default stream.
Responses are received in the same order in which requests are sent. There will be one response for each successful inserted request. Responses may optionally embed error information if the originating AppendRequest was not successfully processed.
The specifics of when successfully appended data is made visible to the table are governed by the type of stream:
- For COMMITTED streams (which includes the default stream), data is visible immediately upon successful append.
- For BUFFERED streams, data is made visible via a subsequent
FlushRows
rpc which advances a cursor to a newer offset in the stream.
- For PENDING streams, data is not made visible until the stream itself is finalized (via
the
FinalizeWriteStream
rpc), and the stream is explicitly committed via theBatchCommitWriteStreams
rpc.
Note: For users coding against the gRPC api directly, it may be necessary to supply the
x-goog-request-params system parameter with write_stream=<full_write_stream_name>
.
More information about system parameters: https://cloud.google.com/apis/docs/system-parameters
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
BidiStream<AppendRowsRequest, AppendRowsResponse> bidiStream =
bigQueryWriteClient.appendRowsCallable().call();
AppendRowsRequest request =
AppendRowsRequest.newBuilder()
.setWriteStream(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.setOffset(Int64Value.newBuilder().build())
.setTraceId("traceId-1067401920")
.build();
bidiStream.send(request);
for (AppendRowsResponse response : bidiStream) {
// Do something when a response is received.
}
}
Type | Description |
BidiStreamingCallable<AppendRowsRequest,AppendRowsResponse> |
awaitTermination(long duration, TimeUnit unit)
public boolean awaitTermination(long duration, TimeUnit unit)
Name | Description |
duration | long |
unit | TimeUnit |
Type | Description |
boolean |
Type | Description |
InterruptedException |
batchCommitWriteStreams(BatchCommitWriteStreamsRequest request)
public final BatchCommitWriteStreamsResponse batchCommitWriteStreams(BatchCommitWriteStreamsRequest request)
Atomically commits a group of PENDING
streams that belong to the same parent
table.
Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
BatchCommitWriteStreamsRequest request =
BatchCommitWriteStreamsRequest.newBuilder()
.setParent(TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString())
.addAllWriteStreams(new ArrayList<String>())
.build();
BatchCommitWriteStreamsResponse response =
bigQueryWriteClient.batchCommitWriteStreams(request);
}
Name | Description |
request | BatchCommitWriteStreamsRequest The request object containing all of the parameters for the API call. |
Type | Description |
BatchCommitWriteStreamsResponse |
batchCommitWriteStreams(TableName parent)
public final BatchCommitWriteStreamsResponse batchCommitWriteStreams(TableName parent)
Atomically commits a group of PENDING
streams that belong to the same parent
table.
Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
TableName parent = TableName.of("[PROJECT]", "[DATASET]", "[TABLE]");
BatchCommitWriteStreamsResponse response =
bigQueryWriteClient.batchCommitWriteStreams(parent);
}
Name | Description |
parent | TableName Required. Parent table that all the streams should belong to, in the form of
|
Type | Description |
BatchCommitWriteStreamsResponse |
batchCommitWriteStreams(String parent)
public final BatchCommitWriteStreamsResponse batchCommitWriteStreams(String parent)
Atomically commits a group of PENDING
streams that belong to the same parent
table.
Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
String parent = TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString();
BatchCommitWriteStreamsResponse response =
bigQueryWriteClient.batchCommitWriteStreams(parent);
}
Name | Description |
parent | String Required. Parent table that all the streams should belong to, in the form of
|
Type | Description |
BatchCommitWriteStreamsResponse |
batchCommitWriteStreamsCallable()
public final UnaryCallable<BatchCommitWriteStreamsRequest,BatchCommitWriteStreamsResponse> batchCommitWriteStreamsCallable()
Atomically commits a group of PENDING
streams that belong to the same parent
table.
Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
BatchCommitWriteStreamsRequest request =
BatchCommitWriteStreamsRequest.newBuilder()
.setParent(TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString())
.addAllWriteStreams(new ArrayList<String>())
.build();
ApiFuture<BatchCommitWriteStreamsResponse> future =
bigQueryWriteClient.batchCommitWriteStreamsCallable().futureCall(request);
// Do something.
BatchCommitWriteStreamsResponse response = future.get();
}
Type | Description |
UnaryCallable<BatchCommitWriteStreamsRequest,BatchCommitWriteStreamsResponse> |
close()
public final void close()
createWriteStream(CreateWriteStreamRequest request)
public final WriteStream createWriteStream(CreateWriteStreamRequest request)
Creates a write stream to the given table. Additionally, every table has a special stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
CreateWriteStreamRequest request =
CreateWriteStreamRequest.newBuilder()
.setParent(TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString())
.setWriteStream(WriteStream.newBuilder().build())
.build();
WriteStream response = bigQueryWriteClient.createWriteStream(request);
}
Name | Description |
request | CreateWriteStreamRequest The request object containing all of the parameters for the API call. |
Type | Description |
WriteStream |
createWriteStream(TableName parent, WriteStream writeStream)
public final WriteStream createWriteStream(TableName parent, WriteStream writeStream)
Creates a write stream to the given table. Additionally, every table has a special stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
TableName parent = TableName.of("[PROJECT]", "[DATASET]", "[TABLE]");
WriteStream writeStream = WriteStream.newBuilder().build();
WriteStream response = bigQueryWriteClient.createWriteStream(parent, writeStream);
}
Name | Description |
parent | TableName Required. Reference to the table to which the stream belongs, in the format of
|
writeStream | WriteStream Required. Stream to be created. |
Type | Description |
WriteStream |
createWriteStream(String parent, WriteStream writeStream)
public final WriteStream createWriteStream(String parent, WriteStream writeStream)
Creates a write stream to the given table. Additionally, every table has a special stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
String parent = TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString();
WriteStream writeStream = WriteStream.newBuilder().build();
WriteStream response = bigQueryWriteClient.createWriteStream(parent, writeStream);
}
Name | Description |
parent | String Required. Reference to the table to which the stream belongs, in the format of
|
writeStream | WriteStream Required. Stream to be created. |
Type | Description |
WriteStream |
createWriteStreamCallable()
public final UnaryCallable<CreateWriteStreamRequest,WriteStream> createWriteStreamCallable()
Creates a write stream to the given table. Additionally, every table has a special stream named '_default' to which data can be written. This stream doesn't need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
CreateWriteStreamRequest request =
CreateWriteStreamRequest.newBuilder()
.setParent(TableName.of("[PROJECT]", "[DATASET]", "[TABLE]").toString())
.setWriteStream(WriteStream.newBuilder().build())
.build();
ApiFuture<WriteStream> future =
bigQueryWriteClient.createWriteStreamCallable().futureCall(request);
// Do something.
WriteStream response = future.get();
}
Type | Description |
UnaryCallable<CreateWriteStreamRequest,WriteStream> |
finalizeWriteStream(FinalizeWriteStreamRequest request)
public final FinalizeWriteStreamResponse finalizeWriteStream(FinalizeWriteStreamRequest request)
Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
FinalizeWriteStreamRequest request =
FinalizeWriteStreamRequest.newBuilder()
.setName(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.build();
FinalizeWriteStreamResponse response = bigQueryWriteClient.finalizeWriteStream(request);
}
Name | Description |
request | FinalizeWriteStreamRequest The request object containing all of the parameters for the API call. |
Type | Description |
FinalizeWriteStreamResponse |
finalizeWriteStream(WriteStreamName name)
public final FinalizeWriteStreamResponse finalizeWriteStream(WriteStreamName name)
Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
WriteStreamName name = WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]");
FinalizeWriteStreamResponse response = bigQueryWriteClient.finalizeWriteStream(name);
}
Name | Description |
name | WriteStreamName Required. Name of the stream to finalize, in the form of
|
Type | Description |
FinalizeWriteStreamResponse |
finalizeWriteStream(String name)
public final FinalizeWriteStreamResponse finalizeWriteStream(String name)
Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
String name = WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString();
FinalizeWriteStreamResponse response = bigQueryWriteClient.finalizeWriteStream(name);
}
Name | Description |
name | String Required. Name of the stream to finalize, in the form of
|
Type | Description |
FinalizeWriteStreamResponse |
finalizeWriteStreamCallable()
public final UnaryCallable<FinalizeWriteStreamRequest,FinalizeWriteStreamResponse> finalizeWriteStreamCallable()
Finalize a write stream so that no new data can be appended to the stream. Finalize is not supported on the '_default' stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
FinalizeWriteStreamRequest request =
FinalizeWriteStreamRequest.newBuilder()
.setName(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.build();
ApiFuture<FinalizeWriteStreamResponse> future =
bigQueryWriteClient.finalizeWriteStreamCallable().futureCall(request);
// Do something.
FinalizeWriteStreamResponse response = future.get();
}
Type | Description |
UnaryCallable<FinalizeWriteStreamRequest,FinalizeWriteStreamResponse> |
flushRows(FlushRowsRequest request)
public final FlushRowsResponse flushRows(FlushRowsRequest request)
Flushes rows to a BUFFERED stream.
If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request.
Flush is not supported on the _default stream, since it is not BUFFERED.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
FlushRowsRequest request =
FlushRowsRequest.newBuilder()
.setWriteStream(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.setOffset(Int64Value.newBuilder().build())
.build();
FlushRowsResponse response = bigQueryWriteClient.flushRows(request);
}
Name | Description |
request | FlushRowsRequest The request object containing all of the parameters for the API call. |
Type | Description |
FlushRowsResponse |
flushRows(WriteStreamName writeStream)
public final FlushRowsResponse flushRows(WriteStreamName writeStream)
Flushes rows to a BUFFERED stream.
If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request.
Flush is not supported on the _default stream, since it is not BUFFERED.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
WriteStreamName writeStream =
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]");
FlushRowsResponse response = bigQueryWriteClient.flushRows(writeStream);
}
Name | Description |
writeStream | WriteStreamName Required. The stream that is the target of the flush operation. |
Type | Description |
FlushRowsResponse |
flushRows(String writeStream)
public final FlushRowsResponse flushRows(String writeStream)
Flushes rows to a BUFFERED stream.
If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request.
Flush is not supported on the _default stream, since it is not BUFFERED.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
String writeStream =
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString();
FlushRowsResponse response = bigQueryWriteClient.flushRows(writeStream);
}
Name | Description |
writeStream | String Required. The stream that is the target of the flush operation. |
Type | Description |
FlushRowsResponse |
flushRowsCallable()
public final UnaryCallable<FlushRowsRequest,FlushRowsResponse> flushRowsCallable()
Flushes rows to a BUFFERED stream.
If users are appending rows to BUFFERED stream, flush operation is required in order for the rows to become available for reading. A Flush operation flushes up to any previously flushed offset in a BUFFERED stream, to the offset specified in the request.
Flush is not supported on the _default stream, since it is not BUFFERED.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
FlushRowsRequest request =
FlushRowsRequest.newBuilder()
.setWriteStream(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.setOffset(Int64Value.newBuilder().build())
.build();
ApiFuture<FlushRowsResponse> future =
bigQueryWriteClient.flushRowsCallable().futureCall(request);
// Do something.
FlushRowsResponse response = future.get();
}
Type | Description |
UnaryCallable<FlushRowsRequest,FlushRowsResponse> |
getSettings()
public final BigQueryWriteSettings getSettings()
Type | Description |
BigQueryWriteSettings |
getStub()
public BigQueryWriteStub getStub()
Type | Description |
BigQueryWriteStub |
getWriteStream(GetWriteStreamRequest request)
public final WriteStream getWriteStream(GetWriteStreamRequest request)
Gets information about a write stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
GetWriteStreamRequest request =
GetWriteStreamRequest.newBuilder()
.setName(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.build();
WriteStream response = bigQueryWriteClient.getWriteStream(request);
}
Name | Description |
request | GetWriteStreamRequest The request object containing all of the parameters for the API call. |
Type | Description |
WriteStream |
getWriteStream(WriteStreamName name)
public final WriteStream getWriteStream(WriteStreamName name)
Gets information about a write stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
WriteStreamName name = WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]");
WriteStream response = bigQueryWriteClient.getWriteStream(name);
}
Name | Description |
name | WriteStreamName Required. Name of the stream to get, in the form of
|
Type | Description |
WriteStream |
getWriteStream(String name)
public final WriteStream getWriteStream(String name)
Gets information about a write stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
String name = WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString();
WriteStream response = bigQueryWriteClient.getWriteStream(name);
}
Name | Description |
name | String Required. Name of the stream to get, in the form of
|
Type | Description |
WriteStream |
getWriteStreamCallable()
public final UnaryCallable<GetWriteStreamRequest,WriteStream> getWriteStreamCallable()
Gets information about a write stream.
Sample code:
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create()) {
GetWriteStreamRequest request =
GetWriteStreamRequest.newBuilder()
.setName(
WriteStreamName.of("[PROJECT]", "[DATASET]", "[TABLE]", "[STREAM]").toString())
.build();
ApiFuture<WriteStream> future =
bigQueryWriteClient.getWriteStreamCallable().futureCall(request);
// Do something.
WriteStream response = future.get();
}
Type | Description |
UnaryCallable<GetWriteStreamRequest,WriteStream> |
isShutdown()
public boolean isShutdown()
Type | Description |
boolean |
isTerminated()
public boolean isTerminated()
Type | Description |
boolean |
shutdown()
public void shutdown()
shutdownNow()
public void shutdownNow()