Summary of entries of Classes for bigquerystorage.
Classes
BigQueryReadClient
Client for interacting with BigQuery Storage API.
The BigQuery storage API can be used to read data stored in BigQuery.
BigQueryWriteClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
For supplementary information about the Write API, see:
ReadRowsIterable
An iterable of rows from a read session.
ReadRowsPage
An iterator of rows from a read session message.
ReadRowsStream
A stream of results from a read rows request.
This stream is an iterable of ReadRowsResponse. Iterate over it to fetch all row messages.
If the fastavro library is installed, use the rows() method to parse all messages into a stream of row dictionaries.
If the pandas and fastavro libraries are installed, use the
to_dataframe()
method to parse all messages into a pandas.DataFrame.
This object should not be created directly, but is returned by other methods in this library.
BigQueryReadAsyncClient
BigQuery Read API.
The Read API can be used to read data from BigQuery.
BigQueryReadClient
BigQuery Read API.
The Read API can be used to read data from BigQuery.
BigQueryWriteAsyncClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
For supplementary information about the Write API, see:
BigQueryWriteClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
For supplementary information about the Write API, see:
AppendRowsRequest
Request message for AppendRows.
Because AppendRows is a bidirectional streaming RPC, certain parts of the AppendRowsRequest need only be specified for the first request before switching table destinations. You can also switch table destinations within the same connection for the default stream.
The size of a single AppendRowsRequest must be less than 10 MB in
size. Requests larger than this return an error, typically
INVALID_ARGUMENT.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
MissingValueInterpretation
An enum to indicate how to interpret missing values of fields that are present in user schema but missing in rows. A missing value can represent a NULL or a column default value defined in BigQuery table schema.
Enum values:
MISSING_VALUE_INTERPRETATION_UNSPECIFIED (0):
Invalid missing value interpretation.
Requests with this value will be rejected.
NULL_VALUE (1):
Missing value is interpreted as NULL.
DEFAULT_VALUE (2):
Missing value is interpreted as column
default value if declared in the table schema,
NULL otherwise.
MissingValueInterpretationsEntry
The abstract base class for a message.
ProtoData
ProtoData contains the data rows and schema when constructing append requests.
AppendRowsResponse
Response message for AppendRows.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
AppendResult
AppendResult is returned for successful append requests.
ArrowRecordBatch
Arrow RecordBatch.
ArrowSchema
Arrow schema as specified in https://arrow.apache.org/docs/python/api/datatypes.html and serialized to bytes using IPC:
https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc
See code samples on how this message can be deserialized.
ArrowSerializationOptions
Contains options specific to Arrow Serialization.
CompressionCodec
Compression codec's supported by Arrow.
Enum values:
COMPRESSION_UNSPECIFIED (0):
If unspecified no compression will be used.
LZ4_FRAME (1):
LZ4 Frame
(https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md)
ZSTD (2):
Zstandard compression.
AvroRows
Avro rows.
AvroSchema
Avro schema.
AvroSerializationOptions
Contains options specific to Avro Serialization.
BatchCommitWriteStreamsRequest
Request message for BatchCommitWriteStreams.
BatchCommitWriteStreamsResponse
Response message for BatchCommitWriteStreams.
CreateReadSessionRequest
Request message for CreateReadSession.
CreateWriteStreamRequest
Request message for CreateWriteStream.
DataFormat
Data format for input or output data.
Enum values:
DATA_FORMAT_UNSPECIFIED (0):
Data format is unspecified.
AVRO (1):
Avro is a standard open source row based file
format. See https://avro.apache.org/ for more
details.
ARROW (2):
Arrow is a standard open source column-based
message format. See https://arrow.apache.org/
for more details.
FinalizeWriteStreamRequest
Request message for invoking FinalizeWriteStream.
FinalizeWriteStreamResponse
Response message for FinalizeWriteStream.
FlushRowsRequest
Request message for FlushRows.
FlushRowsResponse
Respond message for FlushRows.
GetWriteStreamRequest
Request message for GetWriteStreamRequest.
ProtoRows
ProtoSchema
ProtoSchema describes the schema of the serialized protocol buffer data rows.
ReadRowsRequest
Request message for ReadRows.
ReadRowsResponse
Response from calling ReadRows may include row data, progress
and throttling information.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
ReadSession
Information about the ReadSession.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
TableModifiers
Additional attributes when reading a table.
TableReadOptions
Options dictating how we read a table.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
ResponseCompressionCodec
Specifies which compression codec to attempt on the entire serialized response payload (either Arrow record batch or Avro rows). This is not to be confused with the Apache Arrow native compression codecs specified in ArrowSerializationOptions. For performance reasons, when creating a read session requesting Arrow responses, setting both native Arrow compression and application-level response compression will not be allowed - choose, at most, one kind of compression.
Enum values:
RESPONSE_COMPRESSION_CODEC_UNSPECIFIED (0):
Default is no compression.
RESPONSE_COMPRESSION_CODEC_LZ4 (2):
Use raw LZ4 compression.
ReadStream
Information about a single stream that gets data out of the storage
system. Most of the information about ReadStream instances is
aggregated, making ReadStream lightweight.
RowError
The message that presents row level error info in a request.
RowErrorCode
Error code for RowError.
Enum values:
ROW_ERROR_CODE_UNSPECIFIED (0):
Default error.
FIELDS_ERROR (1):
One or more fields in the row has errors.
SplitReadStreamRequest
Request message for SplitReadStream.
SplitReadStreamResponse
Response message for SplitReadStream.
StorageError
Structured custom BigQuery Storage error message. The error can be attached as error details in the returned rpc Status. In particular, the use of error codes allows more structured error handling, and reduces the need to evaluate unstructured error text strings.
StorageErrorCode
Error code for StorageError.
Enum values:
STORAGE_ERROR_CODE_UNSPECIFIED (0):
Default error.
TABLE_NOT_FOUND (1):
Table is not found in the system.
STREAM_ALREADY_COMMITTED (2):
Stream is already committed.
STREAM_NOT_FOUND (3):
Stream is not found.
INVALID_STREAM_TYPE (4):
Invalid Stream type.
For example, you try to commit a stream that is
not pending.
INVALID_STREAM_STATE (5):
Invalid Stream state.
For example, you try to commit a stream that is
not finalized or is garbaged.
STREAM_FINALIZED (6):
Stream is finalized.
SCHEMA_MISMATCH_EXTRA_FIELDS (7):
There is a schema mismatch and it is caused
by user schema has extra field than bigquery
schema.
OFFSET_ALREADY_EXISTS (8):
Offset already exists.
OFFSET_OUT_OF_RANGE (9):
Offset out of range.
CMEK_NOT_PROVIDED (10):
Customer-managed encryption key (CMEK) not
provided for CMEK-enabled data.
INVALID_CMEK_PROVIDED (11):
Customer-managed encryption key (CMEK) was
incorrectly provided.
CMEK_ENCRYPTION_ERROR (12):
There is an encryption error while using
customer-managed encryption key.
KMS_SERVICE_ERROR (13):
Key Management Service (KMS) service returned
an error, which can be retried.
KMS_PERMISSION_DENIED (14):
Permission denied while using
customer-managed encryption key.
StreamStats
Estimated stream statistics for a given read Stream.
Progress
TableFieldSchema
TableFieldSchema defines a single field/column within a table schema.
FieldElementType
Represents the type of a field element.
Mode
Enum values:
MODE_UNSPECIFIED (0):
Illegal value
NULLABLE (1):
No description available.
REQUIRED (2):
No description available.
REPEATED (3):
No description available.
Type
Enum values:
TYPE_UNSPECIFIED (0):
Illegal value
STRING (1):
64K, UTF8
INT64 (2):
64-bit signed
DOUBLE (3):
64-bit IEEE floating point
STRUCT (4):
Aggregate type
BYTES (5):
64K, Binary
BOOL (6):
2-valued
TIMESTAMP (7):
64-bit signed usec since UTC epoch
DATE (8):
Civil date - Year, Month, Day
TIME (9):
Civil time - Hour, Minute, Second,
Microseconds
DATETIME (10):
Combination of civil date and civil time
GEOGRAPHY (11):
Geography object
NUMERIC (12):
Numeric value
BIGNUMERIC (13):
BigNumeric value
INTERVAL (14):
Interval
JSON (15):
JSON, String
RANGE (16):
RANGE
TableSchema
Schema of a table. This schema is a subset of google.cloud.bigquery.v2.TableSchema containing information necessary to generate valid message to write to BigQuery.
ThrottleState
Information on if the current connection is being throttled.
WriteStream
Information about a single stream that gets data inside the storage system.
Type
Type enum of the stream.
Enum values:
TYPE_UNSPECIFIED (0):
Unknown type.
COMMITTED (1):
Data will commit automatically and appear as
soon as the write is acknowledged.
PENDING (2):
Data is invisible until the stream is
committed.
BUFFERED (3):
Data is only visible up to the offset to
which it was flushed.
WriteMode
Mode enum of the stream.
Enum values:
WRITE_MODE_UNSPECIFIED (0):
Unknown type.
INSERT (1):
Insert new records into the table.
It is the default value if customers do not
specify it.
WriteStreamView
WriteStreamView is a view enum that controls what details about a write stream should be returned.
Enum values:
WRITE_STREAM_VIEW_UNSPECIFIED (0):
The default / unset value.
BASIC (1):
The BASIC projection returns basic metadata
about a write stream. The basic view does not
include schema information. This is the default
view returned by GetWriteStream.
FULL (2):
The FULL projection returns all available
write stream metadata, including the schema.
CreateWriteStream returns the full projection of
write stream metadata.
BigQueryReadClient
Client for interacting with BigQuery Storage API.
The BigQuery storage API can be used to read data stored in BigQuery.
BigQueryWriteClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
The google.cloud.bigquery.storage.v1
API </bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1>__
should be used instead of the v1beta2 API for BigQueryWrite
operations.
BigQueryReadAsyncClient
BigQuery Read API.
The Read API can be used to read data from BigQuery.
New code should use the v1 Read API going forward, if they don't use Write API at the same time.
BigQueryReadClient
BigQuery Read API.
The Read API can be used to read data from BigQuery.
New code should use the v1 Read API going forward, if they don't use Write API at the same time.
BigQueryWriteAsyncClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
The google.cloud.bigquery.storage.v1
API </bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1>__
should be used instead of the v1beta2 API for BigQueryWrite
operations.
BigQueryWriteClient
BigQuery Write API.
The Write API can be used to write data to BigQuery.
The google.cloud.bigquery.storage.v1
API </bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1>__
should be used instead of the v1beta2 API for BigQueryWrite
operations.
AppendRowsRequest
Request message for AppendRows.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
ProtoData
Proto schema and data.
AppendRowsResponse
Response message for AppendRows.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
AppendResult
AppendResult is returned for successful append requests.
ArrowRecordBatch
Arrow RecordBatch.
ArrowSchema
Arrow schema as specified in https://arrow.apache.org/docs/python/api/datatypes.html and serialized to bytes using IPC:
https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc
See code samples on how this message can be deserialized.
ArrowSerializationOptions
Contains options specific to Arrow Serialization.
Format
The IPC format to use when serializing Arrow streams.
Enum values:
FORMAT_UNSPECIFIED (0):
If unspecied the IPC format as of 0.15
release will be used.
ARROW_0_14 (1):
Use the legacy IPC message format as of
Apache Arrow Release 0.14.
ARROW_0_15 (2):
Use the message format as of Apache Arrow
Release 0.15.
AvroRows
Avro rows.
AvroSchema
Avro schema.
BatchCommitWriteStreamsRequest
Request message for BatchCommitWriteStreams.
BatchCommitWriteStreamsResponse
Response message for BatchCommitWriteStreams.
CreateReadSessionRequest
Request message for CreateReadSession.
CreateWriteStreamRequest
Request message for CreateWriteStream.
DataFormat
Data format for input or output data.
Enum values:
DATA_FORMAT_UNSPECIFIED (0):
No description available.
AVRO (1):
Avro is a standard open source row based file
format. See https://avro.apache.org/ for more
details.
ARROW (2):
Arrow is a standard open source column-based
message format. See https://arrow.apache.org/
for more details.
FinalizeWriteStreamRequest
Request message for invoking FinalizeWriteStream.
FinalizeWriteStreamResponse
Response message for FinalizeWriteStream.
FlushRowsRequest
Request message for FlushRows.
FlushRowsResponse
Respond message for FlushRows.
GetWriteStreamRequest
Request message for GetWriteStreamRequest.
ProtoRows
ProtoSchema
ProtoSchema describes the schema of the serialized protocol buffer data rows.
ReadRowsRequest
Request message for ReadRows.
ReadRowsResponse
Response from calling ReadRows may include row data, progress
and throttling information.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
ReadSession
Information about the ReadSession.
This message has oneof_ fields (mutually exclusive fields).
For each oneof, at most one member field can be set at the same time.
Setting any member of the oneof automatically clears all other
members.
.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
TableModifiers
Additional attributes when reading a table.
TableReadOptions
Options dictating how we read a table.
ReadStream
Information about a single stream that gets data out of the storage
system. Most of the information about ReadStream instances is
aggregated, making ReadStream lightweight.
SplitReadStreamRequest
Request message for SplitReadStream.
SplitReadStreamResponse
StorageError
Structured custom BigQuery Storage error message. The error can be attached as error details in the returned rpc Status. In particular, the use of error codes allows more structured error handling, and reduces the need to evaluate unstructured error text strings.
StorageErrorCode
Error code for StorageError.
Enum values:
STORAGE_ERROR_CODE_UNSPECIFIED (0):
Default error.
TABLE_NOT_FOUND (1):
Table is not found in the system.
STREAM_ALREADY_COMMITTED (2):
Stream is already committed.
STREAM_NOT_FOUND (3):
Stream is not found.
INVALID_STREAM_TYPE (4):
Invalid Stream type.
For example, you try to commit a stream that is
not pending.
INVALID_STREAM_STATE (5):
Invalid Stream state.
For example, you try to commit a stream that is
not finalized or is garbaged.
STREAM_FINALIZED (6):
Stream is finalized.
StreamStats
Estimated stream statistics for a given Stream.
Progress
TableFieldSchema
A field in TableSchema
Mode
Enum values:
MODE_UNSPECIFIED (0):
Illegal value
NULLABLE (1):
No description available.
REQUIRED (2):
No description available.
REPEATED (3):
No description available.
Type
Enum values:
TYPE_UNSPECIFIED (0):
Illegal value
STRING (1):
64K, UTF8
INT64 (2):
64-bit signed
DOUBLE (3):
64-bit IEEE floating point
STRUCT (4):
Aggregate type
BYTES (5):
64K, Binary
BOOL (6):
2-valued
TIMESTAMP (7):
64-bit signed usec since UTC epoch
DATE (8):
Civil date - Year, Month, Day
TIME (9):
Civil time - Hour, Minute, Second,
Microseconds
DATETIME (10):
Combination of civil date and civil time
GEOGRAPHY (11):
Geography object
NUMERIC (12):
Numeric value
BIGNUMERIC (13):
BigNumeric value
INTERVAL (14):
Interval
JSON (15):
JSON, String
TableSchema
Schema of a table
ThrottleState
Information on if the current connection is being throttled.
WriteStream
Information about a single stream that gets data inside the storage system.
Type
Type enum of the stream.
Enum values:
TYPE_UNSPECIFIED (0):
Unknown type.
COMMITTED (1):
Data will commit automatically and appear as
soon as the write is acknowledged.
PENDING (2):
Data is invisible until the stream is
committed.
BUFFERED (3):
Data is only visible up to the offset to
which it was flushed.
AppendRowsFuture
Encapsulation of the asynchronous execution of an action.
This object is returned from long-running BigQuery Storage API calls, and is the interface to determine the status of those calls.
This object should not be created directly, but is returned by other methods in this library.
AppendRowsStream
A manager object which can append rows to a stream.
Modules
client
Parent client for calling the Cloud BigQuery Storage API.
This is the base from which all interactions with the API occur.
reader
API documentation for bigquery_storage_v1.reader module.
client
Parent client for calling the Cloud BigQuery Storage API.
This is the base from which all interactions with the API occur.
writer
API documentation for bigquery_storage_v1beta2.writer module.