Enumerations

AuthorizationType

static

number

The type of authorization needed for this data source.

Value

AUTHORIZATION_TYPE_UNSPECIFIED

Type unspecified.

AUTHORIZATION_CODE

Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.

GOOGLE_PLUS_AUTHORIZATION_CODE

Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.

DataRefreshType

static

number

Represents how the data source supports data auto refresh.

Value

DATA_REFRESH_TYPE_UNSPECIFIED

The data source won't support data auto refresh, which is default value.

SLIDING_WINDOW

The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.

CUSTOM_SLIDING_WINDOW

The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.

MessageSeverity

static

number

Represents data transfer user facing message severity.

Value

MESSAGE_SEVERITY_UNSPECIFIED

No severity specified.

INFO

Informational message.

WARNING

Warning message.

ERROR

Error message.

RunAttempt

static

number

Represents which runs should be pulled.

Value

RUN_ATTEMPT_UNSPECIFIED

All runs should be returned.

LATEST

Only latest run per day should be returned.

TransferState

constant   static

number

Represents data transfer run state.

Value

TRANSFER_STATE_UNSPECIFIED

State placeholder.

PENDING

Data transfer is scheduled and is waiting to be picked up by data transfer backend.

RUNNING

Data transfer is in progress.

SUCCEEDED

Data transfer completed successsfully.

FAILED

Data transfer failed.

CANCELLED

Data transfer is cancelled.

TransferType

constant   static

number

DEPRECATED. Represents data transfer type.

Value

TRANSFER_TYPE_UNSPECIFIED

Invalid or Unknown transfer type placeholder.

BATCH

Batch data transfer.

STREAMING

Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.

Type

static

number

Parameter type.

Value

TYPE_UNSPECIFIED

Type unspecified.

STRING

String parameter.

INTEGER

Integer parameter (64-bits). Will be serialized to json as string.

DOUBLE

Double precision floating point parameter.

BOOLEAN

Boolean parameter.

RECORD

Record parameter.

PLUS_PAGE

Page ID for a Google+ Page.

Properties

AuthorizationType

static

number

The type of authorization needed for this data source.

Value

AUTHORIZATION_TYPE_UNSPECIFIED

Type unspecified.

AUTHORIZATION_CODE

Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.

GOOGLE_PLUS_AUTHORIZATION_CODE

Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.

DataRefreshType

static

number

Represents how the data source supports data auto refresh.

Value

DATA_REFRESH_TYPE_UNSPECIFIED

The data source won't support data auto refresh, which is default value.

SLIDING_WINDOW

The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.

CUSTOM_SLIDING_WINDOW

The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.

MessageSeverity

static

number

Represents data transfer user facing message severity.

Value

MESSAGE_SEVERITY_UNSPECIFIED

No severity specified.

INFO

Informational message.

WARNING

Warning message.

ERROR

Error message.

RunAttempt

static

number

Represents which runs should be pulled.

Value

RUN_ATTEMPT_UNSPECIFIED

All runs should be returned.

LATEST

Only latest run per day should be returned.

TransferState

constant   static

number

Represents data transfer run state.

Value

TRANSFER_STATE_UNSPECIFIED

State placeholder.

PENDING

Data transfer is scheduled and is waiting to be picked up by data transfer backend.

RUNNING

Data transfer is in progress.

SUCCEEDED

Data transfer completed successsfully.

FAILED

Data transfer failed.

CANCELLED

Data transfer is cancelled.

TransferType

constant   static

number

DEPRECATED. Represents data transfer type.

Value

TRANSFER_TYPE_UNSPECIFIED

Invalid or Unknown transfer type placeholder.

BATCH

Batch data transfer.

STREAMING

Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.

Type

static

number

Parameter type.

Value

TYPE_UNSPECIFIED

Type unspecified.

STRING

String parameter.

INTEGER

Integer parameter (64-bits). Will be serialized to json as string.

DOUBLE

Double precision floating point parameter.

BOOLEAN

Boolean parameter.

RECORD

Record parameter.

PLUS_PAGE

Page ID for a Google+ Page.

Abstract types

CheckValidCredsRequest

static

A request to determine whether the user has valid credentials. This method is used to limit the number of OAuth popups in the user interface. The user id is inferred from the API call context. If the data source has the Google+ authorization type, this method returns false, as it cannot be determined whether the credentials are already valid merely based on the user id.

Property

Parameter

name

string

The data source in the form: projects/{project_id}/dataSources/{data_source_id}

See also

google.cloud.bigquery.datatransfer.v1.CheckValidCredsRequest definition in proto format

CheckValidCredsResponse

static

A response indicating whether the credentials exist and are valid.

Property

Parameter

hasValidCreds

boolean

If set to true, the credentials exist and are valid.

See also

google.cloud.bigquery.datatransfer.v1.CheckValidCredsResponse definition in proto format

CreateTransferConfigRequest

static

A request to create a data transfer configuration. If new credentials are needed for this transfer configuration, an authorization code must be provided. If an authorization code is provided, the transfer configuration will be associated with the user id corresponding to the authorization code. Otherwise, the transfer configuration will be associated with the calling user.

Properties

Parameter

parent

string

The BigQuery project id where the transfer configuration should be created. Must be in the format /projects/{project_id}/locations/{location_id} If specified location and location of the destination bigquery dataset do not match - the request will fail.

transferConfig

Object

Data transfer configuration to create.

This object should have the same structure as TransferConfig

authorizationCode

string

Optional OAuth2 authorization code to use with this transfer configuration. This is required if new credentials are needed, as indicated by CheckValidCreds. In order to obtain authorization_code, please make a request to https:// www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id= &scope=<data_source_scopes>&redirect_uri=<redirect_uri>

  • client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method.
  • data_source_scopes are the scopes returned by ListDataSources method.
  • redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.
See also

google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest definition in proto format

DataSource

static

Represents data source metadata. Metadata is sufficient to render UI and request proper OAuth tokens.

Properties

Parameter

name

string

Output only. Data source resource name.

dataSourceId

string

Data source id.

displayName

string

User friendly data source name.

description

string

User friendly data source description string.

clientId

string

Data source client id which should be used to receive refresh token. When not supplied, no offline credentials are populated for data transfer.

scopes

Array of string

Api auth scopes for which refresh token needs to be obtained. Only valid when client_id is specified. Ignored otherwise. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery

transferType

number

Deprecated. This field has no effect.

The number should be among the values of TransferType

supportsMultipleTransfers

boolean

Indicates whether the data source supports multiple transfers to different BigQuery targets.

updateDeadlineSeconds

number

The number of seconds to wait for an update from the data source before BigQuery marks the transfer as failed.

defaultSchedule

string

Default data transfer schedule. Examples of valid schedules include: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00.

supportsCustomSchedule

boolean

Specifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to true, user can override default schedule.

parameters

Array of Object

Data source parameters.

This object should have the same structure as DataSourceParameter

helpUrl

string

Url for the help document for this data source.

authorizationType

number

Indicates the type of authorization.

The number should be among the values of AuthorizationType

dataRefreshType

number

Specifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.

The number should be among the values of DataRefreshType

defaultDataRefreshWindowDays

number

Default data refresh window on days. Only meaningful when data_refresh_type = SLIDING_WINDOW.

manualRunsDisabled

boolean

Disables backfilling and manual run scheduling for the data source.

minimumScheduleInterval

Object

The minimum interval for scheduler to schedule runs.

This object should have the same structure as Duration

See also

google.cloud.bigquery.datatransfer.v1.DataSource definition in proto format

DataSourceParameter

static

Represents a data source parameter with validation rules, so that parameters can be rendered in the UI. These parameters are given to us by supported data sources, and include all needed information for rendering and validation. Thus, whoever uses this api can decide to generate either generic ui, or custom data source specific forms.

Properties

Parameter

paramId

string

Parameter identifier.

displayName

string

Parameter display name in the user interface.

description

string

Parameter description.

type

number

Parameter type.

The number should be among the values of Type

required

boolean

Is parameter required.

repeated

boolean

Can parameter have multiple values.

validationRegex

string

Regular expression which can be used for parameter validation.

allowedValues

Array of string

All possible values for the parameter.

minValue

Object

For integer and double values specifies minimum allowed value.

This object should have the same structure as DoubleValue

maxValue

Object

For integer and double values specifies maxminum allowed value.

This object should have the same structure as DoubleValue

fields

Array of Object

When parameter is a record, describes child fields.

This object should have the same structure as DataSourceParameter

validationDescription

string

Description of the requirements for this field, in case the user input does not fulfill the regex pattern or min/max values.

validationHelpUrl

string

URL to a help document to further explain the naming requirements.

immutable

boolean

Cannot be changed after initial creation.

recurse

boolean

If set to true, schema should be taken from the parent with the same parameter_id. Only applicable when parameter type is RECORD.

See also

google.cloud.bigquery.datatransfer.v1.DataSourceParameter definition in proto format

DeleteTransferConfigRequest

static

A request to delete data transfer information. All associated transfer runs and log messages will be deleted as well.

Property

Parameter

name

string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}

See also

google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest definition in proto format

DeleteTransferRunRequest

static

A request to delete data transfer run information.

Property

Parameter

name

string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}

See also

google.cloud.bigquery.datatransfer.v1.DeleteTransferRunRequest definition in proto format

GetDataSourceRequest

static

A request to get data source info.

Property

Parameter

name

string

The field will contain name of the resource requested, for example: projects/{project_id}/dataSources/{data_source_id}

See also

google.cloud.bigquery.datatransfer.v1.GetDataSourceRequest definition in proto format

GetTransferConfigRequest

static

A request to get data transfer information.

Property

Parameter

name

string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}

See also

google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest definition in proto format

GetTransferRunRequest

static

A request to get data transfer run information.

Property

Parameter

name

string

The field will contain name of the resource requested, for example: projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}

See also

google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest definition in proto format

ListDataSourcesRequest

static

Request to list supported data sources and their data transfer settings.

Properties

Parameter

parent

string

The BigQuery project id for which data sources should be returned. Must be in the form: projects/{project_id}

pageToken

string

Pagination token, which can be used to request a specific page of ListDataSourcesRequest list results. For multiple-page results, ListDataSourcesResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.

pageSize

number

Page size. The default page size is the maximum value of 1000 results.

See also

google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest definition in proto format

ListDataSourcesResponse

static

Returns list of supported data sources and their metadata.

Properties

Parameter

dataSources

Array of Object

List of supported data sources and their transfer settings.

This object should have the same structure as DataSource

nextPageToken

string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListDataSourcesRequest.page_token to request the next page of list results.

See also

google.cloud.bigquery.datatransfer.v1.ListDataSourcesResponse definition in proto format

ListTransferConfigsRequest

static

A request to list data transfers configured for a BigQuery project.

Properties

Parameter

parent

string

The BigQuery project id for which data sources should be returned: projects/{project_id}.

dataSourceIds

Array of string

When specified, only configurations of requested data sources are returned.

pageToken

string

Pagination token, which can be used to request a specific page of ListTransfersRequest list results. For multiple-page results, ListTransfersResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.

pageSize

number

Page size. The default page size is the maximum value of 1000 results.

See also

google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest definition in proto format

ListTransferConfigsResponse

static

The returned list of pipelines in the project.

Properties

Parameter

transferConfigs

Array of Object

Output only. The stored pipeline transfer configurations.

This object should have the same structure as TransferConfig

nextPageToken

string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListTransferConfigsRequest.page_token to request the next page of list results.

See also

google.cloud.bigquery.datatransfer.v1.ListTransferConfigsResponse definition in proto format

ListTransferLogsRequest

static

A request to get user facing log messages associated with data transfer run.

Properties

Parameter

parent

string

Transfer run name in the form: projects/{project_id}/transferConfigs/{config_Id}/runs/{run_id}.

pageToken

string

Pagination token, which can be used to request a specific page of ListTransferLogsRequest list results. For multiple-page results, ListTransferLogsResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.

pageSize

number

Page size. The default page size is the maximum value of 1000 results.

messageTypes

Array of number

Message types to return. If not populated - INFO, WARNING and ERROR messages are returned.

The number should be among the values of MessageSeverity

See also

google.cloud.bigquery.datatransfer.v1.ListTransferLogsRequest definition in proto format

ListTransferLogsResponse

static

The returned list transfer run messages.

Properties

Parameter

transferMessages

Array of Object

Output only. The stored pipeline transfer messages.

This object should have the same structure as TransferMessage

nextPageToken

string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the GetTransferRunLogRequest.page_token to request the next page of list results.

See also

google.cloud.bigquery.datatransfer.v1.ListTransferLogsResponse definition in proto format

ListTransferRunsRequest

static

A request to list data transfer runs. UI can use this method to show/filter specific data transfer runs. The data source can use this method to request all scheduled transfer runs.

Properties

Parameter

parent

string

Name of transfer configuration for which transfer runs should be retrieved. Format of transfer configuration resource name is: projects/{project_id}/transferConfigs/{config_id}.

states

Array of number

When specified, only transfer runs with requested states are returned.

The number should be among the values of TransferState

pageToken

string

Pagination token, which can be used to request a specific page of ListTransferRunsRequest list results. For multiple-page results, ListTransferRunsResponse outputs a next_page token, which can be used as the page_token value to request the next page of list results.

pageSize

number

Page size. The default page size is the maximum value of 1000 results.

runAttempt

number

Indicates how run attempts are to be pulled.

The number should be among the values of RunAttempt

See also

google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest definition in proto format

ListTransferRunsResponse

static

The returned list of pipelines in the project.

Properties

Parameter

transferRuns

Array of Object

Output only. The stored pipeline transfer runs.

This object should have the same structure as TransferRun

nextPageToken

string

Output only. The next-pagination token. For multiple-page list results, this token can be used as the ListTransferRunsRequest.page_token to request the next page of list results.

See also

google.cloud.bigquery.datatransfer.v1.ListTransferRunsResponse definition in proto format

ScheduleTransferRunsRequest

static

A request to schedule transfer runs for a time range.

Properties

Parameter

parent

string

Transfer configuration name in the form: projects/{project_id}/transferConfigs/{config_id}.

startTime

Object

Start time of the range of transfer runs. For example, "2017-05-25T00:00:00+00:00".

This object should have the same structure as Timestamp

endTime

Object

End time of the range of transfer runs. For example, "2017-05-30T00:00:00+00:00".

This object should have the same structure as Timestamp

See also

google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest definition in proto format

ScheduleTransferRunsResponse

static

A response to schedule transfer runs for a time range.

Property

Parameter

runs

Array of Object

The transfer runs that were scheduled.

This object should have the same structure as TransferRun

See also

google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse definition in proto format

TransferConfig

static

Represents a data transfer configuration. A transfer configuration contains all metadata needed to perform a data transfer. For example, destination_dataset_id specifies where data should be stored. When a new transfer configuration is created, the specified destination_dataset_id is created when needed and shared with the appropriate data source service account.

Properties

Parameter

name

string

The resource name of the transfer config. Transfer config names have the form projects/{project_id}/transferConfigs/{config_id}. Where config_id is usually a uuid, even though it is not guaranteed or required. The name is ignored when creating a transfer config.

destinationDatasetId

string

The BigQuery target dataset id.

displayName

string

User specified display name for the data transfer.

dataSourceId

string

Data source id. Cannot be changed once data transfer is created.

params

Object

Data transfer specific parameters.

This object should have the same structure as Struct

schedule

string

Data transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: the granularity should be at least 8 hours, or less frequent.

dataRefreshWindowDays

number

The number of days to look back to automatically refresh the data. For example, if data_refresh_window_days = 10, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.

disabled

boolean

Is this config disabled. When set to true, no runs are scheduled for a given transfer.

updateTime

Object

Output only. Data transfer modification time. Ignored by server on input.

This object should have the same structure as Timestamp

nextRunTime

Object

Output only. Next time when data transfer will run.

This object should have the same structure as Timestamp

state

number

Output only. State of the most recently updated transfer run.

The number should be among the values of TransferState

userId

number

Output only. Unique ID of the user on whose behalf transfer is done. Applicable only to data sources that do not support service accounts. When set to 0, the data source service account credentials are used. May be negative. Note, that this identifier is not stable. It may change over time even for the same user.

datasetRegion

string

Output only. Region in which BigQuery dataset is located.

See also

google.cloud.bigquery.datatransfer.v1.TransferConfig definition in proto format

TransferMessage

static

Represents a user facing message for a particular data transfer run.

Properties

Parameter

messageTime

Object

Time when message was logged.

This object should have the same structure as Timestamp

severity

number

Message severity.

The number should be among the values of MessageSeverity

messageText

string

Message text.

See also

google.cloud.bigquery.datatransfer.v1.TransferMessage definition in proto format

TransferRun

static

Represents a data transfer run.

Properties

Parameter

name

string

The resource name of the transfer run. Transfer run names have the form projects/{project_id}/locations/{location}/transferConfigs/{config_id}/runs/{run_id}. The name is ignored when creating a transfer run.

scheduleTime

Object

Minimum time after which a transfer run can be started.

This object should have the same structure as Timestamp

runTime

Object

For batch transfer runs, specifies the date and time that data should be ingested.

This object should have the same structure as Timestamp

errorStatus

Object

Status of the transfer run.

This object should have the same structure as Status

startTime

Object

Output only. Time when transfer run was started. Parameter ignored by server for input requests.

This object should have the same structure as Timestamp

endTime

Object

Output only. Time when transfer run ended. Parameter ignored by server for input requests.

This object should have the same structure as Timestamp

updateTime

Object

Output only. Last time the data transfer run state was updated.

This object should have the same structure as Timestamp

params

Object

Output only. Data transfer specific parameters.

This object should have the same structure as Struct

destinationDatasetId

string

Output only. The BigQuery target dataset id.

dataSourceId

string

Output only. Data source id.

state

number

Data transfer run state. Ignored for input requests.

The number should be among the values of TransferState

userId

number

Output only. Unique ID of the user on whose behalf transfer is done. Applicable only to data sources that do not support service accounts. When set to 0, the data source service account credentials are used. May be negative. Note, that this identifier is not stable. It may change over time even for the same user.

schedule

string

Output only. Describes the schedule of this transfer run if it was created as part of a regular schedule. For batch transfer runs that are scheduled manually, this is empty. NOTE: the system might choose to delay the schedule depending on the current load, so schedule_time doesn't always matches this.

See also

google.cloud.bigquery.datatransfer.v1.TransferRun definition in proto format

UpdateTransferConfigRequest

static

A request to update a transfer configuration. To update the user id of the transfer configuration, an authorization code needs to be provided.

Properties

Parameter

transferConfig

Object

Data transfer configuration to create.

This object should have the same structure as TransferConfig

authorizationCode

string

Optional OAuth2 authorization code to use with this transfer configuration. If it is provided, the transfer configuration will be associated with the authorizing user. In order to obtain authorization_code, please make a request to https:// www.gstatic.com/bigquerydatatransfer/oauthz/auth?client_id= &scope=<data_source_scopes>&redirect_uri=<redirect_uri>

  • client_id should be OAuth client_id of BigQuery DTS API for the given data source returned by ListDataSources method.
  • data_source_scopes are the scopes returned by ListDataSources method.
  • redirect_uri is an optional parameter. If not specified, then authorization code is posted to the opener of authorization flow window. Otherwise it will be sent to the redirect uri. A special value of urn:ietf:wg:oauth:2.0:oob means that authorization code should be returned in the title bar of the browser, with the page text prompting the user to copy the code and paste it in the application.

updateMask

Object

Required list of fields to be updated in this request.

This object should have the same structure as FieldMask

See also

google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest definition in proto format