REST Resource: projects.dataSources

Resource: DataSource

Represents data source metadata. Metadata is sufficient to render UI and request proper OAuth tokens.

JSON representation
{
  "name": string,
  "dataSourceId": string,
  "displayName": string,
  "description": string,
  "clientId": string,
  "scopes": [
    string
  ],
  "transferType": enum(TransferType),
  "supportsMultipleTransfers": boolean,
  "updateDeadlineSeconds": number,
  "defaultSchedule": string,
  "supportsCustomSchedule": boolean,
  "parameters": [
    {
      object(DataSourceParameter)
    }
  ],
  "helpUrl": string,
  "authorizationType": enum(AuthorizationType),
  "dataRefreshType": enum(DataRefreshType),
  "defaultDataRefreshWindowDays": number,
  "manualRunsDisabled": boolean,
  "minimumScheduleInterval": string,
}
Fields
name

string

Data source resource name.

dataSourceId

string

Data source id.

displayName

string

User friendly data source name.

description

string

User friendly data source description string.

clientId

string

Data source client id which should be used to receive refresh token. When not supplied, no offline credentials are populated for data transfer.

scopes[]

string

Api auth scopes for which refresh token needs to be obtained. Only valid when clientId is specified. Ignored otherwise. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery

transferType

enum(TransferType)

Transfer type. Currently supports only batch transfers, which are transfers that use the BigQuery batch APIs (load or query) to ingest the data.

supportsMultipleTransfers

boolean

Indicates whether the data source supports multiple transfers to different BigQuery targets.

updateDeadlineSeconds

number

The number of seconds to wait for an update from the data source before BigQuery marks the transfer as failed.

defaultSchedule

string

Default data transfer schedule. Examples of valid schedules include: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00.

supportsCustomSchedule

boolean

Specifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to true, user can override default schedule.

parameters[]

object(DataSourceParameter)

Data source parameters.

helpUrl

string

Url for the help document for this data source.

authorizationType

enum(AuthorizationType)

Indicates the type of authorization.

dataRefreshType

enum(DataRefreshType)

Specifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.

defaultDataRefreshWindowDays

number

Default data refresh window on days. Only meaningful when dataRefreshType = SLIDING_WINDOW.

manualRunsDisabled

boolean

Disables backfilling and manual run scheduling for the data source.

minimumScheduleInterval

string (Duration format)

The minimum interval between two consecutive scheduled runs.

A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".

TransferType

Represents data transfer type.

Enums
TRANSFER_TYPE_UNSPECIFIED Invalid or Unknown transfer type placeholder.
BATCH Batch data transfer.
STREAMING Streaming data transfer. Streaming data source currently doesn't support multiple transfer configs per project.

DataSourceParameter

Represents a data source parameter with validation rules, so that parameters can be rendered in the UI. These parameters are given to us by supported data sources, and include all needed information for rendering and validation. Thus, whoever uses this api can decide to generate either generic ui, or custom data source specific forms.

JSON representation
{
  "paramId": string,
  "displayName": string,
  "description": string,
  "type": enum(Type),
  "required": boolean,
  "repeated": boolean,
  "validationRegex": string,
  "allowedValues": [
    string
  ],
  "minValue": number,
  "maxValue": number,
  "fields": [
    {
      object(DataSourceParameter)
    }
  ],
  "validationDescription": string,
  "validationHelpUrl": string,
  "immutable": boolean,
  "recurse": boolean,
}
Fields
paramId

string

Parameter identifier.

displayName

string

Parameter display name in the user interface.

description

string

Parameter description.

type

enum(Type)

Parameter type.

required

boolean

Is parameter required.

repeated

boolean

Can parameter have multiple values.

validationRegex

string

Regular expression which can be used for parameter validation.

allowedValues[]

string

All possible values for the parameter.

minValue

number

For integer and double values specifies minimum allowed value.

maxValue

number

For integer and double values specifies maxminum allowed value.

fields[]

object(DataSourceParameter)

When parameter is a record, describes child fields.

validationDescription

string

Description of the requirements for this field, in case the user input does not fulfill the regex pattern or min/max values.

validationHelpUrl

string

URL to a help document to further explain the naming requirements.

immutable

boolean

Cannot be changed after initial creation.

recurse

boolean

If set to true, schema should be taken from the parent with the same parameter_id. Only applicable when parameter type is RECORD.

Type

Parameter type.

Enums
TYPE_UNSPECIFIED Type unspecified.
STRING String parameter.
INTEGER Integer parameter (64-bits). Will be serialized to json as string.
DOUBLE Double precision floating point parameter.
BOOLEAN Boolean parameter.
RECORD Record parameter.
PLUS_PAGE Page ID for a Google+ Page.

AuthorizationType

The type of authorization needed for this data source.

Enums
AUTHORIZATION_TYPE_UNSPECIFIED Type unspecified.
AUTHORIZATION_CODE Use OAuth 2 authorization codes that can be exchanged for a refresh token on the backend.
GOOGLE_PLUS_AUTHORIZATION_CODE Return an authorization code for a given Google+ page that can then be exchanged for a refresh token on the backend.

DataRefreshType

Represents how the data source supports data auto refresh.

Enums
DATA_REFRESH_TYPE_UNSPECIFIED The data source won't support data auto refresh, which is default value.
SLIDING_WINDOW The data source supports data auto refresh, and runs will be scheduled for the past few days. Does not allow custom values to be set for each transfer config.
CUSTOM_SLIDING_WINDOW The data source supports data auto refresh, and runs will be scheduled for the past few days. Allows custom values to be set for each transfer config.

Methods

checkValidCreds

Returns true if valid credentials exist for the given data source and requesting user.

get

Retrieves a supported data source and returns its settings, which can be used for UI rendering.

list

Lists supported data sources and returns their settings, which can be used for UI rendering.

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...