This document describes the methods, properties, and configuration options of Dataform core. You can use Dataform core in SQLX and JavaScript files.
assert()
assert
|
(name: string, query?: AContextable
|
Adds a Dataform assertion the compiled graph. Available only in the |
Example:
// definitions/file.js
assert("name").query(ctx => "select 1");
CommonContext
Context methods are available when evaluating contextable SQL code,
such as within SQLX files, or when using a
Contextable
argument with Dataform core.
database
|
() => string
|
Returns the database of this dataset, if applicable. | |
name
|
() => string
|
Returns the name of this table. | |
ref
|
(ref: Resolvable | string[], rest: string[]) => string
|
References another action, adding it as a dependency to this action,
returning valid SQL to be used in a from expression.
This function can be called with a
This function can also be called using individual arguments for the
${ref("database", "schema", "name")}
${ref("schema", "name")}
${ref("name")}
|
|
resolve
|
(ref: Resolvable | string[], rest: string[]) => string
|
Similar to ref , but it does not add the referenced
action as a dependency to this action.
|
|
self
|
() => string
|
Equivalent to resolve(name()) .
Returns a valid SQL string that can be used to reference the table produced by this action. |
|
schema
|
() => string
|
Returns the schema of this dataset. |
Contextable
Contextable arguments can either pass a plain value for their generic type
T
or a function that is called with the context object for this
type of operation.
T | (ctx: Context) => T
Dataform
Global variable that contains the IProjectConfig object.
Required for getting IProjectConfig
properties, for example:
dataform.projectConfig.vars.myVariableName === "myVariableValue"
declare()
declare
|
(dataset: dataform.ITarget)
|
Declares the dataset as a Dataform data source. Available only in the |
Example:
// definitions/file.js
declare({name: "a-declaration"})
IActionConfig
Defines Dataform tags and dependencies applied to a SQL workflow action.
tags
|
string[]
|
A list of user-defined tags with which the action should be labeled. |
|
dependencies
|
Resolvable| Resolvable[]
|
Dependencies of the action. |
|
disabled
|
boolean
|
If set to true, this action won't be executed. However, the action can still be depended upon. Useful for temporarily turning off broken actions. |
IAssertionConfig
Configuration options for assertion
action types.
database
|
string
|
The database (Google Cloud project ID) in which to create the corresponding view for this assertion. | |
description
|
string
|
A description of this assertion. | |
disabled
|
boolean
|
If set to true , this action is not executed.
The action can still be depended upon. Useful for temporarily
turning off broken actions.
|
|
hermetic
|
boolean
|
Declares whether this action is hermetic. An action is hermetic
if all of its dependencies are explicitly declared.
If this action depends on data from a source which is not declared
as a dependency, then set |
|
schema
|
string
|
The schema (BigQuery dataset) in which to create the corresponding view for this assertion. | |
tags
|
string[]
|
A list of user-defined tags applied to this action. |
IBigQueryOptions
BigQuery-specific warehouse options.
additionalOptions
|
|
Key-value pairs for the table, view,
and materialized view options.
Some options, for example,
String values must be encapsulated in double-quotes, for example:
If the option name contains special characters, encapsulate the name in quotes,
for example: |
|
clusterBy
|
string[]
|
The keys by which to cluster partitions by. | |
labels
|
|
Key-value pairs for BigQuery labels.
If the label name contains special characters, for example, hyphens,
then quote its name, for example, |
|
partitionBy
|
string
|
The key by which to partition the table. Typically the name of a timestamp or the date column. | |
partitionExpirationDays
|
number
|
The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. | |
requirePartitionFilter
|
boolean
|
Declares whether the partitioned table requires a WHERE
clause predicate filter that filters the partitioning column.
|
|
updatePartitionFilter
|
string
|
SQL-based filter for when incremental updates are applied. |
IColumnsDescriptor
Describes columns in a table.
{ [name]: string | IRecordDescriptor }
IDeclarationConfig
Configuration options for declaration
action types.
columns
|
IColumnsDescriptor
|
A description of columns within the table. | |
database
|
string
|
The database (Google Cloud project ID) in which to create the source table exists. | |
description
|
string
|
A description of the table. | |
schema
|
string
|
The schema (BigQuery dataset) in which the source table exists. |
IDependenciesConfig
Defines dependencies of a SQL workflow action.
dependencies
|
Resolvable| Resolvable[]
|
One or more explicit dependencies for this action. Dependency actions
will run before dependent actions.
Typically this would remain unset, because most dependencies are declared
as a by-product of using the |
|
hermetic
|
boolean
|
Declares whether or not this action is hermetic. An action is hermetic if
all of its dependencies are explicitly declared. If this action depends on
data from a source which has not been declared as a dependency, then
|
IDocumentableConfig
Defines descriptions of a dataset and its columns.
columns
|
IColumnsDescriptor
|
A description of columns within the dataset. |
|
description
|
string
|
A description of the dataset. |
INamedConfig
Defines the type and name of a SQL workflow action.
type
|
string
|
The type of the action. |
|
name
|
string
|
The name of the action. |
IOperationConfig
Configuration options for operations
action types.
columns
|
IColumnsDescriptor
|
A description of columns within the table. | |
database
|
string
|
The database (Google Cloud project ID) in which to create the output of this action. | |
description
|
string
|
A description of the table. | |
disabled
|
boolean
|
If set to true , this action is not executed.
The action can still be depended upon.
Useful for temporarily turning off broken actions.
|
|
hasOutput
|
boolean
|
Declares that this operations action creates a table
that is referenceable using the ref function.
If set to create or replace table ${self()} as select ...
|
|
hermetic
|
boolean
|
Declares whether this action is hermetic. An action is hermetic
if all of its dependencies are explicitly declared.
If this action depends on data from a source which is not declared
as a dependency, then set |
|
schema
|
string
|
The schema (BigQuery dataset) in which to create the output of this action. | |
tags
|
string[]
|
A list of user-defined tags applied to this action. |
IProjectConfig
Contains compilation settings of a Dataform repository.
defaultDatabase
|
string
|
Required. The default database (Google Cloud project ID). | |
defaultSchema
|
string
|
Required. The default schema (BigQuery dataset ID). | |
defaultLocation
|
string
|
Required. The default BigQuery location to use. For more information on BigQuery locations, see https://cloud.google.com/bigquery/docs/locations. | |
assertionSchema
|
string
|
Required. The default schema (BigQuery dataset ID) for assertions. | |
vars
|
map (key: string, value: string)
|
Optional. User-defined variables that are made available
to project code during compilation.
An object containing a list of "key": value pairs.
Example: { "name": "wrench", "mass": "1.3kg", "count": "3" } .
|
|
databaseSuffix
|
string
|
Optional. The suffix that should be appended to all database (Google Cloud project ID) names. | |
schemaSuffix
|
string
|
Optional. The suffix that should be appended to all schema (BigQuery dataset ID) names. | |
tablePrefix
|
string
|
Optional. The prefix that should be prepended to all table names. | |
warehouse
|
string
|
Required. Must be set to bigquery .
|
You can set IProjectConfig
properties in workflow settings
at the repository level.
You can override the defaultSchema
and defaultDatabase
properties for
individual tables.
You can access all IProjectConfig
properties in a SQL SELECT
statement
in a SQLX or JavaScript file.
The following code sample shows the myVariableName
custom compilation variable
set in workflow settings with the
projectConfig.vars
property,
accessed in a SELECT
statement in a SQLX file:
config { type: "view" }
SELECT ${when(
dataform.projectConfig.vars.myVariableName === "myVariableValue",
"myVariableName is set to myVariableValue!",
"myVariableName is not set to myVariableValue!"
)}
For more information about overriding project configuration settings
for individual compilation results, see the
projects.locations.repositories.compilationResults#CodeCompilationConfig
REST resource in
Dataform API.
IRecordDescriptor
Describes a struct, object or record in a table that has nested columns.
bigqueryPolicyTags
|
string | string[]
|
Full identifiers of BigQuery policy tags applied
to this column. A full identifier of a BigQuery policy tag
includes the project name, location, and taxonomy,
For example: |
|
columns
|
IColumnsDescriptor
|
A description of columns within the struct, object, or record. | |
description
|
string
|
A description of the struct, object, or record. |
ITableAssertions
Options for creating assertions as part of a table definition.
nonNull
|
string | string[]
|
Column(s) which can never be NULL .
If set, the corresponding assertion fails if any row contains
|
|
rowConditions
|
string[]
|
General condition(s) which should hold true for all rows in the table.
If set, the corresponding assertion fails if any row violates any of these condition(s). |
|
uniqueKey
|
string | string[]
|
Column(s) which constitute the unique key index of the table.
If set, the resulting assertion fails if there is more than one row in the table with the same values for all of these column(s). |
|
uniqueKeys
|
[]
|
Combinations of column(s), each of which constitutes
a unique key index of the table.
If set, the resulting assertion fails if there is more than one row in the table with the same values for all of the column(s) in the unique key(s). |
ITableConfig
Configuration options for table
actions, including
table
, view
and incremental
table types.
Extends IActionConfig, IDependenciesConfig, IDocumentableConfig, INamedConfig, and ITargetableConfig.
assertions
|
ITableAssertions
|
Assertions to be run on the table.
If configured, relevant assertions are automatically created and run as a dependency of this table. |
|
bigquery
|
IBigQueryOptions
|
BigQuery-specific warehouse options. | |
columns
|
IColumnsDescriptor
|
A description of columns within the table. | |
database
|
string
|
The database (Google Cloud project ID) in which to create the output of this action. | |
description
|
string
|
A description of the table. | |
disabled
|
boolean
|
If set to true , this action is not executed.
The action can still be depended upon.
Useful for temporarily turning off broken actions.
|
|
hermetic
|
boolean
|
Declares whether this action is hermetic. An action is hermetic if
all of its dependencies are explicitly declared.
If this action depends on data from a source which is not declared
as a dependency, then set |
|
materialized
|
boolean
|
Only valid when the table type is view .
If set to true, a materialized view will be created. |
|
protected
|
boolean
|
Only allowed for the incremental table type.
If set to true, running this action ignores the |
|
schema
|
string
|
The schema (BigQuery dataset) in which to create the output of this action. | |
tags
|
string[]
|
A list of user-defined tags applied to this action. | |
type
|
TableType
|
The type of the table. | |
uniqueKey
|
string[]
|
Unique keys for merge criteria for incremental tables.
If configured, records with matching unique key(s) are updated instead of new rows being inserted. |
ITableContext
Context methods are available when evaluating contextable SQL code,
such as within SQLX files, or when using a Contextable
argument with Dataform core.
incremental
|
() => boolean
|
Returns true when the current context indicates that the table will be built incrementally. | |
name
|
() => string
|
Returns the fully qualified name of this table. | |
ref
|
(ref: Resolvable | string[], rest: string[]) => string
|
References another action, adding it as a dependency to this action,
returning valid SQL to be used in a from expression.
This function can be called with a ${ref({ name: "name", schema: "schema", database: "database" })}
This function can also be called using individual arguments for the
When only two values are provided, the default database is used and the values
are interpreted as When only one value is provided, the default database schema is used, with the provided value interpreted as `"name"`. ${ref("database", "schema", "name")}
${ref("schema", "name")}
${ref("name")}
|
|
resolve
|
(ref: Resolvable | string[], rest: string[]) => string
|
Similar to ref , but instead of adding a dependency,
it resolves the provided reference so that it can be used in SQL,
for example, in a `from` expression.
|
|
self
|
() => string
|
Equivalent to resolve(name()) .
Returns a valid SQL string that can be used to reference the table produced by this action. |
|
when
|
(cond: boolean, trueCase: string, falseCase: string) => string
|
Shorthand for an if condition.
Equivalent to cond ? trueCase : falseCase .
|
ITarget
A reference to a table within BigQuery.
database
|
string
|
name
|
string
|
schema
|
string
|
ITargetableConfig
Defines the target database and schema of a SQL workflow action.
database
|
string
|
The database in which the output of this action should be created. Must be set to BigQuery. |
|
schema
|
string
|
The schema in which the output of this action should be created. |
operate()
operate
|
(name: string, queries?: Contextable
|
Defines a SQL operation. Available only in the |
Example:
// definitions/file.js
operate("an-operation", ["SELECT 1", "SELECT 2"])
publish()
publish
|
(name: string, queryOrConfig?: Contextable
|
Creates a table or view. Available only in the |
Example:
// definitions/file.js
publish("published-table", {
type: "table",
dependencies: ["a-declaration"],
}).query(ctx => "SELECT 1 AS test");
Resolvable
A resolvable can be either the name of a table as string
,
or the object that describes the full path to the relation.
string | ITarget
TableType
Supported types of table actions.
Tables of type view
will be created as views.
Tables of type table
will be created as tables.
Tables of type incremental
must include a where
clause.
For more information, see
Configure incremental tables.