This document shows you how to resolve issues with Dataform.
Access to BigQuery is denied
The following error occurs when you trigger a workflow invocation before granting Dataform access to BigQuery:
Access Denied: Project PROJECT_ID: User does not have bigquery.jobs.create permission in project PROJECT_ID.
To resolve this error, grant Dataform access to BigQuery.
Access token for a remote repository is rejected
The following error occurs when your authentication token for a connected third-party repository does not have access to that repository:
The access token for remote repository REPOSITORY_NAME was rejected
To resolve this error, check required permissions in your Git provider and update the Secret Manager authentication token accordingly. For more information about authenticating third-party Git repositories in Dataform, see Connect to a third-party Git repository.
BigQuery quota is exceeded
The following error occurs when the number of API requests that Dataform sends to BigQuery exceeds the BigQuery quota:
Quota exceeded: Your user_method exceeded quota for concurrent api requests
per user per method.
To resolve this error, reduce the number of parallel queries to less than 250 in the following ways:
- In Dataform, categorize actions with tags, and execute only selected tags at a time.
- In Dataform, introduce dependencies between actions.
- In Dataform, divide executions of actions between different Google Cloud projects.
For instructions to resolve this error in BigQuery, see Troubleshoot quota and limit errors.
BigQuery query concurrency limit is exceeded
The following error occurs when the number of concurrent queries executed to BigQuery exceeds the BigQuery query concurrency limit:
Exceeded rate limits: too many concurrent queries for this project_and_region
To resolve this error, reduce the number of parallel queries to less than 250 in the following ways:
- In Dataform, categorize actions with tags, and execute only selected tags at a time.
- In Dataform, introduce dependencies between actions.
- In Dataform, divide executions of actions between different Google Cloud projects.
For instructions to resolve this error in BigQuery, see Troubleshoot quota and limit errors.
BigQuery workflow invocation errors
The following errors occur during execution of a SQL workflow to BigQuery:
- Workflow invocation errors that begin with BigQuery error messages.
To resolve these errors, see BigQuery error messages.
Conflicting includeDependentAssertions
properties
The following error occurs during compilation when the
includeDependentAssertions
parameter is set for the same
action with different values within one file:
Conflicting "includeDependentAssertions" properties are not allowed. Dependency
dependencyName has different values set for this property.
To resolve this error, edit the file and remove conflicting repetitions of the
includeDependentAssertions
parameter.
For more information about using the includeDependentAssertions
parameter
to set assertions as dependencies, see
Set assertions of a selected action as dependencies.
Compilation is failing
The following errors occur during compilation due to size or number of compiled queries:
Compilation timed out. Reduce the complexity of your project to ensure it can compile within limits.
Compilation exceeded its allowed heap memory limits. Reduce the complexity of your project to ensure it can compile within limits.
Compilation exceeded its allowed ArrayBuffer or string memory limits. Reduce the complexity of your project to ensure it can compile within limits.
To resolve these errors, follow these steps:
- Update Dataform core to the latest version.
- Inspect your SQL workflow to identify and reduce inefficiencies.
- Reduce the size of SQL queries.
Reduce the amount of JavaScript operations in memory, for example:
config { config {type: "table" }} js { const tooBig = new Uint8Array(110_000_000); } SELECT ...
For more information about Dataform compilation resources limits, see Quotas and limits.
@dataform/core
dependency errors
The following errors occur during compilation if the dataform-core
dependency
in package.json
is outdated:
Failed to resolve @dataform/core
@dataform/core version should be X.X.X or newer
The @dataform/core
dependency is required in the package.json
. When you
initialize the first workspace in your repository, Dataform
automatically populates package.json
with the current version of
@dataform/core
. You need to update @dataform/core
to the latest version as
it comes out.
To resolve these errors, update @dataform/core
to the latest
version.
Failure to resolve dataform.json
The following error occurs when you initialize a Dataform workspace, but the initialization process fails to install all packages:
Uncaught Error: Failed to resolve dataform.json
To resolve this error, in your workspace, open package.json
and click Install packages.
Failure to resolve workflow_settings.yaml
The following error occurs when you initialize a Dataform workspace, but the initialization process fails to install all packages:
Uncaught Error: Failed to resolve workflow_settings.yaml
To resolve this error, in your workspace, open workflow_settings.yaml
and
click Install packages.
git+
package targets are not supported
The following error occurs when you define packages in package.json
with
targets prefixed with git+
:
'git+' prefixed package targets are not currently supported. However,
in most cases they can be used via a '.tar.gz' suffixed target instead.
Dataform does not support package targets prefixed with git+
.
To resolve this error, generate a tar.gz
URL of the package and update the
package target in package.json
. For more information about installing packages
in Dataform, see Install a package in
Dataform.
Package installation times out
The following error occurs when the size of packages defined in package.json
exceeds the maximum size of NPM
dependencies:
API request error: Package installation timed out
To resolve this error, remove redundant packages from package.json
. Make sure
the package.json
file does not contain @dataform/cli
, and that the total
size of defined NPM dependencies does not exceed 200Mb.
If your release configurations
reference Git commitishes, make sure that the package.json
files at their
targets are valid.
Private package registry cannot be reached
The following error occurs when Dataform authentication for a private package expires:
Permission denied when fetching one or more npm packages. Please verify that
private registry authentication details are valid for each npm registry
To resolve this error, verify that private registry authentication details are valid for each NPM registry. For more information, see Authenticate a private package in Dataform.
Remote repository cannot be reached
The following error occurs due to Git flakiness or when a branch name in the connected third-party Git repository contains a non-ascii or non-English character:
Remote repository REPOSITORY_NAME could not be reached.
To resolve this error, inspect branch names in the connected third-party repository. If a branch name contains a non-ascii or non-English character, delete the branch.
Remote repository cannot be reached: generic::invalid_argument
The following error occurs on the Release configurations details page when a scheduled release occasionally encounters a slow, flaky, or dropped GitHub, GitLab, or Bitbucket connection:
generic::invalid_argument: Remote repository 'REMOTE_REPOSITORY_URL' could not be reached.
There is no need to take any action. Unless GitHub, GitLab, or Bitbucket Cloud issues persist, the subsequent scheduled releases can be successful.
Secret for a remote repository cannot be reached
The following error occurs when your Dataform service account cannot access your Secret Manager secret for a connected third-party repository:
Dataform's service account is unable to reach the configured secret.
Make sure the secret exists and is shared with your Dataform service account:
SERVICE_ACCOUNT_ID.
To resolve this error, do the following:
- Verify that your Dataform service account has access to the secret.
- Exclude the secret from your VPC-SC service perimeter. Dataform does not support VPC-SC at this time.