Utilizzare l'interfaccia a riga di comando Dataform open source
Mantieni tutto organizzato con le raccolte
Salva e classifica i contenuti in base alle tue preferenze.
Questo documento mostra come utilizzare l'interfaccia a riga di comando (CLI) open source Dataform per sviluppare localmente i flussi di lavoro utilizzando il terminale.
Con la CLI Dataform open source, puoi inizializzare, compilare, testare ed eseguire Dataform Core localmente, al di fuori di Google Cloud.
La CLI Dataform supporta le credenziali predefinite dell'applicazione (ADC).
Con ADC, puoi rendere disponibili le credenziali per la tua applicazione in una serie
di ambienti, ad esempio sviluppo locale o produzione, senza dover
modificare il codice dell'applicazione. Per utilizzare ADC, devi prima
fornire le tue credenziali ad ADC.
Per installare Dataform CLI, esegui questo comando:
npm i -g @dataform/cli@^3.0.0-beta
Inizializza un progetto Dataform
Per inizializzare un nuovo progetto Dataform,
esegui questo comando nella directory del progetto:
dataform init . PROJECT_NAMEDEFAULT_LOCATION
Sostituisci quanto segue:
PROJECT_NAME: il nome del progetto.
DEFAULT_LOCATION (facoltativo): la posizione in cui vuoi che Dataform scriva i dati BigQuery. Se non è impostata,
Dataform determina la posizione in base ai set di dati a cui
fa riferimento la query SQL. Ecco come funziona:
Se la query fa riferimento a set di dati della stessa località,
Dataform utilizza quella località.
Se la query fa riferimento a set di dati di due o più località diverse,
si verifica un errore. Per informazioni dettagliate su questa limitazione, vedi
Replica dei set di dati tra regioni.
Se la query non fa riferimento a nessun set di dati, la località predefinita per Dataform è la multi-regione US. Per scegliere una località diversa, imposta la località predefinita. In alternativa, utilizza la
variabile di sistema @@location
nella query. Per ulteriori informazioni, consulta Specificare le posizioni.
Aggiorna Dataform Core
Per aggiornare il framework principale di Dataform, aggiorna dataformCoreVersion
nel file workflow_settings.yaml, quindi esegui di nuovo l'installazione di NPM:
npm i
Aggiorna la CLI Dataform
Per aggiornare lo strumento CLI Dataform, esegui questo comando:
npm i -g @dataform/cli@^3.0.0-beta.2
Crea un file delle credenziali
Dataform richiede un file delle credenziali per connettersi a servizi remoti
e creare il file .df-credentials.json sul disco.
Per creare il file delle credenziali:
Esegui questo comando:
dataform init-creds
Segui la procedura guidata init-creds che ti guida nella creazione del file delle credenziali.
Crea un progetto
Un progetto Dataform vuoto in Dataform core 3.0.0-beta.0
o versioni successive ha la seguente struttura:
project-dir
├── definitions
├── includes
└── workflow_settings.yaml
Per creare un progetto Dataform per il deployment degli asset in
BigQuery, esegui questo comando:
YOUR_GOOGLE_CLOUD_PROJECT_ID: il tuo ID progetto Google Cloud .
DEFAULT_LOCATION (facoltativo): la località in cui vuoi che Dataform scriva i dati BigQuery. Se non è impostata,
Dataform determina la posizione in base ai set di dati a cui
fa riferimento la query SQL. Ecco come funziona:
Se la query fa riferimento a set di dati della stessa località,
Dataform utilizza quella località.
Se la query fa riferimento a set di dati di due o più località diverse,
si verifica un errore. Per informazioni dettagliate su questa limitazione, vedi
Replica dei set di dati tra regioni.
Se la query non fa riferimento a nessun set di dati, la località predefinita per Dataform è la multi-regione US. Per scegliere una località diversa, imposta la località predefinita. In alternativa, utilizza la
variabile di sistema @@location
nella query. Per ulteriori informazioni, consulta Specificare le posizioni.
Clonare un progetto
Per clonare un progetto Dataform esistente da un repository Git di terze parti,
segui le istruzioni del tuo provider Git.
Una volta clonato il repository, esegui questo comando all'interno
della directory del repository clonato:
dataform install
Definire una tabella
Archivia le definizioni nella cartella definitions/.
[[["Facile da capire","easyToUnderstand","thumb-up"],["Il problema è stato risolto","solvedMyProblem","thumb-up"],["Altra","otherUp","thumb-up"]],[["Difficile da capire","hardToUnderstand","thumb-down"],["Informazioni o codice di esempio errati","incorrectInformationOrSampleCode","thumb-down"],["Mancano le informazioni o gli esempi di cui ho bisogno","missingTheInformationSamplesINeed","thumb-down"],["Problema di traduzione","translationIssue","thumb-down"],["Altra","otherDown","thumb-down"]],["Ultimo aggiornamento 2025-09-04 UTC."],[[["\u003cp\u003eThe Dataform CLI allows for local development of SQL workflows, enabling users to initialize, compile, test, and run Dataform core outside of Google Cloud.\u003c/p\u003e\n"],["\u003cp\u003eThe Dataform CLI supports Application Default Credentials (ADC) for managing credentials across different environments without code modification.\u003c/p\u003e\n"],["\u003cp\u003eDataform projects can be initialized, cloned, and managed locally, and the CLI provides commands to define tables, manual assertions, and custom SQL operations.\u003c/p\u003e\n"],["\u003cp\u003eUsers can compile code and view the output in real-time, either in the terminal or as a JSON object, including options for custom compilation variables.\u003c/p\u003e\n"],["\u003cp\u003eThe CLI facilitates code execution, dry runs, and full refreshes in BigQuery, with commands available to get help on specific command options.\u003c/p\u003e\n"]]],[],null,["This document shows you how to use the open-source Dataform command-line interface (CLI) to locally develop workflows by using the terminal.\n\n\u003cbr /\u003e\n\nWith the open-source Dataform CLI, you can initialize, compile, test,\nand run [Dataform core](/dataform/docs/overview#dataform-core) locally,\noutside of Google Cloud.\n\nThe Dataform CLI supports [Application Default Credentials (ADC)](/docs/authentication/application-default-credentials).\nWith ADC, you can make credentials available to your application in a variety\nof environments, such as local development or production, without needing to\nmodify your application code. To use ADC, you must first\n[provide your credentials](/docs/authentication/provide-credentials-adc) to ADC.\n| **Note:** As of [Dataform core `3.0.0.`](https://github.com/dataform-co/dataform/releases/tag/3.0.0), Dataform doesn't distribute a Docker image. You can build your own Docker image of Dataform, which you can use to run the equivalent of Dataform CLI commands. To build your own Docker image, see [Containerize an application](https://docs.docker.com/guides/workshop/02_our_app/) in the Docker documentation.\n\nBefore you begin\n\nBefore installing the Dataform CLI, install\n[NPM](https://www.npmjs.com/get-npm).\n\nInstall Dataform CLI\n\n- To install Dataform CLI, run the following command:\n\n npm i -g @dataform/cli@^3.0.0-beta\n\nInitialize a Dataform project\n\n- To initialize a new Dataform project,\n run the following command inside your project directory:\n\n dataform init . \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: the name of your project.\n - \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e (optional): the location where\n you want Dataform to write BigQuery data. If unset,\n Dataform determines the location based on the datasets that\n your SQL query references. This works as follows:\n\n - If your query references datasets from the same location, Dataform uses that location.\n - If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see [Cross-region dataset replication](/bigquery/docs/data-replication).\n - If your query doesn't reference any datasets, the default location for Dataform is the `US` multi-region. To choose a different location, set the default location. Alternatively, use the [`@@location` system variable](/bigquery/docs/reference/system-variables) in your query. For more information, see [Specify locations](/bigquery/docs/locations#specify_locations).\n\nUpdate Dataform core\n\n- To update the Dataform core framework, update the `dataformCoreVersion`\n in `workflow_settings.yaml` file, then re-run NPM install:\n\n npm i\n\nUpdate Dataform CLI\n\n- To update the Dataform CLI tool, run the following command:\n\n npm i -g @dataform/cli@^3.0.0-beta.2\n\nCreate a credentials file\n\nDataform requires a credentials file to connect to remote services\nand create the `.df-credentials.json` file on your disk.\n| **Warning:** If you use a source control system, don't commit the `df-credentials.json`. file to your repository to protect these access credentials. We recommend that you add the `.df-credentials.json` file to `.gitignore`.\n\nTo create the credentials file, follow these steps:\n\n1. Run the following command:\n\n dataform init-creds\n\n2. Follow the `init-creds` wizard that walks you through credentials file creation.\n\nCreate a project\n\nAn empty Dataform project in Dataform core `3.0.0-beta.0`\nor later has the following structure: \n\n project-dir\n ├── definitions\n ├── includes\n └── workflow_settings.yaml\n\n- To create a Dataform project to deploy assets to\n BigQuery, run the following command:\n\n dataform init \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e --default-database \u003cvar translate=\"no\"\u003eYOUR_GOOGLE_CLOUD_PROJECT_ID\u003c/var\u003e --default-location \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: the name of your project.\n - \u003cvar translate=\"no\"\u003eYOUR_GOOGLE_CLOUD_PROJECT_ID\u003c/var\u003e: your Google Cloud project ID.\n - \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e (optional): the location where you\n want Dataform to write BigQuery data. If unset,\n Dataform determines the location based on the datasets that\n your SQL query references. This works as follows:\n\n - If your query references datasets from the same location, Dataform uses that location.\n - If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see [Cross-region dataset replication](/bigquery/docs/data-replication).\n - If your query doesn't reference any datasets, the default location for Dataform is the `US` multi-region. To choose a different location, set the default location. Alternatively, use the [`@@location` system variable](/bigquery/docs/reference/system-variables) in your query. For more information, see [Specify locations](/bigquery/docs/locations#specify_locations).\n\nClone a project\n\nTo clone an existing Dataform project from a third-party Git repository,\nfollow the instructions from your Git provider.\n\n- Once the repository is cloned, run the following command inside\n the cloned repository directory:\n\n dataform install\n\nDefine a table\n\nStore definitions in the `definitions/` folder.\n\n- To define a table, run the following command:\n\n echo \"config { type: '\u003cvar translate=\"no\"\u003eTABLE_TYPE\u003c/var\u003e' } \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eTABLE_TYPE\u003c/var\u003e: the type of the table: `table`, `incremental`, or `view`.\n - \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e: a `SELECT` statement that defines the table.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the table definition file.\n\nThe following code sample defines a view in the `example` SQLX file. \n\n echo \"config { type: 'view' } SELECT 1 AS test\" \u003e definitions/example.sqlx\n\nDefine a manual assertion\n\nStore definitions in the `definitions/` folder.\n\n- To define a manual assertion, run the following command:\n\n echo \"config { type: 'assertion' } \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e: a `SELECT` statement that defines the assertion.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the custom SQL operation definition file.\n\nDefine a custom SQL operation\n\nStore definitions in the `definitions/` folder.\n\n- To define a custom SQL operation, run the following command:\n\n echo \"config { type: 'operations' } \u003cvar translate=\"no\"\u003eSQL_QUERY\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSQL_QUERY\u003c/var\u003e: your custom SQL operation.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the custom SQL operation definition file.\n\nView compilation output\n\nDataform compiles your code in real time.\n\n- To view the output of the compilation process in the terminal,\n run the following command:\n\n dataform compile\n\n- To view the output of the compilation process as a JSON object,\n run the following command:\n\n dataform compile --json\n\n- To view the output of the compilation with\n [custom compilation variables](/dataform/docs/configure-compilation#compilation-variables),\n run the following command:\n\n dataform compile --vars=\u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e=\u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e,foo=bar\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e: your custom compilation variable.\n - \u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e: the value of your custom compilation variable.\n\nRun code\n\nTo run your code, Dataform accesses BigQuery to\ndetermine its current state and tailor the resulting SQL accordingly.\n\n- To run the code of your Dataform project, run the following command:\n\n dataform run\n\n- To run the code of your Dataform project in BigQuery\n with [custom compilation variables](/dataform/docs/configure-compilation#compilation-variables),\n run the following command:\n\n dataform run --vars=\u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e=\u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e,sampleVar2=sampleValue2\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e: your custom compilation variable.\n - \u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e: the value of your custom compilation variable.\n- To run the code of your Dataform project in BigQuery\n and rebuild all tables from scratch, run the following command:\n\n dataform run --full-refresh\n\nWithout `--full-refresh`, Dataform updates incremental tables\nwithout rebuilding them from scratch.\n\n- To perform a dry run of your code against BigQuery,\n run the following command:\n\n dataform run --dry-run\n\n| **Note:** In Dataform core `3.0.2` or later, Dataform performs a dry run of each action in your compilation output against BigQuery, and requires corresponding IAM permissions. To verify compilation without a dry run, [view compilation output](#view_compilation_output).\n\nGet help\n\n- To list all of the available commands and options,\n run the following command:\n\n dataform help\n\n- To view a description of a specific command,\n run the following command:\n\n dataform help \u003cvar translate=\"no\"\u003eCOMMAND\u003c/var\u003e\n\n Replace \u003cvar translate=\"no\"\u003eCOMMAND\u003c/var\u003e with the command you want to learn about.\n\nWhat's next\n\n- To learn more about Dataform CLI, see [Dataform CLI reference](/dataform/docs/reference/dataform-cli-reference)\n- To learn more about Dataform, see [Dataform overview](/dataform/docs/overview)."]]
