Tetap teratur dengan koleksi
Simpan dan kategorikan konten berdasarkan preferensi Anda.
Dokumen ini menunjukkan cara menggunakan antarmuka command line (CLI) Dataform open source untuk mengembangkan alur kerja secara lokal menggunakan terminal.
Dengan Dataform CLI open source, Anda dapat melakukan inisialisasi, mengompilasi, menguji, dan menjalankan Dataform core secara lokal, di luar Google Cloud.
CLI Dataform mendukung Kredensial Default Aplikasi (ADC).
Dengan ADC, Anda dapat menyediakan kredensial untuk aplikasi di berbagai lingkungan, seperti produksi atau pengembangan lokal, tanpa perlu mengubah kode aplikasi. Untuk menggunakan ADC, Anda harus
memberikan kredensial Anda ke ADC terlebih dahulu.
Untuk menginstal Dataform CLI, jalankan perintah berikut:
npm i -g @dataform/cli@^3.0.0-beta
Melakukan inisialisasi project Dataform
Untuk menginisialisasi project Dataform baru,
jalankan perintah berikut di dalam direktori project Anda:
dataform init . PROJECT_NAMEDEFAULT_LOCATION
Ganti kode berikut:
PROJECT_NAME: nama project Anda.
DEFAULT_LOCATION (opsional): lokasi tempat Anda ingin Dataform menulis data BigQuery. Jika tidak disetel,
Dataform menentukan lokasi berdasarkan set data yang
dirujuk oleh kueri SQL Anda. Cara kerjanya sebagai berikut:
Jika kueri Anda mereferensikan set data dari lokasi yang sama,
Dataform menggunakan lokasi tersebut.
Jika kueri Anda mereferensikan set data dari dua atau beberapa lokasi yang berbeda, akan terjadi error. Untuk mengetahui detail tentang batasan ini, lihat
Replikasi set data lintas-region.
Jika kueri Anda tidak mereferensikan set data apa pun, lokasi default untuk
Dataform adalah multi-region US. Untuk memilih lokasi lain, tetapkan lokasi default. Atau, gunakan
variabel sistem @@location
dalam kueri Anda. Untuk mengetahui informasi selengkapnya, lihat
Menentukan lokasi.
Memperbarui inti Dataform
Untuk mengupdate framework inti Dataform, perbarui dataformCoreVersion
di file workflow_settings.yaml, lalu jalankan kembali penginstalan NPM:
npm i
Mengupdate Dataform CLI
Untuk mengupdate alat CLI Dataform, jalankan perintah berikut:
npm i -g @dataform/cli@^3.0.0-beta.2
Buat file kredensial
Dataform memerlukan file kredensial untuk terhubung ke layanan jarak jauh dan membuat file .df-credentials.json di disk Anda.
Untuk membuat file kredensial, ikuti langkah-langkah berikut:
Jalankan perintah berikut:
dataform init-creds
Ikuti wizard init-creds yang memandu Anda membuat file kredensial.
Membuat project
Project Dataform kosong di Dataform core 3.0.0-beta.0
atau yang lebih baru memiliki struktur berikut:
project-dir
├── definitions
├── includes
└── workflow_settings.yaml
Untuk membuat project Dataform guna men-deploy aset ke BigQuery, jalankan perintah berikut:
YOUR_GOOGLE_CLOUD_PROJECT_ID: ID project Google Cloud Anda.
DEFAULT_LOCATION (opsional): lokasi tempat Anda ingin Dataform menulis data BigQuery. Jika tidak disetel,
Dataform menentukan lokasi berdasarkan set data yang
dirujuk oleh kueri SQL Anda. Cara kerjanya sebagai berikut:
Jika kueri Anda mereferensikan set data dari lokasi yang sama,
Dataform menggunakan lokasi tersebut.
Jika kueri Anda mereferensikan set data dari dua atau beberapa lokasi yang berbeda, akan terjadi error. Untuk mengetahui detail tentang batasan ini, lihat
Replikasi set data lintas-region.
Jika kueri Anda tidak mereferensikan set data apa pun, lokasi default untuk
Dataform adalah multi-region US. Untuk memilih lokasi lain, tetapkan lokasi default. Atau, gunakan
variabel sistem @@location
dalam kueri Anda. Untuk mengetahui informasi selengkapnya, lihat
Menentukan lokasi.
Meng-clone project
Untuk meng-clone project Dataform yang ada dari repositori Git pihak ketiga, ikuti petunjuk dari penyedia Git Anda.
Setelah repositori di-clone, jalankan perintah berikut di dalam
direktori repositori yang di-clone:
dataform install
Menentukan tabel
Simpan definisi di folder definitions/.
Untuk menentukan tabel, jalankan perintah berikut:
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-09-04 UTC."],[[["\u003cp\u003eThe Dataform CLI allows for local development of SQL workflows, enabling users to initialize, compile, test, and run Dataform core outside of Google Cloud.\u003c/p\u003e\n"],["\u003cp\u003eThe Dataform CLI supports Application Default Credentials (ADC) for managing credentials across different environments without code modification.\u003c/p\u003e\n"],["\u003cp\u003eDataform projects can be initialized, cloned, and managed locally, and the CLI provides commands to define tables, manual assertions, and custom SQL operations.\u003c/p\u003e\n"],["\u003cp\u003eUsers can compile code and view the output in real-time, either in the terminal or as a JSON object, including options for custom compilation variables.\u003c/p\u003e\n"],["\u003cp\u003eThe CLI facilitates code execution, dry runs, and full refreshes in BigQuery, with commands available to get help on specific command options.\u003c/p\u003e\n"]]],[],null,["This document shows you how to use the open-source Dataform command-line interface (CLI) to locally develop workflows by using the terminal.\n\n\u003cbr /\u003e\n\nWith the open-source Dataform CLI, you can initialize, compile, test,\nand run [Dataform core](/dataform/docs/overview#dataform-core) locally,\noutside of Google Cloud.\n\nThe Dataform CLI supports [Application Default Credentials (ADC)](/docs/authentication/application-default-credentials).\nWith ADC, you can make credentials available to your application in a variety\nof environments, such as local development or production, without needing to\nmodify your application code. To use ADC, you must first\n[provide your credentials](/docs/authentication/provide-credentials-adc) to ADC.\n| **Note:** As of [Dataform core `3.0.0.`](https://github.com/dataform-co/dataform/releases/tag/3.0.0), Dataform doesn't distribute a Docker image. You can build your own Docker image of Dataform, which you can use to run the equivalent of Dataform CLI commands. To build your own Docker image, see [Containerize an application](https://docs.docker.com/guides/workshop/02_our_app/) in the Docker documentation.\n\nBefore you begin\n\nBefore installing the Dataform CLI, install\n[NPM](https://www.npmjs.com/get-npm).\n\nInstall Dataform CLI\n\n- To install Dataform CLI, run the following command:\n\n npm i -g @dataform/cli@^3.0.0-beta\n\nInitialize a Dataform project\n\n- To initialize a new Dataform project,\n run the following command inside your project directory:\n\n dataform init . \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: the name of your project.\n - \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e (optional): the location where\n you want Dataform to write BigQuery data. If unset,\n Dataform determines the location based on the datasets that\n your SQL query references. This works as follows:\n\n - If your query references datasets from the same location, Dataform uses that location.\n - If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see [Cross-region dataset replication](/bigquery/docs/data-replication).\n - If your query doesn't reference any datasets, the default location for Dataform is the `US` multi-region. To choose a different location, set the default location. Alternatively, use the [`@@location` system variable](/bigquery/docs/reference/system-variables) in your query. For more information, see [Specify locations](/bigquery/docs/locations#specify_locations).\n\nUpdate Dataform core\n\n- To update the Dataform core framework, update the `dataformCoreVersion`\n in `workflow_settings.yaml` file, then re-run NPM install:\n\n npm i\n\nUpdate Dataform CLI\n\n- To update the Dataform CLI tool, run the following command:\n\n npm i -g @dataform/cli@^3.0.0-beta.2\n\nCreate a credentials file\n\nDataform requires a credentials file to connect to remote services\nand create the `.df-credentials.json` file on your disk.\n| **Warning:** If you use a source control system, don't commit the `df-credentials.json`. file to your repository to protect these access credentials. We recommend that you add the `.df-credentials.json` file to `.gitignore`.\n\nTo create the credentials file, follow these steps:\n\n1. Run the following command:\n\n dataform init-creds\n\n2. Follow the `init-creds` wizard that walks you through credentials file creation.\n\nCreate a project\n\nAn empty Dataform project in Dataform core `3.0.0-beta.0`\nor later has the following structure: \n\n project-dir\n ├── definitions\n ├── includes\n └── workflow_settings.yaml\n\n- To create a Dataform project to deploy assets to\n BigQuery, run the following command:\n\n dataform init \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e --default-database \u003cvar translate=\"no\"\u003eYOUR_GOOGLE_CLOUD_PROJECT_ID\u003c/var\u003e --default-location \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: the name of your project.\n - \u003cvar translate=\"no\"\u003eYOUR_GOOGLE_CLOUD_PROJECT_ID\u003c/var\u003e: your Google Cloud project ID.\n - \u003cvar translate=\"no\"\u003eDEFAULT_LOCATION\u003c/var\u003e (optional): the location where you\n want Dataform to write BigQuery data. If unset,\n Dataform determines the location based on the datasets that\n your SQL query references. This works as follows:\n\n - If your query references datasets from the same location, Dataform uses that location.\n - If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see [Cross-region dataset replication](/bigquery/docs/data-replication).\n - If your query doesn't reference any datasets, the default location for Dataform is the `US` multi-region. To choose a different location, set the default location. Alternatively, use the [`@@location` system variable](/bigquery/docs/reference/system-variables) in your query. For more information, see [Specify locations](/bigquery/docs/locations#specify_locations).\n\nClone a project\n\nTo clone an existing Dataform project from a third-party Git repository,\nfollow the instructions from your Git provider.\n\n- Once the repository is cloned, run the following command inside\n the cloned repository directory:\n\n dataform install\n\nDefine a table\n\nStore definitions in the `definitions/` folder.\n\n- To define a table, run the following command:\n\n echo \"config { type: '\u003cvar translate=\"no\"\u003eTABLE_TYPE\u003c/var\u003e' } \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eTABLE_TYPE\u003c/var\u003e: the type of the table: `table`, `incremental`, or `view`.\n - \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e: a `SELECT` statement that defines the table.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the table definition file.\n\nThe following code sample defines a view in the `example` SQLX file. \n\n echo \"config { type: 'view' } SELECT 1 AS test\" \u003e definitions/example.sqlx\n\nDefine a manual assertion\n\nStore definitions in the `definitions/` folder.\n\n- To define a manual assertion, run the following command:\n\n echo \"config { type: 'assertion' } \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSELECT_STATEMENT\u003c/var\u003e: a `SELECT` statement that defines the assertion.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the custom SQL operation definition file.\n\nDefine a custom SQL operation\n\nStore definitions in the `definitions/` folder.\n\n- To define a custom SQL operation, run the following command:\n\n echo \"config { type: 'operations' } \u003cvar translate=\"no\"\u003eSQL_QUERY\u003c/var\u003e\" \u003e definitions/\u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e.sqlx\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSQL_QUERY\u003c/var\u003e: your custom SQL operation.\n - \u003cvar translate=\"no\"\u003eFILE\u003c/var\u003e: the name for the custom SQL operation definition file.\n\nView compilation output\n\nDataform compiles your code in real time.\n\n- To view the output of the compilation process in the terminal,\n run the following command:\n\n dataform compile\n\n- To view the output of the compilation process as a JSON object,\n run the following command:\n\n dataform compile --json\n\n- To view the output of the compilation with\n [custom compilation variables](/dataform/docs/configure-compilation#compilation-variables),\n run the following command:\n\n dataform compile --vars=\u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e=\u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e,foo=bar\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e: your custom compilation variable.\n - \u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e: the value of your custom compilation variable.\n\nRun code\n\nTo run your code, Dataform accesses BigQuery to\ndetermine its current state and tailor the resulting SQL accordingly.\n\n- To run the code of your Dataform project, run the following command:\n\n dataform run\n\n- To run the code of your Dataform project in BigQuery\n with [custom compilation variables](/dataform/docs/configure-compilation#compilation-variables),\n run the following command:\n\n dataform run --vars=\u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e=\u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e,sampleVar2=sampleValue2\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eSAMPLE_VAR\u003c/var\u003e: your custom compilation variable.\n - \u003cvar translate=\"no\"\u003eSAMPLE_VALUE\u003c/var\u003e: the value of your custom compilation variable.\n- To run the code of your Dataform project in BigQuery\n and rebuild all tables from scratch, run the following command:\n\n dataform run --full-refresh\n\nWithout `--full-refresh`, Dataform updates incremental tables\nwithout rebuilding them from scratch.\n\n- To perform a dry run of your code against BigQuery,\n run the following command:\n\n dataform run --dry-run\n\n| **Note:** In Dataform core `3.0.2` or later, Dataform performs a dry run of each action in your compilation output against BigQuery, and requires corresponding IAM permissions. To verify compilation without a dry run, [view compilation output](#view_compilation_output).\n\nGet help\n\n- To list all of the available commands and options,\n run the following command:\n\n dataform help\n\n- To view a description of a specific command,\n run the following command:\n\n dataform help \u003cvar translate=\"no\"\u003eCOMMAND\u003c/var\u003e\n\n Replace \u003cvar translate=\"no\"\u003eCOMMAND\u003c/var\u003e with the command you want to learn about.\n\nWhat's next\n\n- To learn more about Dataform CLI, see [Dataform CLI reference](/dataform/docs/reference/dataform-cli-reference)\n- To learn more about Dataform, see [Dataform overview](/dataform/docs/overview)."]]
