[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-09-04 (世界標準時間)。"],[[["\u003cp\u003eThe BigQuery source plugin in Cloud Data Fusion enables connecting to and loading data from BigQuery tables, by first exporting the data to a temporary Cloud Storage location before reading it into the pipeline.\u003c/p\u003e\n"],["\u003cp\u003eConfiguring the BigQuery batch source plugin involves granting specific IAM roles and permissions to both the Cloud Data Fusion API Service Agent and the Compute Engine Service Account, such as \u003ccode\u003ebigquery.jobUser\u003c/code\u003e, \u003ccode\u003ebigquery.dataEditor\u003c/code\u003e, and \u003ccode\u003estorage.legacyBucketWriter\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eWhen configuring the plugin properties, users can choose between creating a new one-time connection or reusing an existing connection to BigQuery, and they must also specify the dataset and table to read from, along with optional settings like partition dates and filter clauses.\u003c/p\u003e\n"],["\u003cp\u003eThe plugin supports various BigQuery data types, and the provided table outlines their corresponding CDAP schema data types, such as mapping \u003ccode\u003eBOOL\u003c/code\u003e to \u003ccode\u003eboolean\u003c/code\u003e and \u003ccode\u003eINT64\u003c/code\u003e to \u003ccode\u003elong\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eUsers have the option to utilize a temporary Cloud Storage bucket for data, or have one automatically created and deleted during the process, which would require \u003ccode\u003estorage.buckets.create\u003c/code\u003e and \u003ccode\u003estorage.buckets.delete\u003c/code\u003e permissions.\u003c/p\u003e\n"]]],[],null,["# BigQuery batch source\n\nThis page provides guidance about configuring the BigQuery batch source plugin in Cloud Data Fusion.\n\n\u003cbr /\u003e\n\nThe BigQuery source plugin lets you connect and load data from\nBigQuery tables. Data from a BigQuery table is\nexported into a temporary location in Cloud Storage, and then gets read into\nthe pipeline from there.\n\nBefore you begin\n----------------\n\nCloud Data Fusion typically has two service accounts:\n\n- Design-time service account: [Cloud Data Fusion API Service Agent](/data-fusion/docs/concepts/service-accounts)\n- Execution-time service account: [Compute Engine Service Account](/data-fusion/docs/concepts/service-accounts)\n\nBefore using the BigQuery batch source plugin, grant the\nfollowing roles or permissions to each service account.\n\n#### Cloud Data Fusion API Service Agent\n\nThis service account already has all the required permissions and you don't need\nto add additional permissions. For reference, it has the following permissions:\n\n- `bigquery.datasets.get`\n- `bigquery.tables.create`\n- `bigquery.tables.get`\n- `bigquery.tables.updateData`\n- `bigquery.tables.update`\n- `bigquery.tables.export`\n\nIf you're using a [namespace service\naccount](/data-fusion/docs/how-to/control-access-in-namespace)\nin addition to the default design-time service account, add the permissions from\nthe preceding list to it.\n\n#### Compute Engine Service Account\n\nIn your Google Cloud project, grant the following IAM roles or\npermissions to the Compute Engine Service Account:\n\n- [BigQuery Job User](/iam/docs/understanding-roles#bigquery.jobUser) (`roles/bigquery.jobUser`). This predefined role contains the required `bigquery.jobs.create` permission.\n- [BigQuery Data\n Editor](/iam/docs/understanding-roles#bigquery.dataEditor) (`roles/bigquery.dataEditor`). This\n predefined role contains the following required permissions:\n\n - `bigquery.datasets.get`\n - `bigquery.tables.create`\n - `bigquery.tables.get`\n - `bigquery.tables.updateData`\n - `bigquery.tables.update`\n - `bigquery.tables.export`\n\nThese roles and permissions can also be assigned on the BigQuery\ndataset or table, depending on your use case.\n\n- [Storage Legacy Bucket Writer](/iam/docs/understanding-roles#storage.legacyBucketWriter) (`roles/storage.legacyBucketWriter`).\n This predefined role contains the following required permissions:\n\n - `storage.buckets.get`\n - `storage.objects.get`\n - `storage.objects.list`\n\nThis role and these permissions can also be assigned on the\nCloud Storage bucket, depending on your use case.\n| **Note:** If you don't provide a temporary bucket name when you configure the plugin properties, you need `storage.buckets.create` and `storage.buckets.delete` permissions, which can be assigned using the [Storage\n| Admin](/iam/docs/understanding-roles#storage.admin) role (`roles/storage.admin`) on the project.\n\nConfigure the plugin\n--------------------\n\n1. [Go to the Cloud Data Fusion web interface](/data-fusion/docs/create-data-pipeline#navigate-web-interface) and click **Studio**.\n2. Check that **Data Pipeline - Batch** is selected (not **Realtime**).\n3. In the **Source** menu, click **BigQuery**. The BigQuery node appears in your pipeline.\n4. To configure the source, go to the BigQuery node and click **Properties**.\n5. Enter the following properties. For a complete list, see\n [Properties](#properties).\n\n 1. Enter a **Label** for the BigQuery node---for example, `BigQuery tables`.\n 2. Enter the connection details. You can set up a new, one-time connection,\n or an existing, reusable connection.\n\n ### New connection\n\n\n To add a one-time connection to BigQuery, follow these\n steps:\n 1. In the **Project ID** field, leave the value as auto-detect.\n 2. If the BigQuery dataset is in a different project, in the **Dataset Project ID** field, enter the ID.\n 3. In the **Service account type** field, choose one of the following\n and enter the content in the next field:\n\n - **File path**\n - **JSON**\n\n ### Reusable connection\n\n\n To reuse an existing connection, follow these steps:\n 1. Turn on **Use connection**.\n 2. Click **Browse connections**.\n 3. Click the connection name---for example,\n **BigQuery Default**.\n\n | **Note:** For more information about adding, importing, and editing the connections that appear when you browse connections, see [Manage connections](/data-fusion/docs/how-to/managing-connections).\n 4. Optional: If a connection doesn't exist and you want to create a\n new reusable connection, click **Add connection** and refer to the\n steps in the [**New connection**](#configure) tab on this page.\n\n 3. In the **Reference name** field, enter a name to use for lineage.\n\n 4. Optional: If your dataset is already available in your instance,\n click **Browse** and select the data to read.\n\n 5. In the **Dataset** field, enter the name of the dataset that\n contains the table.\n\n 6. In the **Table** field, enter the name of the table.\n\n 7. To test connectivity, click **Get schema**.\n\n 8. Optional: In the **Partition start date** field, enter the\n inclusive start date string---for example, `2021-01-11`.\n\n 9. Optional: In the **Partition end date** field, enter the\n inclusive end date string---for example, `2024-01-11`.\n\n 10. Optional: In the **Filter** field, enter a BigQuery\n [`WHERE` clause](/bigquery/docs/reference/standard-sql/query-syntax#where_clause).\n\n 11. Optional: In the **Temporary bucket name** field, enter a name\n for the Cloud Storage bucket.\n\n 12. Optional: In the **Encryption Key Name** field, enter the Cloud Key Management Service\n (Cloud KMS) encryption key name. For more information, see [Get\n the resource name for the\n key](/data-fusion/docs/how-to/customer-managed-encryption-keys#get-the-resource-name).\n\n 13. Optional: Turn on **Enable querying views**. If you enable them, do the\n following:\n\n - In the **Temporary table creation project** field, enter the project name where the temporary table is created.\n - In the **Temporary table creation dataset** field, enter the dataset name where the temporary table is created.\n 14. Optional: Click **Validate** and address any errors found.\n\n 15. Click close **Close**. Properties\n are saved and you can continue to build your data pipeline in the\n Cloud Data Fusion web interface.\n\n### Properties\n\n### Data type mappings\n\nThe following table is a list of BigQuery data types with\ncorresponding CDAP types.\n\nRelease notes\n-------------\n\n- [February 8, 2024](https://cdap.atlassian.net/wiki/spaces/DOCS/pages/1280901131/CDAP+Hub+Release+Log#Feb-8%2C-2024)\n- [January 16, 2024](/data-fusion/docs/release-notes#January_16_2024)\n- [September 6, 2023](https://cdap.atlassian.net/wiki/spaces/DOCS/pages/1280901131/CDAP+Hub+Release+Log#September-6%2C-2023)\n- [June 14, 2023](/data-fusion/docs/release-notes#June_14_2023)\n- [December 6, 2022](/data-fusion/docs/release-notes#December_06_2022)\n\nWhat's next\n-----------\n\n- Learn more about [plugins in Cloud Data Fusion](/data-fusion/docs/concepts/plugins)."]]
