Restez organisé à l'aide des collections
Enregistrez et classez les contenus selon vos préférences.
Utiliser le modèle de flux de modifications Bigtable vers BigQuery
Dans ce guide de démarrage rapide, vous allez apprendre à configurer une table Bigtable avec un flux de modifications activé, à exécuter un pipeline de flux de modifications, à apporter des modifications à votre table, puis à voir les modifications diffusées.
Avant de commencer
In the Google Cloud console, on the project selector page,
select or create a Google Cloud project.
Cliquez sur l'ID de l'instance que vous utilisez pour ce guide de démarrage rapide.
Si vous n'avez pas d'instance disponible, créez-en une avec les configurations par défaut dans une région proche de vous.
Dans le volet de navigation de gauche, cliquez sur Tables.
Cliquez sur Créer une table.
Nommez la table bigquery-changestream-quickstart.
Ajoutez une famille de colonnes nommée cf.
Sélectionnez Activer le flux de modifications.
Cliquez sur Créer.
Sur la page Tables de Bigtable, recherchez votre table bigquery-changestream-quickstart.
Dans la colonne Flux de modifications, cliquez sur Connecter.
Dans la boîte de dialogue, sélectionnez BigQuery.
Cliquez sur Créer un job Dataflow.
Dans les champs fournis, saisissez vos valeurs de paramètres. Vous n'avez pas besoin de fournir de paramètres facultatifs.
Définissez l'ID du profil d'application Bigtable sur default.
Définissez l'ensemble de données BigQuery sur bigtable_bigquery_quickstart.
Cliquez sur Run Job (Exécuter la tâche).
Attendez que l'état du job soit Démarrage ou En cours d'exécution avant de continuer.
Une fois le job mis en file d'attente, il faut environ cinq minutes pour qu'il soit exécuté.
Laissez le job ouvert dans un onglet pour pouvoir l'arrêter lorsque vous nettoierez vos ressources.
Écrire des données dans Bigtable
Dans Cloud Shell, écrivez quelques lignes dans Bigtable afin que le journal des modifications puisse écrire des données dans BigQuery. Tant que vous écrivez les données après la création du job, les modifications apparaissent. Vous n'avez pas besoin d'attendre que l'état du job devienne running.
Sauf indication contraire, le contenu de cette page est régi par une licence Creative Commons Attribution 4.0, et les échantillons de code sont régis par une licence Apache 2.0. Pour en savoir plus, consultez les Règles du site Google Developers. Java est une marque déposée d'Oracle et/ou de ses sociétés affiliées.
Dernière mise à jour le 2025/09/04 (UTC).
[[["Facile à comprendre","easyToUnderstand","thumb-up"],["J'ai pu résoudre mon problème","solvedMyProblem","thumb-up"],["Autre","otherUp","thumb-up"]],[["Difficile à comprendre","hardToUnderstand","thumb-down"],["Informations ou exemple de code incorrects","incorrectInformationOrSampleCode","thumb-down"],["Il n'y a pas l'information/les exemples dont j'ai besoin","missingTheInformationSamplesINeed","thumb-down"],["Problème de traduction","translationIssue","thumb-down"],["Autre","otherDown","thumb-down"]],["Dernière mise à jour le 2025/09/04 (UTC)."],[[["\u003cp\u003eThis quickstart guides you through setting up a Bigtable table with change stream enabled, running a change stream pipeline, and observing streamed changes in BigQuery.\u003c/p\u003e\n"],["\u003cp\u003eBefore starting, ensure you have a Google Cloud project with billing enabled and that the Dataflow, Cloud Bigtable API, Cloud Bigtable Admin API, and BigQuery APIs are enabled.\u003c/p\u003e\n"],["\u003cp\u003eYou will create a BigQuery dataset named \u003ccode\u003ebigtable_bigquery_quickstart\u003c/code\u003e and a Bigtable table named \u003ccode\u003ebigquery-changestream-quickstart\u003c/code\u003e with a change stream enabled, and connect it to a BigQuery Dataflow job.\u003c/p\u003e\n"],["\u003cp\u003eThe process includes writing data to the Bigtable, viewing the change logs in BigQuery, and then cleaning up the resources by disabling the change stream, deleting the table and dataset, and stopping the change stream pipeline.\u003c/p\u003e\n"],["\u003cp\u003eAfter completing this tutorial, you can further learn how to write queries for the change log and refer to the Bigtable change stream to BigQuery template reference page.\u003c/p\u003e\n"]]],[],null,["# Quickstart: Stream changes to BigQuery using a Dataflow template\n\nUse the Bigtable change stream to BigQuery template\n===================================================\n\nIn this quickstart, you learn how to set up a Bigtable table with a change stream enabled, run a\nchange stream pipeline, make changes to your table, and then see the changes\nstreamed.\n\nBefore you begin\n----------------\n\n1. In the Google Cloud console, on the project selector page,\n select or create a Google Cloud project.\n\n [Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)\n2.\n [Verify that billing is enabled for your Google Cloud project](/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project).\n\n3.\n\n\n Enable the Dataflow, Cloud Bigtable API, Cloud Bigtable Admin API, and BigQuery APIs.\n\n\n [Enable the APIs](https://console.cloud.google.com/flows/enableapi?apiid=dataflow.googleapis.com,bigtable.googleapis.com,bigtableadmin.googleapis.com,bigquery.googleapis.com)\n4. In the Google Cloud console, activate Cloud Shell.\n\n [Activate Cloud Shell](https://console.cloud.google.com/?cloudshell=true)\n\nCreate a BigQuery dataset\n-------------------------\n\nUse the Google Cloud console to create a dataset that stores the data.\n\n1. In the Google Cloud console, go to the BigQuery page.\n\n [Go to BigQuery](https://console.cloud.google.com/bigquery)\n2. In the **Explorer** pane, click your project name.\n\n3. Expand the more_vert **Actions** option and click **Create dataset**.\n\n4. On the **Create dataset** page, do the following:\n\n 1. For **Dataset ID** , enter `bigtable_bigquery_quickstart`.\n 2. Leave the remaining default settings as they are, and click **Create dataset**.\n\nCreate a table with a change stream enabled\n-------------------------------------------\n\n1. In the Google Cloud console, go to the Bigtable **Instances** page.\n\n [Go to Instances](https://console.cloud.google.com/bigtable/instances)\n2. Click the ID of the instance that you are using for this quickstart.\n\n If you don't have an instance available, [create an instance](/bigtable/docs/creating-instance) with the default\n configurations in a region near you.\n3. In the left navigation pane, click **Tables**.\n\n4. Click **Create a table**.\n\n5. Name the table `bigquery-changestream-quickstart`.\n\n6. Add a column family named `cf`.\n\n7. Select **Enable change stream**.\n\n8. Click **Create**.\n\n9. On the Bigtable **Tables** page, find your table\n `bigquery-changestream-quickstart`.\n\n10. In the **Change stream** column, click **Connect**.\n\n11. In the dialog, select **BigQuery**.\n\n12. Click **Create Dataflow job**.\n\n13. In the provided parameter fields, enter your parameter values. You don't\n need to provide any optional parameters.\n\n 1. Set the Bigtable application profile ID to `default`.\n 2. Set the BigQuery dataset to `bigtable_bigquery_quickstart`.\n14. Click **Run job**.\n\n15. Wait until the job status is **Starting** or **Running** before proceeding.\n It takes around 5 minutes once the job is queued.\n\n16. Keep the job open in a tab, so you can stop the job when cleaning up\n your resources.\n\nWrite some data to Bigtable\n---------------------------\n\n1. In the Cloud Shell, write a few rows to Bigtable so the\n change log can write some data to BigQuery. As long as you write\n the data after the job is created, the changes appear. You don't have to\n wait for the job status to become `running`.\n\n **Note:** To install cbt CLI if not already installed, see [Install the cbt CLI](/bigtable/docs/cbt-overview#installing). \n\n cbt -instance=\u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e -project=\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e \\\n set bigquery-changestream-quickstart user123 cf:col1=abc\n cbt -instance=\u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e -project=\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e \\\n set bigquery-changestream-quickstart user546 cf:col1=def\n cbt -instance=\u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e -project=\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e \\\n set bigquery-changestream-quickstart user789 cf:col1=ghi\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the ID of the project that you are using\n - \u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e: the ID of the instance that contains the `bigquery-changestream-quickstart` table\n\nView the change logs in BigQuery\n--------------------------------\n\n1. In the Google Cloud console, go to the **BigQuery** page.\n\n [Go to BigQuery](https://console.cloud.google.com/bigquery)\n2. In the **Explorer** pane, expand your project and the dataset\n `bigtable_bigquery_quickstart`.\n\n3. Click the table `bigquery-changestream-quickstart_changelog`.\n\n4. To see the change log, click **Preview**.\n\nClean up\n--------\n\n\nTo avoid incurring charges to your Google Cloud account for\nthe resources used on this page, follow these steps.\n\n1. Disable the change stream on the table:\n\n gcloud bigtable instances tables update bigquery-changestream-quickstart \\\n --project=\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e --instance=\u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e \\\n --clear-change-stream-retention-period\n\n2. Delete the table `bigquery-changestream-quickstart`:\n\n cbt --instance=\u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e --project=\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e deletetable bigquery-changestream-quickstart\n\n3. Stop the change stream pipeline:\n\n 1. In the Google Cloud console, go to the Dataflow **Jobs**\n page.\n\n [Go to Jobs](https://console.cloud.google.com/dataflow/jobs)\n 2. Select your streaming job from the job list.\n\n 3. In the navigation, click **Stop**.\n\n 4. In the **Stop job** dialog, select **Cancel** , and then click **Stop\n job.**\n\n4. Delete the BigQuery dataset:\n\n 1. In the Google Cloud console, go to the BigQuery page.\n\n [Go to BigQuery](https://console.cloud.google.com/bigquery)\n 2. In the **Explorer** panel, find the dataset `bigtable_bigquery_quickstart` and click it.\n\n 3. Click **Delete** , type `delete`, and then click **Delete** to confirm.\n\n5. Optional: Delete the instance if you created a new one for this quickstart:\n\n cbt deleteinstance \u003cvar translate=\"no\"\u003eBIGTABLE_INSTANCE_ID\u003c/var\u003e\n\nWhat's next\n-----------\n\n- [Learn how to write queries for the change log.](/bigtable/docs/query-change-log)\n- [Refer to the Bigtable change stream to\n BigQuery template reference\n page.](/dataflow/docs/guides/templates/provided/cloud-bigtable-change-streams-to-bigquery)"]]
