[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-09-03。"],[[["\u003cp\u003eCloud Composer creates a Cloud Storage bucket for each environment to store data, including DAGs, plugins, data, and logs, and synchronizes specific folders with Airflow components.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders are synchronized uni-directionally, meaning local changes on Airflow components are overwritten, whereas \u003ccode\u003edata/\u003c/code\u003e and \u003ccode\u003elogs/\u003c/code\u003e folders are synchronized bi-directionally.\u003c/p\u003e\n"],["\u003cp\u003eStoring large amounts of data in \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders can cause issues like worker or scheduler evictions due to insufficient local storage, delayed synchronization, or failed synchronization.\u003c/p\u003e\n"],["\u003cp\u003eFiles in the \u003ccode\u003edata/\u003c/code\u003e folder are only synchronized to specific Airflow components during the initial read, either during DAG parsing or execution, and are not automatically synchronized with the web server.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003eplugins/\u003c/code\u003e folder is automatically synchronized with the Airflow web server, allowing plugins to be loaded in the UI, and users can enable or disable web server plugins as needed without deleting files from the environment's bucket.\u003c/p\u003e\n"]]],[],null,["# Data stored in Cloud Storage\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\n**Cloud Composer 3** \\| [Cloud Composer 2](/composer/docs/composer-2/cloud-storage \"View this page for Cloud Composer 2\") \\| [Cloud Composer 1](/composer/docs/composer-1/cloud-storage \"View this page for Cloud Composer 1\")\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nThis page describes what data Cloud Composer stores for your environment\nin Cloud Storage.\n\nWhen you create an environment, Cloud Composer creates a\n[Cloud Storage](/storage) bucket and associates the bucket\nwith your environment. The name of the bucket is based on the environment\nregion, name, and a random ID such as `us-central1-b1-6efabcde-bucket`.\n\nCloud Composer synchronizes specific folders in\nyour environment's bucket to Airflow components that run in your environment.\nFor example, when you update a file with the code of your Airflow DAG in the\nenvironment's bucket, Airflow components also receive the updated version.\nCloud Composer uses [Cloud Storage FUSE](/storage/docs/gcs-fuse)\nfor synchronization.\n| **Note:** Deleting your environment does not delete the environment's bucket. To avoid incurring charges to your Cloud Billing account, download your data and then [delete the Cloud Storage bucket](/storage/docs/deleting-buckets).\n\nFolders in the Cloud Storage bucket\n-----------------------------------\n\nCloud Composer synchronizes folders uni-directionally: from the\nbucket to Airflow components. Unidirectional synchronizing means that local\nchanges in these folders on an Airflow component are overwritten.\n| **Important:** The content of the `data/` folder is synchronized to Airflow schedulers and workers. This folder **is not** synchronized to Airflow triggerers or the web server. Apply suggestions from the related [known limitations](/composer/docs/composer-3/composer/docs/composer-3/known-issues#sharing-files-with-webserver) to synchronize files to the Airflow web server.\n\nData synchronization is eventually consistent. To send messages\nfrom one operator to another,\nuse [XComs](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html).\n\nCapacity considerations\n-----------------------\n\nData from `dags/`, `plugins/` and `data/` folders are synchronized to Airflow\nscheduler(s) and workers.\n\nThe contents of the `plugins/` folder is synchronized to the Airflow web server.\n\n\n| **Caution:** Don't put more than 100 MB of data into `/dags` or `/plugins` folder. Avoid storing additional files generated by Airflow components in these folders.\n\nThe more data is put into these folders, the more space is occupied\nin the local storage of Airflow components. Saving too much data in\n`dags/` and `plugins/` can disrupt your operations and lead to issues such as:\n\n- A worker or a scheduler runs out of local storage and is evicted because\n of insufficient space on the local disk of the component.\n\n- Synchronization of files from `dags/` and `plugins/` folders to workers and\n schedulers takes a long time.\n\n- Synchronizing files from `dags/` and `plugins/` folders to workers and\n schedulers becomes impossible. For example, you store a 2 GB file in the\n `dags/` folder, but the local disk of an Airflow worker can only\n accommodate 1 GB. During the synchronization, the worker runs out of local\n storage and synchronization can't be completed.\n\nDAGs and plugins folders\n------------------------\n\nTo avoid DAG run failures, store your DAGs, plugins, and Python modules in the\n`dags/` or `plugins/` folders, even if your Python modules don't contain DAGs\nor plugins.\n\nFor example, you use a `DataFlowPythonOperator` that references a `py_file`\nDataflow pipeline. That `py_file` doesn't contain DAGs or\nplugins, but you must still store it in the `dags/` or `plugins/` folder.\n\nData folder\n-----------\n\n| **Important:** In general, the contents of the `data/` folder are not synchronized to Airflow components by default.\n\nThere are scenarios when certain files from the `data/` folder are\nsynchronized to a specific Airflow component. For example, when\nCloud Composer attempts to read a given file for the first time during:\n\n- DAG parsing: When a file is read for the first time during DAG parsing,\n Cloud Composer synchronizes it to the scheduler that parses the DAG.\n\n- DAG execution: When a file is read for the first time during DAG execution,\n Cloud Composer synchronizes it to the worker running the execution.\n\nAirflow components have limited local storage, so consider deleting\ndownloaded files to free disk space in your components. Notice that local\nstorage usage can also temporarily go up if you have concurrent tasks that\ndownload the same file to a single Airflow worker.\n| **Note:** Do not put Python modules required during the DAG parsing into the `data/` folder. It can have a negative impact on your DAG parsing and task scheduling latency.\n\nLogs folder\n-----------\n\nThe `logs/` folder is synchronized from Airflow workers to the environment's\nbucket using the Cloud Storage API.\n\nCloud Storage API quota is calculated by the amount of data moved, so\nthe number of Airflow tasks your system runs can increase\nyour Cloud Storage API usage: the more tasks you run, the bigger your log\nfiles.\n| **Note:** In rare cases you might be increasing log levels from `INFO` to `DEBUG`, as described in [Troubleshooting DAGs](/composer/docs/composer-3/troubleshooting-dags). Note that such operations produce additional logs and could increase your syncing and storing costs.\n\nSynchronization with the web server\n-----------------------------------\n\nThe `plugins/` folder is automatically synchronized to the web server so that\nplugins can be loaded by Airflow UI. In Cloud Composer 3, you can\n[disable and enable web server plugins](/composer/docs/composer-3/install-plugins#disable-webserver-plugins) on\ndemand, without deleting files from the environment's bucket.\n| **Important:** The contents of the `data/` folder aren't synchronized to the web server.\n\nWhat's next\n-----------\n\n- [View logs](/composer/docs/composer-3/view-logs)\n- [Manage DAGs](/composer/docs/composer-3/manage-dags)\n- [Write DAGs](/composer/docs/composer-3/write-dags)"]]
