[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-08-26。"],[[["\u003cp\u003eCloud Composer creates a Cloud Storage bucket for each environment, which is used to store data like DAGs, plugins, data, and logs.\u003c/p\u003e\n"],["\u003cp\u003eSpecific folders within the Cloud Storage bucket, such as \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e, are synchronized to Airflow components, enabling dynamic updates to DAGs and custom code, with \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e synchronizing unidirectionally, and \u003ccode\u003edata/\u003c/code\u003e and \u003ccode\u003elogs/\u003c/code\u003e bidirectionally.\u003c/p\u003e\n"],["\u003cp\u003eStoring large amounts of data in the \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders can lead to operational issues, including insufficient storage space and slow synchronization.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003edata/\u003c/code\u003e folder is not synchronized to Airflow triggerers or the web server by default, but files may be synchronized on-demand during DAG parsing or execution, and the \u003ccode\u003eplugins/\u003c/code\u003e folder is synchronized with the web server.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003elogs/\u003c/code\u003e folder stores Airflow task logs, which are synchronized to the environment's bucket via the Cloud Storage API, so an increase in tasks can increase storage use and API costs.\u003c/p\u003e\n"]]],[],null,["# Data stored in Cloud Storage\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\n[Cloud Composer 3](/composer/docs/composer-3/cloud-storage \"View this page for Cloud Composer 3\") \\| **Cloud Composer 2** \\| [Cloud Composer 1](/composer/docs/composer-1/cloud-storage \"View this page for Cloud Composer 1\")\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nThis page describes what data Cloud Composer stores for your environment\nin Cloud Storage.\n\nWhen you create an environment, Cloud Composer creates a\n[Cloud Storage](/storage) bucket and associates the bucket\nwith your environment. The name of the bucket is based on the environment\nregion, name, and a random ID such as `us-central1-b1-6efabcde-bucket`.\n\nCloud Composer synchronizes specific folders in\nyour environment's bucket to Airflow components that run in your environment.\nFor example, when you update a file with the code of your Airflow DAG in the\nenvironment's bucket, Airflow components also receive the updated version.\nCloud Composer uses [Cloud Storage FUSE](/storage/docs/gcs-fuse)\nfor synchronization.\n| **Note:** Deleting your environment does not delete the environment's bucket. To avoid incurring charges to your Cloud Billing account, download your data and then [delete the Cloud Storage bucket](/storage/docs/deleting-buckets).\n\nFolders in the Cloud Storage bucket\n-----------------------------------\n\nCloud Composer synchronizes folders uni-directionally: from the\nbucket to Airflow components. Unidirectional synchronizing means that local\nchanges in these folders on an Airflow component are overwritten.\n| **Important:** The content of the `data/` folder is synchronized to Airflow schedulers and workers. This folder **is not** synchronized to Airflow triggerers or the web server. Apply suggestions from the related [known limitations](/composer/docs/composer-2/composer/docs/composer-2/known-issues#sharing-files-with-webserver) to synchronize files to the Airflow web server.\n\nData synchronization is eventually consistent. To send messages\nfrom one operator to another,\nuse [XComs](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html).\n\nCapacity considerations\n-----------------------\n\nData from `dags/`, `plugins/` and `data/` folders are synchronized to Airflow\nscheduler(s) and workers.\n\nThe contents of the `plugins/` folder is synchronized to the Airflow web server.\n\n\n| **Caution:** Don't put more than 100 MB of data into `/dags` or `/plugins` folder. Avoid storing additional files generated by Airflow components in these folders.\n\nThe more data is put into these folders, the more space is occupied\nin the local storage of Airflow components. Saving too much data in\n`dags/` and `plugins/` can disrupt your operations and lead to issues such as:\n\n- A worker or a scheduler runs out of local storage and is evicted because\n of insufficient space on the local disk of the component.\n\n- Synchronization of files from `dags/` and `plugins/` folders to workers and\n schedulers takes a long time.\n\n- Synchronizing files from `dags/` and `plugins/` folders to workers and\n schedulers becomes impossible. For example, you store a 2 GB file in the\n `dags/` folder, but the local disk of an Airflow worker can only\n accommodate 1 GB. During the synchronization, the worker runs out of local\n storage and synchronization can't be completed.\n\nDAGs and plugins folders\n------------------------\n\nTo avoid DAG run failures, store your DAGs, plugins, and Python modules in the\n`dags/` or `plugins/` folders, even if your Python modules don't contain DAGs\nor plugins.\n\nFor example, you use a `DataFlowPythonOperator` that references a `py_file`\nDataflow pipeline. That `py_file` doesn't contain DAGs or\nplugins, but you must still store it in the `dags/` or `plugins/` folder.\n\nData folder\n-----------\n\n| **Important:** In general, the contents of the `data/` folder are not synchronized to Airflow components by default.\n\nThere are scenarios when certain files from the `data/` folder are\nsynchronized to a specific Airflow component. For example, when\nCloud Composer attempts to read a given file for the first time during:\n\n- DAG parsing: When a file is read for the first time during DAG parsing,\n Cloud Composer synchronizes it to the scheduler that parses the DAG.\n\n- DAG execution: When a file is read for the first time during DAG execution,\n Cloud Composer synchronizes it to the worker running the execution.\n\nAirflow components have limited local storage, so consider deleting\ndownloaded files to free disk space in your components. Notice that local\nstorage usage can also temporarily go up if you have concurrent tasks that\ndownload the same file to a single Airflow worker.\n| **Note:** Do not put Python modules required during the DAG parsing into the `data/` folder. It can have a negative impact on your DAG parsing and task scheduling latency.\n\nLogs folder\n-----------\n\nThe `logs/` folder is synchronized from Airflow workers to the environment's\nbucket using the Cloud Storage API.\n\nCloud Storage API quota is calculated by the amount of data moved, so\nthe number of Airflow tasks your system runs can increase\nyour Cloud Storage API usage: the more tasks you run, the bigger your log\nfiles.\n| **Note:** In rare cases you might be increasing log levels from `INFO` to `DEBUG`, as described in [Troubleshooting DAGs](/composer/docs/composer-2/troubleshooting-dags). Note that such operations produce additional logs and could increase your syncing and storing costs.\n\nSynchronization with the web server\n-----------------------------------\n\nThe `plugins/` folder is automatically synchronized to the web server so that\nplugins can be loaded by Airflow UI.\n\n\n| **Important:** The contents of the `data/` folder aren't synchronized to the web server.\n\nWhat's next\n-----------\n\n- [View logs](/composer/docs/composer-2/view-logs)\n- [Manage DAGs](/composer/docs/composer-2/manage-dags)\n- [Write DAGs](/composer/docs/composer-2/write-dags)"]]
