[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-08-26。"],[[["\u003cp\u003eCloud Composer environments use a Cloud Storage bucket to store and synchronize data such as DAGs, plugins, data, and logs with Airflow components.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders are synchronized uni-directionally, meaning local changes on Airflow components are overwritten, while \u003ccode\u003edata/\u003c/code\u003e and \u003ccode\u003elogs/\u003c/code\u003e synchronize bi-directionally.\u003c/p\u003e\n"],["\u003cp\u003eStoring excessive data in \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders can cause issues like storage exhaustion and synchronization delays, so managing the size of files in these folders is important.\u003c/p\u003e\n"],["\u003cp\u003eWhile the \u003ccode\u003edata/\u003c/code\u003e folder is not synchronized by default, Cloud Composer may synchronize files from it to specific Airflow components during DAG parsing or execution.\u003c/p\u003e\n"],["\u003cp\u003eIn Airflow 1, the synchronization of the \u003ccode\u003edags/\u003c/code\u003e and \u003ccode\u003eplugins/\u003c/code\u003e folders to the web server depends on the DAG serialization setting, whereas in Airflow 2, \u003ccode\u003eplugins/\u003c/code\u003e is automatically synchronized to the web server.\u003c/p\u003e\n"]]],[],null,["# Data stored in Cloud Storage\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\n[Cloud Composer 3](/composer/docs/composer-3/cloud-storage \"View this page for Cloud Composer 3\") \\| [Cloud Composer 2](/composer/docs/composer-2/cloud-storage \"View this page for Cloud Composer 2\") \\| **Cloud Composer 1**\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nThis page describes what data Cloud Composer stores for your environment\nin Cloud Storage.\n\nWhen you create an environment, Cloud Composer creates a\n[Cloud Storage](/storage) bucket and associates the bucket\nwith your environment. The name of the bucket is based on the environment\nregion, name, and a random ID such as `us-central1-b1-6efabcde-bucket`.\n\nCloud Composer synchronizes specific folders in\nyour environment's bucket to Airflow components that run in your environment.\nFor example, when you update a file with the code of your Airflow DAG in the\nenvironment's bucket, Airflow components also receive the updated version.\nCloud Composer uses [Cloud Storage FUSE](/storage/docs/gcs-fuse)\nfor synchronization.\n| **Note:** Deleting your environment does not delete the environment's bucket. To avoid incurring charges to your Cloud Billing account, download your data and then [delete the Cloud Storage bucket](/storage/docs/deleting-buckets).\n\nFolders in the Cloud Storage bucket\n-----------------------------------\n\nCloud Composer synchronizes folders uni-directionally: from the\nbucket to Airflow components. Unidirectional synchronizing means that local\nchanges in these folders on an Airflow component are overwritten.\n| **Important:** The content of the `data/` folder is synchronized to Airflow schedulers and workers. This folder **is not** synchronized to Airflow triggerers or the web server. Apply suggestions from the related [known limitations](/composer/docs/composer-1/known-issues#sharing-files-with-webserver) to synchronize files to the Airflow web server.\n\nData synchronization is eventually consistent. To send messages\nfrom one operator to another,\nuse [XComs](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html).\n\nCapacity considerations\n-----------------------\n\nData from `dags/`, `plugins/` and `data/` folders are synchronized to Airflow\nscheduler(s) and workers.\n\n\u003cbr /\u003e\n\nIn Airflow 2, the content of the `plugins/` folder is also synchronized to the\nAirflow web server. In Airflow 1, the content `dags/` and `plugins/` folders\nis synchronized to Airflow web server only if DAG Serialization is turned off.\nOtherwise, the synchronization is not performed.\n| **Caution:** Don't put more than 100 MB of data into `/dags` or `/plugins` folder. Avoid storing additional files generated by Airflow components in these folders.\n\nThe more data is put into these folders, the more space is occupied\nin the local storage of Airflow components. Saving too much data in\n`dags/` and `plugins/` can disrupt your operations and lead to issues such as:\n\n- A worker or a scheduler runs out of local storage and is evicted because\n of insufficient space on the local disk of the component.\n\n- Synchronization of files from `dags/` and `plugins/` folders to workers and\n schedulers takes a long time.\n\n- Synchronizing files from `dags/` and `plugins/` folders to workers and\n schedulers becomes impossible. For example, you store a 2 GB file in the\n `dags/` folder, but the local disk of an Airflow worker can only\n accommodate 1 GB. During the synchronization, the worker runs out of local\n storage and synchronization can't be completed.\n\nDAGs and plugins folders\n------------------------\n\nTo avoid DAG run failures, store your DAGs, plugins, and Python modules in the\n`dags/` or `plugins/` folders, even if your Python modules don't contain DAGs\nor plugins.\n\nFor example, you use a `DataFlowPythonOperator` that references a `py_file`\nDataflow pipeline. That `py_file` doesn't contain DAGs or\nplugins, but you must still store it in the `dags/` or `plugins/` folder.\n\nData folder\n-----------\n\n| **Important:** In general, the contents of the `data/` folder are not synchronized to Airflow components by default.\n\nThere are scenarios when certain files from the `data/` folder are\nsynchronized to a specific Airflow component. For example, when\nCloud Composer attempts to read a given file for the first time during:\n\n- DAG parsing: When a file is read for the first time during DAG parsing,\n Cloud Composer synchronizes it to the scheduler that parses the DAG.\n\n- DAG execution: When a file is read for the first time during DAG execution,\n Cloud Composer synchronizes it to the worker running the execution.\n\nAirflow components have limited local storage, so consider deleting\ndownloaded files to free disk space in your components. Notice that local\nstorage usage can also temporarily go up if you have concurrent tasks that\ndownload the same file to a single Airflow worker.\n| **Note:** Do not put Python modules required during the DAG parsing into the `data/` folder. It can have a negative impact on your DAG parsing and task scheduling latency.\n\nLogs folder\n-----------\n\nThe `logs/` folder is synchronized from Airflow workers to the environment's\nbucket using the Cloud Storage API.\n\nCloud Storage API quota is calculated by the amount of data moved, so\nthe number of Airflow tasks your system runs can increase\nyour Cloud Storage API usage: the more tasks you run, the bigger your log\nfiles.\n| **Note:** In rare cases you might be increasing log levels from `INFO` to `DEBUG`, as described in [Troubleshooting DAGs](/composer/docs/composer-1/troubleshooting-dags). Note that such operations produce additional logs and could increase your syncing and storing costs.\n\nSynchronization with the web server\n-----------------------------------\n\n\u003cbr /\u003e\n\nAirflow 2 uses DAG serialization out of the box. The `plugins/` folder\nis automatically synchronized to the web server so that plugins can be loaded by\nAirflow UI. You can't turn off DAG serialization in Airflow 2.\n\nIn Airflow 1, DAG serialization is supported and is turned on by default in\nCloud Composer.\n\n- When DAG serialization is **turned on** , the files from `dags/` and `plugins/` folders **aren't synchronized to the web server**.\n- When DAG serialization is **turned off** , the files from `dags/` and `plugins/` **are synchronized to the web server**.\n\n| **Important:** The contents of the `data/` folder aren't synchronized to the web server.\n\nWhat's next\n-----------\n\n- [View logs](/composer/docs/composer-1/view-logs)\n- [Manage DAGs](/composer/docs/composer-1/manage-dags)\n- [Write DAGs](/composer/docs/composer-1/write-dags)"]]
