Mapconfig=ImmutableMap.<String,Object>builder().put("config1","abc").put("config2",1);pipeline.apply(Managed.read(/*Which source to read*/).withConfig(config)).getSinglePCollection();
[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-09-04 (世界標準時間)。"],[[["\u003cp\u003eManaged I/O simplifies the management of Apache Beam pipelines by providing a common API for I/O connectors (sources and sinks) and a Dataflow service to manage them.\u003c/p\u003e\n"],["\u003cp\u003eDataflow automatically upgrades managed I/O connectors, ensuring pipelines receive security fixes, performance improvements, and bug fixes without requiring code changes.\u003c/p\u003e\n"],["\u003cp\u003eManaged I/O offers a consistent configuration API using key-value properties, resulting in simpler and more uniform pipeline code compared to traditional Apache Beam connectors.\u003c/p\u003e\n"],["\u003cp\u003eFor sinks, managed I/O supports dynamic destinations, enabling the selection of a destination at runtime based on field values in the incoming records, using template strings.\u003c/p\u003e\n"],["\u003cp\u003eManaged I/O includes filtering options like \u003ccode\u003edrop\u003c/code\u003e, \u003ccode\u003ekeep\u003c/code\u003e, and \u003ccode\u003eonly\u003c/code\u003e to specify which fields to include or exclude when writing to the destination.\u003c/p\u003e\n"]]],[],null,["# Dataflow managed I/O\n\nManaged I/O enables Dataflow to manage specific I/O connectors\nused in Apache Beam pipelines. Managed I/O simplifies the management of\npipelines that integrate with supported sources and sinks.\n\nManaged I/O consists of two components that work together:\n\n- An Apache Beam transform that provides a common API for creating I/O\n connectors (sources and sinks).\n\n- A Dataflow service that manages these I/O connectors on your\n behalf, including the ability to upgrade them independently of the\n Apache Beam version.\n\nAdvantages of managed I/O include the following:\n\n- **Automatic upgrades** . Dataflow automatically upgrades the\n managed I/O connectors in your pipeline. That means your pipeline\n receives security fixes, performance improvements, and bug fixes for these\n connectors, without requiring any code changes. For more information, see\n [Automatic upgrades](#upgrades).\n\n- **Consistent API** . Traditionally, I/O connectors in Apache Beam have\n distinct APIs, and each connector is configured in a different way. Managed\n I/O provides a single configuration API that uses key-value properties,\n resulting in simpler and more consistent pipeline code. For more\n information, see [Configuration API](#configuration).\n\nRequirements\n------------\n\n- The following SDKs support managed I/O:\n\n - Apache Beam SDK for Java version 2.58.0 or later.\n - Apache Beam SDK for Python version 2.61.0 or later.\n- The backend service requires\n [Dataflow Runner v2](/dataflow/docs/runner-v2). If Runner v2\n is not enabled, your pipeline still runs, but it doesn't get the benefits of\n the managed I/O service.\n\nAutomatic upgrades\n------------------\n\nDataflow pipelines with managed I/O connectors automatically use\nthe latest reliable version of the connector, as follows:\n\n- When you submit a job, Dataflow uses the newest version of the\n connector that has been tested and works well.\n\n- For streaming jobs, Dataflow checks for updates whenever you\n [launch a replacement job](/dataflow/docs/guides/updating-a-pipeline#Launching),\n and automatically uses the latest known-good version. Dataflow\n performs this check even if you don't change any code in the replacement job.\n\nYou don't have to worry about manually updating the connector or your pipeline's\nApache Beam version.\n\nThe following diagram shows the upgrade process. The user creates an\nApache Beam pipeline using SDK version *X* . When the user submits the job,\nDataflow checks the version of the Managed I/O and upgrades it\nto version *Y*.\n\nThe upgrade process adds about two minutes to the startup time for a job. To\ncheck the status of managed I/O operations, look for\n[log entries](/dataflow/docs/guides/logging) that include the string\n\"`Managed Transform(s)`\".\n\nConfiguration API\n-----------------\n\nManaged I/O is a turnkey Apache Beam transform that provides a consistent\nAPI to configure sources and sinks. \n\n### Java\n\nTo create any source or sink supported by Managed I/O, you use the\n[`Managed`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/managed/Managed.html) class. Specify which source or sink to instantiate,\nand pass in a set of configuration parameters, similar to the following: \n\n Map config = ImmutableMap.\u003cString, Object\u003ebuilder()\n .put(\"config1\", \"abc\")\n .put(\"config2\", 1);\n\n pipeline.apply(Managed.read(/*Which source to read*/).withConfig(config))\n .getSinglePCollection();\n\nYou can also pass configuration parameters as a YAML file. For a complete code\nexample, see\n[Read from Apache Iceberg](/dataflow/docs/guides/read-from-iceberg#example).\n\n### Python\n\nImport the [`apache_beam.transforms.managed`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.managed.html) module\nand call the `managed.Read` or `managed.Write` method. Specify which source or\nsink to instantiate, and pass in a set of configuration parameters, similar to\nthe following: \n\n pipeline\n | beam.managed.Read(\n beam.managed.\u003cvar translate=\"no\"\u003e\u003cspan class=\"devsite-syntax-n\"\u003eSOURCE\u003c/span\u003e\u003c/var\u003e, # Example: beam.managed.KAFKA\n config={\n \"config1\": \"abc\",\n \"config2\": 1\n }\n )\n\nYou can also pass configuration parameters as a YAML file. For a complete code\nexample, see\n[Read from Apache Kafka](/dataflow/docs/guides/read-from-kafka#dataflow_kafka_read-python).\n\nDynamic destinations\n--------------------\n\nFor some sinks, the managed I/O connector can dynamically select a destination\nbased on field values in the incoming records.\n\nTo use dynamic destinations, provide a template string for the destination. The\ntemplate string can include field names within curly brackets, such as\n`\"tables.{field1}\"`. At runtime, the connector substitutes the value of the\nfield for each incoming record, to determine the destination for that record.\n\nFor example, suppose your data has a field named `airport`. You could set the\ndestination to `\"flights.{airport}\"`. If `airport`=`SFO`, the record is written\nto `flights.SFO`. For nested fields, use dot-notation. For example:\n`{top.middle.nested}`.\n\nFor example code that shows how to use dynamic destinations, see\n[Write with dynamic destinations](/dataflow/docs/guides/write-to-iceberg#dynamic-destinations-example).\n\n### Filtering\n\nYou might want to filter out certain fields before they are written to the\ndestination table. For sinks that support dynamic destinations, you can use\nthe `drop`, `keep`, or `only` parameter for this purpose. These parameters let\nyou include destination metadata in the input records, without writing the\nmetadata to the destination.\n\nYou can set at most one of these parameters for a given sink.\n\nSupported sources and sinks\n---------------------------\n\nManaged I/O supports the following sources and sinks.\n\n- [Apache Iceberg](/dataflow/docs/guides/managed-io-iceberg)\n- [Apache Kafka](/dataflow/docs/guides/managed-io-kafka)\n- [BigQuery](/dataflow/docs/guides/managed-io-bigquery)\n\nFor more information, see\n[Managed I/O Connectors](https://beam.apache.org/documentation/io/managed-io/)\nin the Apache Beam documentation."]]
