[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-09-04 (世界標準時間)。"],[[["\u003cp\u003eDataflow Prime is a serverless platform for processing Apache Beam pipelines, offering improved efficiency through a compute and state-separated architecture.\u003c/p\u003e\n"],["\u003cp\u003eDataflow Prime supports both batch and streaming pipelines, with default features like Dataflow Shuffle and Runner v2 for batch processing.\u003c/p\u003e\n"],["\u003cp\u003eKey features of Dataflow Prime include Vertical Autoscaling for memory management, Right fitting for resource optimization, Job Visualizer, Smart Recommendations, and Data Pipelines.\u003c/p\u003e\n"],["\u003cp\u003eTo use Dataflow Prime, you must enable the Cloud Autoscaling API and then enable the Prime option in your pipeline configurations.\u003c/p\u003e\n"],["\u003cp\u003eDataflow Prime does not support designating specific VM types, SSH into worker VMs, Flexible Resource Scheduling, or the use of VPC Service Controls with Vertical Autoscaling.\u003c/p\u003e\n"]]],[],null,["# Use Dataflow Prime\n\nDataflow Prime is a serverless data processing platform for\nApache Beam pipelines. Based on Dataflow, Dataflow Prime\nuses a compute and state-separated architecture. In the following cases,\nDataflow Prime might improve pipeline efficiency:\n\n- Your pipeline would benefit from [Vertical Autoscaling](/dataflow/docs/vertical-autoscaling).\n\nDataflow Prime supports both batch and streaming pipelines.\nBy default, Dataflow Prime uses\n[Dataflow Shuffle](/dataflow/docs/shuffle-for-batch)\nand\n[Dataflow Runner v2](/dataflow/docs/runner-v2)\nfor batch pipelines.\n\nSDK version support\n-------------------\n\nDataflow Prime supports the following Apache Beam SDKs:\n\n- Apache Beam Python SDK version 2.21.0 or later\n\n- Apache Beam Java SDK version 2.30.0 or later\n\n- Apache Beam Go SDK version 2.44.0 or later\n\nTo download the SDK package or to read the Release Notes, see\n[Apache Beam Downloads](https://beam.apache.org/get-started/downloads/).\n\nDataflow Prime features\n-----------------------\n\nThe following is the list of supported Dataflow Prime features for different kinds of pipelines:\n\n- **Vertical Autoscaling (memory).** Supports streaming pipelines in Python, Java, and Go.\n- **Right fitting (resource hints).** Supports batch pipelines in Python and Java.\n- **Job Visualizer.** Supports batch pipelines in Python and Java.\n- **Smart Recommendations.** Supports both streaming and batch pipelines in Python and Java.\n- **Data Pipelines.** Supports both streaming and batch pipelines in Python and Java.\n\nThe features Job Visualizer, Smart Recommendations, and Data Pipelines\nare also supported for non-Dataflow Prime jobs.\n\n### Vertical Autoscaling\n\nThis feature automatically adjusts the memory available to the\nDataflow worker VMs to fit the needs of the pipeline and\nhelp prevent out-of-memory errors. In Dataflow Prime, Vertical\nAutoscaling works alongside Horizontal Autoscaling to scale resources dynamically.\n\nFor more information,\nsee [Vertical Autoscaling](/dataflow/docs/vertical-autoscaling).\n\n### Right fitting\n\nThis feature uses\n[resource hints](https://beam.apache.org/documentation/runtime/resource-hints/),\na feature of Apache Beam. By using resource hints, you can\nspecify resource requirements either for the entire pipeline or for\nspecific steps of the pipeline. This feature lets you create customized workers for\ndifferent steps of a pipeline. Right fitting lets you specify pipeline\nresources to maximize efficiency, lower operational cost,\nand avoid out-of-memory and other resource errors. It supports memory and GPU\nresource hints.\n\nRight fitting requires\n[Apache Beam 2.30.0](https://beam.apache.org/blog/beam-2.30.0/)\nor later.\n\nFor more information,\nsee [Right fitting](/dataflow/docs/guides/right-fitting).\n\n### Job Visualizer\n\nThis feature lets you see the performance of a Dataflow job\nand optimize the performance of the job by finding inefficient code, including\nparallelization bottlenecks. In the Google Cloud console, you can click\nany Dataflow job in the **Jobs** page to view details about the job.\nYou can also see the list of steps associated with each stage of the pipeline.\n\nFor more information,\nsee [Execution details](/dataflow/docs/concepts/execution-details).\n\n### Smart Recommendations\n\nThis feature lets you optimize and troubleshoot\nthe pipeline based on the recommendations provided in the\n**Diagnostics** tab of the job details page. In the Google Cloud console, you can click\nany Dataflow job in the **Jobs** page to view details about the job.\n\nFor more information,\nsee [Diagnostics](/dataflow/docs/guides/logging#diagnostics).\n\n### Data Pipelines\n\nThis feature lets you schedule jobs, observe resource utilizations,\ntrack data freshness objectives for streaming data, and optimize pipelines.\n\nFor more information, see [Working with Data Pipelines](/dataflow/docs/guides/data-pipelines).\n\nQuota and limit requirements\n----------------------------\n\nQuotas and limits are the same for Dataflow and Dataflow Prime.\nFor more information, see [Quotas and limits](https://cloud.google.com/dataflow/quotas).\n\nIf you opt for Data Pipelines, there are additional\nimplications for [quotas and regions](/dataflow/docs/guides/data-pipelines#overview).\n\nUnsupported features\n--------------------\n\nDataflow Prime does not support the following:\n\n- Designating specific VM types by using the flag `--worker_machine_type` or `--machine_type` for Python pipelines and `--workerMachineType` for Java pipelines.\n- Viewing or using SSH to log into worker VMs.\n- [Flexible Resource Scheduling (FlexRS)](/dataflow/docs/guides/flexrs).\n- Using [VPC Service Controls](/vpc-service-controls/docs/overview) with Vertical Autoscaling. If you enable Dataflow Prime and launch a new job within a VPC Service Controls perimeter, the job uses Dataflow Prime without Vertical Autoscaling.\n- [NVIDIA Multi-Process Service (MPS)](/dataflow/docs/gpu/use-nvidia-mps).\n- Java pipelines that meet the following requirements can use the [`MapState`](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/state/MapState.html) and [`SetState`](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/state/SetState.html) classes:\n - use Streaming Engine\n - use Apache Beam SDK versions 2.58.0 and later\n - don't use Runner v2\n\nAll [pipeline options](/dataflow/docs/reference/pipeline-options) not explicitly\nmentioned previously or in the [feature comparison table](#feature-comparison) work\nthe same for Dataflow and Dataflow Prime.\n\nBefore using Dataflow Prime\n---------------------------\n\nTo use Dataflow Prime, you can reuse your existing pipeline code\nand also enable the Dataflow Prime option either through\nCloud Shell or programmatically.\n\nDataflow Prime is backward compatible with batch jobs that use\nDataflow Shuffle and streaming jobs that use Streaming Engine.\nHowever, we recommend testing your pipelines with\nDataflow Prime before you use them in a production environment.\n\nIf your streaming pipeline is running in production, to use Dataflow Prime,\nperform the following steps:\n\n1. [Stop](/dataflow/docs/guides/stopping-a-pipeline) the pipeline.\n\n2. Enable [Dataflow Prime](#enable-prime).\n\n3. Rerun the pipeline.\n\nEnable Dataflow Prime\n---------------------\n\nTo enable Dataflow Prime for a pipeline:\n\n1. Enable the Cloud Autoscaling API.\n\n [Enable the API](https://console.cloud.google.com/flows/enableapi?apiid=autoscaling.googleapis.com)\n\n Dataflow Prime uses the Cloud Autoscaling API to dynamically adjust memory.\n2. Enable Prime in your pipeline options.\n\n You can set the [pipeline options](/dataflow/docs/guides/setting-pipeline-options) either programmatically or by using the command line. For [supported Apache Beam SDK versions](/dataflow/docs/guides/enable-dataflow-prime#sdk_version_support), enable the following flag:\n\n### Java\n\n --dataflowServiceOptions=enable_prime\n\n### Python\n\nApache Beam Python SDK version 2.29.0 or later: \n\n --dataflow_service_options=enable_prime\n\nApache Beam Python SDK version 2.21.0 to 2.28.0: \n\n --experiments=enable_prime\n\n### Go\n\n --dataflow_service_options=enable_prime\n\nUse Dataflow Prime with templates\n---------------------------------\n\nIf you're using Dataflow templates, you can choose to\nenable Dataflow Prime in one of the following ways:\n\n1. For jobs launched from the **Create job from template** page:\n\n 1. Go to the **Create job from template** page.\n\n [Go to Create job from template](https://console.cloud.google.com/dataflow/createjob)\n 2. In the **Additional experiment** field, enter `enable_prime`.\n\n2. For jobs launched from a template through the command line interface, pass the `--additional-experiments=enable_prime` flag.\n\n3. To enable Dataflow Prime when you create a template, set the `--experiments=enable_prime` flag.\n\nUse Dataflow Prime in Apache Beam notebooks\n-------------------------------------------\n\nIf you're using an\n[Apache Beam notebook](/dataflow/docs/guides/interactive-pipeline-development),\nyou can enable Dataflow Prime\n[programmatically](/dataflow/docs/guides/setting-pipeline-options#setting_pipeline_options_programmatically)\nusing `PipelineOptions`: \n\n options = pipeline_options.PipelineOptions(\n flags=[],\n dataflow_service_options=['enable_prime'],\n )\n\nTo learn more about setting Dataflow options in a notebook, see\n[Launch Dataflow jobs from a pipeline created in your notebook](/dataflow/docs/guides/interactive-pipeline-development#launch-jobs-from-pipeline).\n\nFeature comparison between Dataflow and Dataflow Prime\n------------------------------------------------------\n\nThe following table compares the available features for both variants of Dataflow.\n\nWhat's next\n-----------\n\n- Read about Dataflow [quotas](https://cloud.google.com/dataflow/quotas).\n- Learn how to set [pipeline options](/dataflow/docs/guides/setting-pipeline-options).\n- See available [pipeline options](/dataflow/docs/reference/pipeline-options#worker-level_options) for Java and Python pipelines.\n- Learn more about [autotuning features](/dataflow/docs/guides/deploying-a-pipeline#autotuning-features) for Dataflow Prime.\n- Learn more about Dataflow [GPUs](/dataflow/docs/gpu)."]]
