A Dataflow regional endpoint stores and handles metadata about your Dataflow job and deploys and controls your Dataflow workers.
Regional endpoint names follow a standard convention based on Compute Engine
region names.
For example, the name for the Central US region is us-central1.
This feature is available in all regions where Dataflow is supported. To see available locations, read Dataflow locations.
Guidelines for choosing a regional endpoint
Specifying a regional endpoint in a Dataflow job is mandatory.
Security and compliance
You might need to constrain Dataflow job processing to a specific geographic region in support of your project’s security and compliance needs.
Data locality
You can minimize network latency and network transport costs by running a Dataflow job from the same region as its sources, sinks, staging file locations, and temporary file locations. If you use sources, sinks, staging file locations, or temporary file locations that are located outside of your job's region, your data might be sent across regions.
In running a pipeline, user data is only handled by the Dataflow worker pool and the movement of the data is restricted to the network paths that connect the Dataflow workers in the pool.
If you need more control over the location of pipeline log messages, you can do the following:
- Create an exclusion filter
for the
_Defaultlog router sink to prevent Dataflow logs from being exported to the_Defaultlog bucket. - Create a log bucket in the region of your choice.
- Configure a new log router sink that exports your Dataflow logs to your new log bucket.
To learn more about configuring logging, see Routing and storage overview and Log routing overview.
Notes about common Dataflow job sources:
- When using a Cloud Storage bucket as a source, we recommend that you perform read operations in the same region as the bucket.
- Pub/Sub topics, when published to the global Pub/Sub endpoint, are stored in the nearest Google Cloud region. However, you can modify the topic storage policy to a specific region or a set of regions. Similarly, Pub/Sub Lite topics support only zonal storage.
Resilience and geographic separation
You might want to isolate your normal Dataflow operations from outages that could occur in other geographic regions. Or, you may need to plan alternate sites for business continuity in the event of a region-wide disaster.
Regional placement
By default, a regional endpoint configures the Dataflow worker pool to utilize all available zones within the region. Zone selection is calculated for each worker at its creation time, optimizing for resource acquisition and utilization of unused reservations.
Regional placement offers benefits such as:
- Improved resource availability: Dataflow jobs are more resilient to zonal resoure availability errors, as workers can continue to be created in other zones with remaining availability.
- Improved reliability: In the event of a zonal failure, Dataflow jobs can continue to run, as workers will be re-created in other zones.
The following limitations apply:
- Regional placement is supported only for jobs using Streaming Engine or Dataflow Shuffle. Jobs that have opted out of Streaming Engine or Dataflow Shuffle cannot use regional placement.
- Regional placement applies to VMs only, and does not apply to Streaming Engine and Dataflow Shuffle-related resources.
- VMs are not replicated across multiple zones. If a VM becomes unavailable, for example, its work items are considered lost and are reprocessed by another VM.
- If a region-wide stockout occurs, the Dataflow service cannot create any more VMs.
Auto zone placement
For jobs not supported for regional placement, the regional endpoint automatically selects the best zone within the region, based on the available zone capacity at the time of the job creation request. Automatic zone selection helps ensure that job workers run in the best zone for your job.
Because the job is configured to run in a single zone, the operation might fail with a zonal resource availability error if sufficient Compute Engine resources are not available.
Specifying a regional endpoint
To specify a regional endpoint for your job, set the --region option to one of
the supported regional endpoints.
The --region option overrides the default region that is set in the metadata server,
your local client, or environment variables.
The Dataflow command-line interface
also supports the --region option to specify regional endpoints.
Overriding the worker region or zone
By default, when you submit a job with the --region option, the regional
endpoint automatically assigns workers to
either zones across the region or the
single best zone within the region, depending on the job type.
In cases where you want to ensure that the workers for your
Dataflow job run strictly in a specific zone, you can specify
the zone using --worker_zone. This is not a common
usage pattern for Dataflow jobs.
For all other cases, we do not recommend overriding the worker location. The common scenarios table contains usage recommendations for these situations.
Because the job is configured to run in a single zone, the operation might fail with a zonal resource availability error if sufficient Compute Engine resources are not available.
You can run the gcloud compute regions list command to see a listing of
regions and zones that are available for worker deployment.
Common scenarios
The following table contains usage recommendations for common scenarios.
| Scenario | Recommendation |
|---|---|
| I want to use a supported regional endpoint and have no zone preference within the region. In this case, the regional endpoint automatically selects the best zone based on available capacity. | Use --region to specify a regional endpoint.
This ensures that Dataflow manages your job and processes
data within the specified region. |
| I need worker processing to occur in a specific zone of a region that has a regional endpoint. | Specify both --region and --worker_zone.Use |