Common error guidance

This page describes some common errors you might encounter when running your Dataflow job, and suggests some courses of action for dealing with those errors.

Job messages:

Job submission errors:

Worker logs:

Input and output errors:

Worker startup errors:

Logs:

Recommendations:

Job messages

"The job failed because a work item failed 4 times."

If a single operation causes the worker code to fail four times, by the code throwing an exception or crashing, Dataflow fails the job, and the message a work item failed 4 times is displayed. You can't configure this failure threshold. For more details, refer to pipeline error and exception handling .

Look in the job's Cloud Monitoring logs for the four individual failures. Look for Error-level or Fatal-level log entries in the worker logs that show exceptions or errors. You should see at least four of these exceptions or errors.

Encoding errors, IOExceptions, or unexpected behavior in user code.

The Apache Beam SDKs and the Dataflow workers depend on common third-party components. These components import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible. You may need to pin to the listed versions that will be in scope during execution. SDK and Worker Dependencies contains a list of dependencies and their required versions.

Jobs that used to run, now fail with "Staged package...is inaccessible"

  • Verify that the Cloud Storage bucket used for staging does not have TTL settings that cause staged packages to be deleted.

  • Verify that your Dataflow project's controller service account has the permission to access the Cloud Storage bucket used for staging. Gaps in permission can be due to any of the following reasons:

    • The Cloud Storage bucket used for staging is present in a different project.
    • The Cloud Storage bucket used for staging was migrated from fine-grained access to uniform bucket-level access. Due to the inconsistency between IAM and ACL policies, migrating the staging bucket to uniform bucket-level access disallows ACLs for Cloud Storage resources, which includes the permissions held by your Dataflow project's controller service account over the staging bucket.

For more information, see Accessing Cloud Storage buckets across Google Cloud projects.

Flex Template polling timeout

Your Flex Template job might return the following error message:

Timeout in polling result file: ${file_path}.
Possible causes are:1. Your launch takes too long time to finish. Please check
the logs on stackdriver.
2. Service account ${service_account_email} may not have enough permissions to
pull container image ${image_url} or create new objects in ${file_path}.
3. Transient errors occurred, please try again.

Verifying your Docker container entrypoint. Check for the container entrypoint using the following command:

docker inspect $TEMPLATE_IMAGE

You should see the following:

Java

/opt/google/dataflow/java_template_launcher

Python

/opt/google/dataflow/python_template_launcher

If you get a different output, then your Docker container's entrypoint has been overridden. Restore $TEMPLATE_IMAGE to the default.

Job submission errors

"413 Request Entity Too Large" / "The size of serialized JSON representation of the pipeline exceeds the allowable limit"

If you encounter an error about the JSON payload when submitting your job, it means your pipeline's JSON representation exceeds the maximum 20 MB request size. These errors might appear as one of the following messages in your console or terminal window:

  • 413 Request Entity Too Large
  • "The size of serialized JSON representation of the pipeline exceeds the allowable limit"
  • "Failed to create a workflow job: Invalid JSON payload received"
  • "Failed to create a workflow job: Request payload exceeds the allowable limit"

The size of your job is specifically tied to the JSON representation of the pipeline. A larger pipeline means a larger request. Dataflow currently has a limitation that caps requests at 20 MB.

To estimate the size of your pipeline's JSON request, run your pipeline with the following option:

Java: SDK 2.x

--dataflowJobFile=< path to output file >

Python

--dataflow_job_file=< path to output file >

Java: SDK 1.x

This command writes a JSON representation of your job to a file. The size of the serialized file is a good estimate of the size of the request. The actual size will be slightly larger due to some additional information included the request.

Certain conditions in your pipeline can cause the JSON representation to exceed the limit. Common conditions include:

  • A Create transform that includes a large amount of in-memory data.
  • A large DoFn instance that is serialized for transmission to remote workers.
  • A DoFn as an anonymous inner class instance that (possibly inadvertently) pulls in a large amount of data to be serialized.

To avoid these conditions, consider restructuring your pipeline.

"The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs."

Your job's graph size must not exceed 10 MB. Certain conditions in your pipeline can cause the job graph to exceed the limit. Common conditions include:

  • A Create transform that includes a large amount of in-memory data.
  • A large DoFn instance that is serialized for transmission to remote workers.
  • A DoFn as an anonymous inner class instance that (possibly inadvertently) pulls in a large amount of data to be serialized.

To avoid these conditions, consider restructuring your pipeline.

"Total number of BoundedSource objects generated by splitIntoBundles() operation is larger than the allowable limit" or "Total size of the BoundedSource objects generated by splitIntoBundles() operation is larger than the allowable limit".

Java: SDK 2.x

You might encounter this error if you're reading from a very large number of files via TextIO, AvroIO, or some other file-based source. The particular limit depends on the details of your source (e.g. embedding schema in AvroIO.Read will allow fewer files), but it is on the order of tens of thousands of files in one pipeline.

You might also encounter this error if you've created a custom data source for your pipeline and your source's splitIntoBundles method returned a list of BoundedSource objects which takes up more than 20 MB when serialized.

The allowable limit for the total size of the BoundedSource objects generated by your custom source's splitIntoBundles() operation is 20 MB. You can work around this limitation by modifying your custom BoundedSource subclass so that the total size of the generated BoundedSource objects is smaller than the 20 MB limit. For example, your source might generate fewer splits initially, and rely on Dynamic Work Rebalancing to further split inputs on demand.

Java: SDK 1.x

"Startup of the worker pool in zone $ZONE failed to bring up any of the desired $N workers. The project quota may have been exceeded or access control policies may be preventing the operation; review the Cloud Logging "VM Instance" log for diagnostics."

This error has two possible causes:

  • You might have exceeded one of the Compute Engine quotas that Dataflow worker creation relies on.
  • Your organization has constraints in place that prohibit some aspect of the VM instance creation process, like the account being used, or the zone being targeted.

To troubleshoot this issue:

Review the VM Instance log

  1. Go to the Cloud Logging viewer
  2. In the Audited Resource drop-down list, select VM Instance.
  3. In the All logs drop-down list, select compute.googleapis.com/activity_log.
  4. Scan the log for any entries related to VM instance creation failure.

Check your usage of Compute Engine quotas

  1. On your local machine command line or in Cloud Shell, run the following command to view Compute Engine resource usage compared to Dataflow quotas for the zone you are targeting:

    gcloud compute regions describe [REGION]

  2. Review the results for the following resources to see if any are exceeding quota:

    • CPUS
    • DISKS_TOTAL_GB
    • IN_USE_ADDRESSES
    • INSTANCE_GROUPS
    • INSTANCE_GROUP_MANAGERS
    • INSTANCES

  3. If needed, request a quota change.

Review your organization policy constraints

  1. Go to the Organization policies page
  2. Review the constraints for any that might limit VM instance creation for either the account you are using (this is the Dataflow service account by default) or in the zone you are targeting.

Worker logs

"Processing stuck/Operation ongoing in step <step_id> for at least <time_interval> without outputting or completing in state finish at <stack_trace>"

If Dataflow spends more time executing a DoFn than the time specified in <time_interval> without returning, this message is displayed.

This error has two possible causes:

  • Your DoFn code is simply slow, or waiting for some slow external operation to complete.
  • Or, your DoFn code might be stuck, deadlocked, or abnormally slow to finish processing.

To determine which of these is the case, expand the Cloud Monitoring log entry to see a stack trace. Look for messages that indicate that the DoFn code is stuck or otherwise encountering issues. If none are present, the issue might be the execution speed of the DoFn code. Consider using a code profiler or other tool to investigate the code's performance.

You can further investigate the cause of your stuck code if your pipeline is built on the Java VM (using either Java or Scala); take a full thread dump of the whole JVM (not just the stuck thread) by following these steps:

  1. Make note of the worker name from the log entry.
  2. In the Compute Engine section of the Cloud Console, find the Compute Engine instance with the worker name you noted.
  3. SSH into the instance with that name.

  4. Run the following command:

    curl http://localhost:8081/threadz

Shuffle key too large exceptions

If you encounter exceptions indicating the shuffle key is too large, it means that the serialized key emitted to a particular (Co-)GroupByKey, after the corresponding coder is applied, is too large. Dataflow has a limit for serialized shuffle keys. Consider reducing the size of the keys or using more space-efficient coders.

KeyCommitTooLargeException

In streaming scenarios, this might be caused by grouping a very large amount of data without using a Combine transform, or by producing a large amount of data from a single input element. The following strategies can reduce the chance of raising this exception:

  • Ensure that processing a single element cannot result in outputs or state modifications exceeding the limit.
  • If multiple elements were grouped by a key, consider increasing the key space to reduce the elements grouped per key.
  • If elements for a key are emitted at a high frequency over a short period of time, that may result in many GB of events for that key in windows. Rewrite the pipeline to detect keys like this and only emit an output indicating the key was frequently present in that window.
  • Make sure to use sub-linear space Combine transforms for commutative and associate operations. Don't use a combiner if it doesn't reduce space. For example, combiner for strings that just appends strings together is worse than not using combiner.

RPC timeout exceptions, "DEADLINE_EXCEEDED" exceptions, or "Server Unresponsive" errors

If you encounter RPC timeouts, DEADLINE_EXCEEDED exceptions, or Server Unresponsive errors while your job runs, these typically indicate one of two problems:

  • The VPC network used for your job might be missing a firewall rule. The firewall rule needs to enable all TCP traffic among VMs in the VPC network you specified in your pipeline options. See Specifying your network and subnetwork for more details.

  • Your job is shuffle-bound. Consider one of, or a combination of, the following courses of action:

    Java: SDK 2.x

    • If the job is not using the service-based shuffle, switch to using the service-based Dataflow Shuffle by setting --experiments=shuffle_mode=service. For details and availability, read Dataflow Shuffle.
    • Add more workers. Try setting --numWorkers with a higher value when you run your pipeline.
    • Increase the size of the attached disk for workers. Try setting --diskSizeGb with a higher value when you run your pipeline.
    • Use an SSD-backed persistent disk. Try setting --workerDiskType="compute.googleapis.com/projects/<project>/zones/<zone>/diskTypes/pd-ssd" when you run your pipeline.

    Python

    • If the job is not using the service-based shuffle, switch to using the service-based Dataflow Shuffle by setting --experiments=shuffle_mode=service. For details and availability, read Dataflow Shuffle.
    • Add more workers. Try setting --num_workers with a higher value when you run your pipeline.
    • Increase the size of the attached disk for workers. Try setting --disk_size_gb with a higher value when you run your pipeline.
    • Use an SSD-backed persistent disk. Try setting --worker_disk_type="compute.googleapis.com/projects/<project>/zones/<zone>/diskTypes/pd-ssd" when you run your pipeline.

    Java: SDK 1.x

A call from the Java worker harness to a Python DoFn failed with the error <error message>

If a DoFn implemented in Python fails and throws an exception, a relevant error message is displayed.

Expand the Cloud Monitoring error log entry and look at the error message and traceback. It shows you which code failed so you can correct it if necessary. If you believe that this is a bug in Apache Beam or Dataflow, please report the bug.

No space left on device

If a job shuffles a large amount of data or writes temporary data to local disks, the worker persistent storage might run out of free space. It is also possible for the worker boot disk to fill up if it is logging more than 50 entries per second.

You can track disk space usage on the worker VM instances by using Cloud Monitoring. See Receive worker VM metrics from the Monitoring agent for instructions on setting this up.

You can also look for boot disk space issues by Viewing serial port output on the worker VM instances and looking for messages like "Failed to open system journal: No space left on device". If you have a large number of worker VM instances, you might want to create a script to run gcloud compute instances get-serial-port-output on all of them at once and review the output from that instead.

To increase persistent disk or boot disk space, follow the guidance in the disk_size_gb (for Python) or diskSizeGb (for Java) entries in Setting other Dataflow pipeline options.

Disk space errors such as "RESOURCE_EXHAUSTED: IO error: No space left on disk"

These errors usually indicate that you have allocated insufficient local disk space to process your job. If you are running your job with default settings, your job is running on 3 workers, each with 25 GB of local disk space, and with no autoscaling. Consider modifying the default settings to increase the number of workers available to your job, to increase the default disk size per worker, or to enable autoscaling.

"Bad request" warnings in shuffler logs

During the execution of a Dataflow job, Cloud Monitoring logs might display a series of warnings similar to:

Unable to update setup work item <step_id> error: generic::invalid_argument: Http(400) Bad Request
Update range task returned 'invalid argument'. Assuming lost lease for work with id <lease_id>
with expiration time: <timestamp>, now: <timestamp>. Full status: generic::invalid_argument: Http(400) Bad Request

"Bad request" warnings appear because worker state information is stale or out of sync due to processing delays. Often, your Dataflow job will succeed despite these "bad request" warnings. If that is the case, ignore the warnings.

A hot key <hot-key-name> was detected in...

These errors identify a hot key in your data. A hot key is a key with enough elements to negatively impact pipeline performance. These keys limit Dataflow's ability to process elements in parallel, which increases execution time.

To log the presence of a hot key, use the hot key pipeline option.

Check that your data is evenly distributed. If a key has disproportionately many values, consider the following courses of action:

Python custom container pipeline fails with SystemError: unknown opcode

If you encounter an unexpected Python error SystemError: unknown opcode immediately after job submission, make sure that the Python version that you are using locally matches the version on the container.

Input and output errors

The I/O metric charts use canonical error codes. If these error codes persist in your sources and sinks, refer to the following list for potential causes and actions you can take.

  • RESOURCE_EXHAUSTED. The project might have run out of resource quota for the service the source or sink is using.

    If the error occurs occasionally or when the Requests per sec chart indicates a high volume of requests being made, then this might indicate that you have reached an API rate limiting quota and need to increase the quota.

  • DEADLINE_EXCEEDED. Source or sink might have timed out reading or writing a large batch of data. Check the latency chart and worker logs. If the error persists, contact support.

  • INVALID_ARGUMENT. Parameters specified to the source or sink might be malformed (such as a Pub/Sub topic). Check configuration of the source or sink, and check the worker logs.

  • FAILED_PRECONDITION. Check configuration of the source or sink, and check the worker logs. This could also indicate a bug.

  • OUT_OF_RANGE. Check that the resource being used by the source or sink exists (such as a Pub/Sub topic or subscription).

  • UNAUTHENTICATED. Check that the Dataflow service account has Identity and Access Management permissions to the specific service and relevant APIs are enabled for the project.

  • PERMISSION_DENIED. Check that the Dataflow service account has Identity and Access Management permissions to the specific service and relevant APIs are enabled for the project.

  • NOT_FOUND. Check that the entities being used by the source or sink exist (such as a Pub/Sub topic or subscription).

  • ABORTED. Service might not be properly handling the source or sinks attempts to read or write data. If the error persists, contact support.

  • ALREADY_EXISTS. I/O might be trying to create an entity which already exists (such as a Pub/Sub topic or subscription). If the error persists, contact support.

  • CANCELLED. This can occur when a Dataflow worker is shut down or source or sink logic intentionally decides to cancel attempts to read or write data.

  • DATALOSS. Indicates unrecoverable data loss or corruption occurred. You might want to create a new dataset for your sources and rerun the Dataflow job.

    You might also see if there are any backup and restoring instructions available for the underlying Google Cloud service.

  • UNKNOWN. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • INTERNAL. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • UNAVAILABLE. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • UNIMPLEMENTED. The source or sink attempted to use the service in an invalid way. Your pipeline might be misconfigured. If the error persists, contact support.

Worker startup errors

Errors in the log types dataflow.googleapis.com/worker-startup, dataflow.googleapis.com/harness-startup, and dataflow.googleapis.com/kubelet indicate configuration problems with a job. They can also indicate conditions that prevent the normal logging path from functioning.

NoClassDefFound errors

These errors can also be of the form Unable to initialize main class [...]. NoClassDefFound errors indicate that the JAR files being uploaded to the Dataflow service do not contain the required classes to run Dataflow. This most frequently occurs when the --filesToStage PipelineOption is overridden and some required JAR or class files are omitted.

Error syncing pod ... failed to "StartContainer"

This error indicates that a worker Docker container failed to start. This typically happens when one of the containers is continuously crashing on startup, or a container image cannot be pulled.

If the container is crashing, look for an error message describing the failure in worker-startup, harness-startup or docker logs. Startup crashes are commonly caused by dependency issues with Python jobs, which can be due to incompatible specified dependencies or network issues that cause the specified repositories to be unreachable. Note that Apache Beam Python SDK installed in the container should be the same as the SDK used to launch the job.

If you are using a custom container image with your job and the workers are unable to pull the image, verify that the image name and tag are correct and that the image is accessible by the Dataflow workers. To verify, locally run docker pull image:tag or use ssh to connect with a running VM and run the same command to rule out authentication issues on the worker.

"A fatal error has been detected by the Java Runtime Environment"

This indicates that the pipeline is using Java Native Interface (JNI) to run non-Java code and that there is an error in that code or the JNI bindings.

Logs

I don't see any logs

If you don't see any logs for your jobs, remove any exclusion filters containing resource.type="dataflow_step" from all of your Cloud Logging Logs Router sinks.

Go to Logs Router

For more details on removing your logs exclusions, refer to the Removing exclusions guide.

Recommendations

High Streaming Engine transfer quota usage

Jobs using Streaming Engine have a regional limit to how much data they can transfer to/from Streaming Engine. It is important to stay below the limit since reaching it may affect job performance. Also, reducing the amount of bytes transferred may reduce billed cost.

To view your project's limit, see the instructions in Quotas & limits. To view your jobs' current usage, follow this link to Metrics Explorer. If necessary, select the project where your job is running. Then replace REGION with the region you're interested in, and click "Run Query".

Suggestions to reduce usage:

  • Reduce the amount of data processed by GroupByKey and Combine transforms. This could be done by reducing the number of elements passed or omitting unnecessary fields.

  • Encode data more efficiently to take up less space. In some cases this may increase CPU usage on the worker.

    • For custom data types, the default coder may be inefficient (SerializableCoder in Java, PickleCoder in Python). Try selecting a more efficient default coder for the custom type, or a PCollection with a schema.

      For a more detailed explanation on coders and how to specify them, see the Beam Programming Guide chapter on data encoding. To learn more about PCollections with schemas, see the chapter on schemas.

    • For large elements, applying compression could help.

  • Reduce large state reads. For example, a pipeline that has a large enough BagState will need to transfer it from Streaming Engine whenever it is fetched.

    • Try reducing state size.
    • Try reducing the frequency of state fetches.
    • State is tied to a specific key and window. Using non-global windowing could be used to limit state size.

    To learn more about state, see the Beam Programming Guide chapter on State and Timers.

Autoscaling: maxNumWorkers could be increased

Dataflow has detected that a job is using the maximum number of allowed workers, maxNumWorkers (or max_num_workers), and that it might utilize more workers if this maximum was raised. For example, this recommendation would be given for a job that has maxNumWorkers set to 50, and is using 50 workers with an average worker CPU utilization above 80%.

Typically, increasing maxNumWorkers increases pipeline throughput: a batch pipeline could complete in less time, and a streaming pipeline could handle larger spikes in data and process more elements per second. However, this may come at an increased cost (see Worker resources pricing). See the Autoscaling guide for details on how the Autoscaling algorithm works and how to configure it.

Suggestions:

  • Try increasing the maxNumWorkers pipeline option, or removing it. Without the option, Dataflow will use the defaults listed in the Autoscaling guide.
  • It's okay to do nothing if pipeline performance is adequate.
    • For batch pipelines, check that the total running time meets your requirements.
    • For streaming pipelines, check the Data freshness graph in the Job Metrics tab on the job page. Make sure it isn't continuously increasing, and that the values are within acceptable bounds.