Common error guidance

This page describes some common errors you might encounter when running your Dataflow job, and suggests some courses of action for dealing with those errors.

Job messages:

Job submission errors:

Worker logs (Stackdriver):

Job messages

"The job failed because a work item failed 4 times."

If a single operation causes the worker code to fail four times, by the code throwing an exception or crashing, Dataflow fails the job, and the message a work item failed 4 times is displayed. You can't configure this failure threshold.

Look in the job's Stackdriver logs for the four individual failures. Look for Error-level or Fatal-level log entries in the worker logs that show exceptions or errors. You should see at least four of these exceptions or errors.

Encoding errors, IOExceptions, or unexpected behavior in user code.

The Apache Beam SDKs and the Dataflow workers depend on common third-party components. These components import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible. You may need to pin to the listed versions that will be in scope during execution. SDK and Worker Dependencies contains a list of dependencies and their required versions.

Jobs that used to run, now fail with "Staged package...is inaccessible"

Verify that the Cloud Storage bucket used for staging does not have TTL settings that cause staged packages to be deleted.

Job submission errors

"413 Request Entity Too Large" / "The size of serialized JSON representation of the pipeline exceeds the allowable limit"

If you encounter an error about the JSON payload when submitting your job, it means your pipeline's JSON representation exceeds the maximum 20 MB request size. These errors might appear as one of the following messages in your console or terminal window:

  • 413 Request Entity Too Large
  • "The size of serialized JSON representation of the pipeline exceeds the allowable limit"
  • "Failed to create a workflow job: Invalid JSON payload received"
  • "Failed to create a workflow job: Request payload exceeds the allowable limit"

The size of your job is specifically tied to the JSON representation of the pipeline. A larger pipeline means a larger request. Dataflow currently has a limitation that caps requests at 20 MB.

To estimate the size of your pipeline's JSON request, run your pipeline with the following option:

Java: SDK 2.x

--dataflowJobFile=< path to output file >

Python

--dataflow_job_file=< path to output file >

Java: SDK 1.x

This command writes a JSON representation of your job to a file. The size of the serialized file is a good estimate of the size of the request. The actual size will be slightly larger due to some additional information included the request.

Certain conditions in your pipeline can cause the JSON representation to exceed the limit. Common conditions include:

  • A Create transform that includes a large amount of in-memory data.
  • A large DoFn instance that is serialized for transmission to remote workers.
  • A DoFn as an anonymous inner class instance that (possibly inadvertently) pulls in a large amount of data to be serialized.

To avoid these conditions, consider restructuring your pipeline.

"The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs."

Your job's graph size must not exceed 10 MB. Certain conditions in your pipeline can cause the job graph to exceed the limit. Common conditions include:

  • A Create transform that includes a large amount of in-memory data.
  • A large DoFn instance that is serialized for transmission to remote workers.
  • A DoFn as an anonymous inner class instance that (possibly inadvertently) pulls in a large amount of data to be serialized.

To avoid these conditions, consider restructuring your pipeline.

"Total number of BoundedSource objects generated by splitIntoBundles() operation is larger than the allowable limit" or "Total size of the BoundedSource objects generated by splitIntoBundles() operation is larger than the allowable limit".

Java: SDK 2.x

You might encounter this error if you're reading from a very large number of files via TextIO, AvroIO, or some other file-based source. The particular limit depends on the details of your source (e.g. embedding schema in AvroIO.Read will allow fewer files), but it is on the order of tens of thousands of files in one pipeline.

You might also encounter this error if you've created a custom data source for your pipeline and your source's splitIntoBundles method returned a list of BoundedSource objects which takes up more than 20 MB when serialized.

The allowable limit for the total size of the BoundedSource objects generated by your custom source's splitIntoBundles() operation is 20 MB. You can work around this limitation by modifying your custom BoundedSource subclass so that the total size of the generated BoundedSource objects is smaller than the 20 MB limit. For example, your source might generate fewer splits initially, and rely on Dynamic Work Rebalancing to further split inputs on demand.

Java: SDK 1.x

"Startup of the worker pool in zone $ZONE failed to bring up any of the desired $N workers. The project quota may have been exceeded or access control policies may be preventing the operation; review the Stackdriver Logging "GCE VM Instance" log for diagnostics."

This error has two possible causes:

  • You might have exceeded one of the Compute Engine quotas that Dataflow worker creation relies on.
  • Your organization has constraints in place that prohibit some aspect of the VM instance creation process, like the account being used, or the zone being targeted.

To troubleshoot this issue:

Review the GCE VM Instance log

  1. Go to the Cloud Logging viewer
  2. In the Audited Resource drop-down list, select GCE VM Instance.
  3. In the All logs drop-down list, select compute.googleapis.com/activity_log.
  4. Scan the log for any entries related to VM instance creation failure.

Check your usage of Compute Engine quotas

  1. On your local machine command line or in Cloud Shell, run the following command to view Compute Engine resource usage compared to Dataflow quotas for the zone you are targeting:

    gcloud compute regions describe [REGION]

  2. Review the results for the following resources to see if any are exceeding quota:

    • CPUS
    • DISKS_TOTAL_GB
    • IN_USE_ADDRESSES
    • INSTANCE_GROUPS
    • INSTANCE_GROUP_MANAGERS
    • INSTANCES

  3. If needed, request a quota change.

Review your organization policy constraints

  1. Go to the Organization policies page
  2. Review the constraints for any that might limit VM instance creation for either the account you are using (this is the Dataflow service account by default) or in the zone you are targeting.

Worker logs

"Processing stuck in step <step_id> for at least <time_interval> without outputting or completing in state finish at <stack_trace>"

If Dataflow spends more time executing a DoFn than the time specified in <time_interval> without returning, this message is displayed.

This error has two possible causes:

  • Your DoFn code is simply slow, or waiting for some slow external operation to complete.
  • Or, your DoFn code might be stuck, deadlocked, or abnormally slow to finish processing.

To determine which of these is the case, expand the Cloud Monitoring log entry to see a stack trace. Look for messages that indicate that the DoFn code is stuck or otherwise encountering issues. If none are present, the issue might be the execution speed of the DoFn code. Consider using a code profiler or other tool to investigate the code's performance.

You can further investigate the cause of your stuck code if your pipeline is built on the Java VM (using either Java or Scala); take a full thread dump of the whole JVM (not just the stuck thread) by following these steps:

  1. Make note of the worker name from the log entry.
  2. In the Compute Engine section of the Cloud Console, find the Compute Engine instance with the worker name you noted.
  3. SSH into the instance with that name.

  4. Run the following command:

    curl http://localhost:8081/threadz

Shuffle key too large exceptions

If you encounter exceptions indicating the shuffle key is too large, it means that the serialized key emitted to a particular (Co-)GroupByKey, after the corresponding coder is applied, is too large. Dataflow has a limit for serialized shuffle keys. Consider reducing the size of the keys or using more space-efficient coders.

RPC timeout exceptions, "DEADLINE_EXCEEDED" exceptions, or "Server Unresponsive" errors

If you encounter RPC timeouts, DEADLINE_EXCEEDED exceptions, or Server Unresponsive errors while your job runs, these typically indicate one of two problems:

  • The VPC network used for your job might be missing a firewall rule. The firewall rule needs to enable all TCP traffic among VMs in the VPC network you specified in your pipeline options. See Specifying your network and subnetwork for more details.

  • Your job is shuffle-bound. Consider one of, or a combination of, the following courses of action:

    Java: SDK 2.x

    • Add more workers. Try setting --numWorkers with a higher value when you run your pipeline.
    • Increase the size of the attached disk for workers. Try setting --diskSizeGb with a higher value when you run your pipeline.
    • Use an SSD-backed persistent disk. Try setting --workerDiskType="compute.googleapis.com/projects/<project>/zones/<zone>/diskTypes/pd-ssd" when you run your pipeline.

    Python

    • Add more workers. Try setting --num_workers with a higher value when you run your pipeline.
    • Increase the size of the attached disk for workers. Try setting --disk_size_gb with a higher value when you run your pipeline.
    • Use an SSD-backed persistent disk. Try setting --worker_disk_type="compute.googleapis.com/projects/<project>/zones/<zone>/diskTypes/pd-ssd" when you run your pipeline.

    Java: SDK 1.x

A call from the Java worker harness to a Python DoFn failed with the error <error message>

If a DoFn implemented in Python fails and throws an exception, a relevant error message is displayed.

Expand the Stackdriver error log entry and look at the error message and traceback. It shows you which code failed so you can correct it if necessary. If you believe that this is a bug in Apache Beam or Dataflow, please report the bug.

No space left on device

If a job shuffles a large amount of data or writes temporary data to local disks, the worker persistent storage might run out of free space. It is also possible for the worker boot disk to fill up if it is logging more than 50 entries per second.

You can track disk space usage on the worker VM instances by using Cloud Monitoring. See Receive worker VM metrics from the Monitoring agent for instructions on setting this up.

You can also look for boot disk space issues by Viewing serial port output on the worker VM instances and looking for messages like "Failed to open system journal: No space left on device". If you have a large number of worker VM instances, you might want to create a script to run gcloud compute instances get-serial-port-output on all of them at once and review the output from that instead.

To increase persistent disk or boot disk space, follow the guidance in the disk_size_gb (for Python) or diskSizeGb (for Java) entries in Setting other Cloud Dataflow pipeline options.

Disk space errors such as "RESOURCE_EXHAUSTED: IO error: No space left on disk"

These errors usually indicate that you have allocated insufficient local disk space to process your job. If you are running your job with default settings, your job is running on 3 workers, each with 25 GB of local disk space, and with no autoscaling. Consider modifying the default settings to increase the number of workers available to your job, to increase the default disk size per worker, or to enable autoscaling.

"Bad request" warnings in shuffler logs

During the execution of a Dataflow job, Stackdriver logs might display a series of warnings similar to:

Unable to update setup work item <step_id> error: generic::invalid_argument: Http(400) Bad Request
Update range task returned 'invalid argument'. Assuming lost lease for work with id <lease_id>
with expiration time: <timestamp>, now: <timestamp>. Full status: generic::invalid_argument: Http(400) Bad Request

"Bad request" warnings appear because worker state information is stale or out of sync due to processing delays. Often, your Dataflow job will succeed despite these "bad request" warnings. If that is the case, ignore the warnings.

A hot key was detected in step <step_id> with age of <time_interval>s

These errors indicate that you have a hot key. A hot key is a key with enough elements to negatively impact pipeline performance. These keys limit Dataflow's ability to process elements in parallel, which increases execution time.

Check that your data is evenly distributed. If a key has disproportionately many values, consider the following courses of action:

I/O debuggability errors

The I/O debuggability charts use canonical error codes. If these error codes persist in your sources and sinks, refer to the following list for potential causes and actions you can take.

  • RESOURCE_EXHAUSTED. The project might have run out of resource quota for the service the source or sink is using.

    If the error occurs occasionally or when the Requests per sec chart indicates a high volume of requests being made, then this might indicate an API rate limiting quota has been hit and might need to be increased.

  • INVALID_ARGUMENT. Parameters specified to the source or sink might be malformed (such as a Pub/Sub topic). Check configuration of the source or sink, and check the worker logs.

  • FAILED_PRECONDITION. Check configuration of the source or sink, and check the worker logs. This could also indicate a bug.

  • OUT_OF_RANGE. Check that the resource being used by the source or sink exists (such as a Pub/Sub topic or subscription).

  • UNAUTHENTICATED. Check that the Dataflow service account has Cloud Identity and Access Management permissions to the specific service and relevant APIs are enabled for the project.

  • PERMISSION_DENIED. Check that the Dataflow service account has Cloud Identity and Access Management permissions to the specific service and relevant APIs are enabled for the project.

  • NOT_FOUND. Check that the entities being used by the source or sink exist (such as a Pub/Sub topic or subscription).

  • ABORTED. Service might not be properly handling the source or sinks attempts to read or write data. If the error persists, contact support.

  • ALREADY_EXISTS. I/O might be trying to create an entity which already exists (such as a Pub/Sub topic or subscription). If the error persists, contact support.

  • CANCELLED. This can occur when a Dataflow worker is shut down or source or sink logic intentionally decides to cancel attempts to read or write data.

  • DATALOSS. Indicates unrecoverable data loss or corruption occurred. You might want to create a new dataset for your sources and rerun the Dataflow job.

    You might also see if there are any backup and restoring instructions available for the underlying Google Cloud service.

  • UNKNOWN. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • INTERNAL. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • UNAVAILABLE. Service might be down. Check for updates on Cloud Status Dashboard for more information.

  • UNIMPLEMENTED. The source or sink attempted to use the service in an invalid way. Your pipeline might be misconfigured. If the error persists, contact support.