Dataflow pipelines using Scio fails to start container
Stay organized with collections
Save and categorize content based on your preferences.
Problem
Dataflow streaming pipeline does not produce any elements. The worker log contains the following messages:
"... skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "back-off" ... "restarting failed container=java-streaming pod= ..."
"java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: com.fasterxml.jackson.module.scala.DefaultScalaModule Unable to get public no-arg constructor"
Environment
- Beam SDK 2.24.0 for Java
- Scio 0.9.4
- Scala 2.13
Solution
- Wait until Scio supports Beam SDK 2.25.0.
OR
- Experiment with previous Scio and Scala versions depending on the pipeline requirements.
Cause
This can be caused by
BEAM-9399 that signals a potential deadlock between the
DataflowWorkerLoggingHandler and overridden
System.err PrintStream.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-12-12 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-12-12 UTC."],[],[]]
