SDK and Worker Dependencies

The Apache Beam SDKs and the Cloud Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

Apache Beam SDKs

SDK for Java

Version 2.8.0

Apache Beam SDK for Java 2.8.0 has the following compile and runtime dependencies.

GroupIdArtifactIdVersion
activemq-amqp org.apache.activemq 5.13.1
activemq-broker org.apache.activemq 5.13.1
activemq-client org.apache.activemq 5.13.1
activemq-jaas org.apache.activemq 5.13.1
activemq-junit org.apache.activemq.tooling 5.13.1
activemq-kahadb-store org.apache.activemq 5.13.1
activemq-mqtt org.apache.activemq 5.13.1
apex-common org.apache.apex 3.7.0
apex-engine org.apache.apex 3.7.0
api-common com.google.api 1.6.0
args4j args4j 2.33
avro org.apache.avro 1.8.2
bigtable-client-core com.google.cloud.bigtable 1.4.0
byte-buddy net.bytebuddy 1.8.11
commons-compress org.apache.commons 1.16.1
commons-csv org.apache.commons 1.4
commons-io commons-io 1.3.2
commons-io commons-io 2.5
commons-lang3 org.apache.commons 3.6
commons-math3 org.apache.commons 3.6.1
datastore-v1-proto-client com.google.cloud.datastore 1.6.0
error_prone_annotations com.google.errorprone 2.0.15
gax-grpc com.google.api 1.29.0
gcsio com.google.cloud.bigdataoss 1.9.0
google-api-client-jackson2 com.google.api-client 1.23.0
google-api-client-java6 com.google.api-client 1.23.0
google-api-client com.google.api-client 1.23.0
google-api-services-bigquery com.google.apis v2-rev402-1.23.0
google-api-services-clouddebugger com.google.apis v2-rev253-1.23.0
google-api-services-cloudresourcemanager com.google.apis v1-rev502-1.23.0
google-api-services-dataflow com.google.apis v1b3-rev257-1.23.0
google-api-services-pubsub com.google.apis v1-rev399-1.23.0
google-api-services-storage com.google.apis v1-rev136-1.23.0
google-auth-library-credentials com.google.auth 0.10.0
google-auth-library-oauth2-http com.google.auth 0.10.0
google-cloud-core-grpc com.google.cloud 1.36.0
google-cloud-core com.google.cloud 1.36.0
google-cloud-dataflow-java-proto-library-all com.google.cloud.dataflow 0.5.160304
google-cloud-spanner com.google.cloud 0.54.0-beta
google-http-client-jackson2 com.google.http-client 1.23.0
google-http-client-jackson com.google.http-client 1.23.0
google-http-client-protobuf com.google.http-client 1.23.0
google-http-client com.google.http-client 1.23.0
google-oauth-client-java6 com.google.oauth-client 1.23.0
google-oauth-client com.google.oauth-client 1.23.0
grpc-all io.grpc 1.13.1
grpc-auth io.grpc 1.13.1
grpc-core io.grpc 1.13.1
grpc-google-cloud-bigtable-v2 com.google.api.grpc 0.19.0
grpc-google-cloud-pubsub-v1 com.google.api.grpc 1.18.0
grpc-netty io.grpc 1.13.1
grpc-protobuf-lite io.grpc 1.13.1
grpc-protobuf io.grpc 1.13.1
grpc-stub io.grpc 1.13.1
guava com.google.guava 20.0
guava-testlib com.google.guava 20.0
hadoop-client org.apache.hadoop 2.7.3
hadoop-common org.apache.hadoop 2.7.3
hadoop-hdfs org.apache.hadoop 2.7.3
hadoop-hdfs org.apache.hadoop 2.7.3
hadoop-mapreduce-client-core org.apache.hadoop 2.7.3
hadoop-minicluster org.apache.hadoop 2.7.3
hamcrest-core org.hamcrest 1.3
hamcrest-library org.hamcrest 1.3
jackson-annotations com.fasterxml.jackson.core 2.9.5
jackson-core com.fasterxml.jackson.core 2.9.5
jackson-databind com.fasterxml.jackson.core 2.9.5
jackson-dataformat-cbor com.fasterxml.jackson.dataformat 2.9.5
jackson-dataformat-yaml com.fasterxml.jackson.dataformat 2.9.5
jackson-datatype-joda com.fasterxml.jackson.datatype 2.9.5
jackson-module-scala_2.11 com.fasterxml.jackson.module 2.9.5
jaxb-api javax.xml.bind 2.2.12
joda-time joda-time 2.4
junit-quickcheck-core com.pholser 0.8
junit junit 4.12
kafka_2.11 org.apache.kafka 1.0.0
kafka-clients org.apache.kafka 1.0.0
malhar-library org.apache.apex 3.4.0
mockito-core org.mockito 1.10.19
netty-handler io.netty 4.1.25.Final
netty-tcnative-boringssl-static io.netty 2.0.8.Final
netty-transport-native-epoll io.netty 4.1.25.Final
postgresql org.postgresql 42.2.2
powermock-mockito-release-full org.powermock 1.6.4
protobuf-java com.google.protobuf 3.6.0
protobuf-java-util com.google.protobuf 3.6.0
proto-google-cloud-datastore-v1 com.google.api.grpc 0.19.0
proto-google-cloud-pubsub-v1 com.google.api.grpc 1.18.0
proto-google-cloud-spanner-admin-database-v1 com.google.api.grpc 0.19.0
proto-google-common-protos com.google.api.grpc 1.12.0
slf4j-api org.slf4j 1.7.25
slf4j-jdk14 org.slf4j 1.7.25
slf4j-log4j12 org.slf4j 1.7.25
slf4j-simple org.slf4j 1.7.25
snappy-java org.xerial.snappy 1.1.4
spark-core_2.11 org.apache.spark 2.3.1
spark-network-common_2.11 org.apache.spark 2.3.1
spark-streaming_2.11 org.apache.spark 2.3.1
stax2-api org.codehaus.woodstox 3.1.4
util com.google.cloud.bigdataoss 1.9.0
woodstox-core-asl org.codehaus.woodstox 4.4.1

Workers have the following packages already installed.

GroupIdArtifactIdVersion
com.google.api-client google-api-client 1.23.0
com.google.oauth-client google-oauth-client 1.23.0
com.google.http-client google-http-client-jackson2 1.23.0
com.google.apis google-api-services-dataflow v1b3-rev221-1.23.0
org.apache.beam beam-model-pipeline 2.8.0
org.apache.beam beam-model-fn-execution 2.8.0
org.apache.beam beam-sdks-java-extensions-google-cloud-platform-core 2.8.0
com.google.auth google-auth-library-oauth2-http 0.7.1
com.google.cloud.bigdataoss gcsio 1.4.5
com.google.api-client google-api-client-java6 1.20.0
com.google.api-client google-api-client-jackson2 1.20.0
com.google.oauth-client google-oauth-client-java6 1.20.0
com.google.cloud.bigdataoss util 1.4.5
com.google.apis google-api-services-cloudresourcemanager v1-rev477-1.23.0
com.google.apis google-api-services-storage v1-rev124-1.23.0
com.google.auth google-auth-library-credentials 0.7.1
org.apache.beam beam-sdks-java-core 2.8.0
org.xerial.snappy snappy-java 1.1.4
org.tukaani xz 1.5
org.apache.beam beam-sdks-java-io-google-cloud-platform 2.8.0
org.apache.beam beam-sdks-java-extensions-protobuf 2.8.0
com.google.apis google-api-services-bigquery v2-rev374-1.23.0
com.google.api gax-grpc 0.20.0
com.google.api gax 1.3.1
org.threeten threetenbp 1.3.3
com.google.cloud google-cloud-core-grpc 1.2.0
com.google.apis google-api-services-pubsub v1-rev382-1.23.0
com.google.api.grpc grpc-google-cloud-pubsub-v1 0.1.18
com.google.cloud.datastore datastore-v1-proto-client 1.4.0
com.google.http-client google-http-client-protobuf 1.20.0
com.google.cloud.datastore datastore-v1-protos 1.3.0
com.google.api.grpc grpc-google-common-protos 0.1.0
com.google.cloud google-cloud-core 1.0.2
org.json json 20160810
com.google.cloud google-cloud-spanner 0.20.0b-beta
com.google.api.grpc proto-google-cloud-spanner-v1 0.1.11b
com.google.api.grpc proto-google-cloud-spanner-admin-instance-v1 0.1.11
com.google.api.grpc grpc-google-cloud-spanner-v1 0.1.11b
com.google.api.grpc grpc-google-cloud-spanner-admin-database-v1 0.1.11
com.google.api.grpc grpc-google-cloud-spanner-admin-instance-v1 0.1.11
com.google.api.grpc grpc-google-longrunning-v1 0.1.11
com.google.api.grpc proto-google-longrunning-v1 0.1.11
com.google.cloud.bigtable bigtable-protos 1.0.0-pre3
com.google.cloud.bigtable bigtable-client-core 1.0.0
commons-logging commons-logging 1.2
com.google.auth google-auth-library-appengine 0.7.0
io.opencensus opencensus-contrib-grpc-util 0.7.0
io.dropwizard.metrics metrics-core 3.1.2
com.google.api.grpc proto-google-cloud-spanner-admin-database-v1 0.1.9
com.google.api.grpc proto-google-common-protos 0.1.9
io.grpc grpc-all 1.2.0
io.grpc grpc-okhttp 1.2.0
com.squareup.okhttp okhttp 2.5.0
com.squareup.okio okio 1.6.0
io.grpc grpc-protobuf-nano 1.2.0
com.google.protobuf.nano protobuf-javanano 3.0.0-alpha-5
org.apache.beam beam-runners-core-construction-java 2.8.0
org.apache.beam beam-model-job-management 2.8.0
org.apache.beam beam-runners-core-java 2.8.0
org.apache.beam beam-runners-java-fn-execution 2.8.0
org.apache.beam beam-vendor-sdks-java-extensions-protobuf 2.8.0
org.apache.beam beam-sdks-java-fn-execution 2.8.0
org.apache.beam beam-runners-google-cloud-dataflow-java 2.8.0
com.google.apis google-api-services-clouddebugger v2-rev233-1.23.0
com.fasterxml.jackson.core jackson-annotations 2.9.5
com.fasterxml.jackson.core jackson-databind 2.9.5
com.fasterxml.jackson.core jackson-core 2.9.5
joda-time joda-time 2.4
org.apache.avro avro 1.8.2
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
com.thoughtworks.paranamer paranamer 2.7
org.apache.commons commons-compress 1.8.1
org.slf4j slf4j-api 1.7.25
com.google.code.findbugs jsr305 3.0.1
com.google.auto.service auto-service 1.0-rc2
com.google.auto auto-common 0.3
com.google.auto.value auto-value 1.5.3
com.google.http-client google-http-client 1.23.0
org.apache.httpcomponents httpclient 4.0.1
org.apache.httpcomponents httpcore 4.0.1
commons-codec commons-codec 1.3
com.google.http-client google-http-client-jackson 1.23.0
com.google.api.grpc proto-google-cloud-pubsub-v1 0.1.18
com.google.api api-common 1.1.0
com.google.api.grpc proto-google-iam-v1 0.1.18
com.google.guava guava 20.0
com.google.guava guava-testlib 20.0
com.google.errorprone error_prone_annotations 2.0.15
com.google.protobuf protobuf-java 3.2.0
io.grpc grpc-core 1.2.0
io.grpc grpc-context 1.2.0
com.google.instrumentation instrumentation-api 0.3.0
io.grpc grpc-stub 1.2.0
io.grpc grpc-netty 1.2.0
io.netty netty-codec-http2 4.1.8.Final
io.netty netty-codec-http 4.1.8.Final
io.netty netty-handler-proxy 4.1.8.Final
io.netty netty-codec-socks 4.1.8.Final
io.grpc grpc-protobuf 1.2.0
com.google.protobuf protobuf-java-util 3.2.0
com.google.code.gson gson 2.7
io.grpc grpc-protobuf-lite 1.2.0
io.grpc grpc-auth 1.2.0
io.netty netty-handler 4.1.8.Final
io.netty netty-buffer 4.1.8.Final
io.netty netty-transport 4.1.8.Final
io.netty netty-resolver 4.1.8.Final
io.netty netty-codec 4.1.8.Final
io.netty netty-common 4.1.8.Final
io.netty netty-transport-native-epoll linux-x86_64
io.netty netty-tcnative-boringssl-static 1.1.33.Fork18
org.conscrypt conscrypt-openjdk linux-x86_64
org.eclipse.jetty jetty-server 9.2.10.v20150310
org.eclipse.jetty jetty-http 9.2.10.v20150310
org.eclipse.jetty jetty-util 9.2.10.v20150310
org.eclipse.jetty jetty-io 9.2.10.v20150310
org.eclipse.jetty jetty-servlet 9.2.10.v20150310
org.eclipse.jetty jetty-security 9.2.10.v20150310
javax.servlet javax.servlet-api 3.1.0
org.slf4j slf4j-jdk14 1.7.25
org.apache.beam beam-sdks-java-core 2.8.0
org.apache.beam beam-sdks-java-extensions-google-cloud-platform-core 2.8.0
org.apache.beam beam-runners-core-java 2.8.0
org.apache.beam beam-runners-direct-java 2.8.0
args4j args4j 2.33
org.hamcrest hamcrest-all 1.3
junit junit 4.12
org.hamcrest hamcrest-core 1.3
org.mockito mockito-all 1.10.19
io.opencensus opencensus-api 0.12.2

SDK for Python

Version 2.8.0

Apache Beam SDK for Python 2.8.0 has the following compile and runtime dependencies.

PackageVersion
avro>=1.8.1, <2.0.0
crcmod>=1.7, <2.0
dill>=0.2.6, <=0.2.8.2
fastavro>=0.21.4, <0.22
future>=0.16.0, <1.0.0
futures>=3.1.1, <4.0.0
grpcio>=1.8, <2
hdfs>=2.1.0, <3.0.0
httplib2>=0.8, <=0.11.3
mock>=1.0.1, <3.0.0
oauth2client>=2.0.1, <5
protobuf>=3.5.0.post1, <4
pydot>=1.2.0, <1.3
pytz>=2018.3, <=2018.4
pyyaml>=3.12, <4.0.0
pyvcf>=0.6.8, <0.7.0
six>=1.9, <1.12
typing>=3.6.0, <3.7.0

Workers have the following packages already installed.

PackageVersion
avro1.8.2
beautifulsoup44.5.1
bs40.0.1
crcmod1.7
cython0.28.1
dill0.2.6
fastavro0.21.4
future0.16.0
futures3.2.0
grpcio1.15.0
google-apitools0.5.20
google-cloud-bigquery0.25.0
google-cloud-pubsub0.26.0
google-python-cloud-debugger1.9.1
googledatastore7.0.1
guppy0.1.10
hdfs2.1.0
httplib20.9.2
mock2.0.0
nltk3.2.1
nose1.3.7
numpy1.15.2
oauth2client3.0.0
pandas0.23.4
pillow3.4.1
proto-google-cloud-datastore-v10.90.4
proto-google-cloud-pubsub-v10.15.4
protobuf3.6.1
protorpc0.11.1
python-gflags3.0.6
python-snappy0.5.1
pytz2018.4
pyvcf0.6.8
pyyaml3.12
requests2.18.4
scikit-learn0.19.1
scipy1.0.0
six1.10.0
tensorflow1.11.0
typing3.6.1
wheel0.32.1

Cloud Dataflow SDKs

SDK for Java

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

SDK for Python

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow
Need help? Visit our support page.