SDK and Worker Dependencies

The Cloud Dataflow SDKs and workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

SDK for Java 2.x

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

Version 2.4.0

Cloud Dataflow SDK for Java 2.4.0 has the following compile and runtime dependencies.

GroupIdArtifactIdVersion
com.fasterxml.jackson.corejackson-annotations2.8.9
com.fasterxml.jackson.corejackson-core2.8.9
com.fasterxml.jackson.corejackson-databind2.8.9
com.github.stephenc.findbugsfindbugs-annotations1.3.9-1
com.google.apiapi-common1.1.0
com.google.api-clientgoogle-api-client1.22.0
com.google.api-clientgoogle-api-client-jackson21.20.0
com.google.api-clientgoogle-api-client-java61.20.0
com.google.apigax1.3.1
com.google.apigax-grpc0.20.0
com.google.api.grpcgrpc-google-cloud-pubsub-v10.1.18
com.google.api.grpcgrpc-google-cloud-spanner-admin-database-v10.1.11
com.google.api.grpcgrpc-google-cloud-spanner-admin-instance-v10.1.11
com.google.api.grpcgrpc-google-cloud-spanner-v10.1.11b
com.google.api.grpcgrpc-google-common-protos0.1.0
com.google.api.grpcgrpc-google-longrunning-v10.1.11
com.google.api.grpcproto-google-cloud-pubsub-v10.1.18
com.google.api.grpcproto-google-cloud-spanner-admin-database-v10.1.9
com.google.api.grpcproto-google-cloud-spanner-admin-instance-v10.1.11
com.google.api.grpcproto-google-cloud-spanner-v10.1.11b
com.google.api.grpcproto-google-common-protos0.1.9
com.google.api.grpcproto-google-iam-v10.1.18
com.google.api.grpcproto-google-longrunning-v10.1.11
com.google.apisgoogle-api-services-bigqueryv2-rev374-1.22.0
com.google.apisgoogle-api-services-clouddebuggerv2-rev8-1.22.0
com.google.apisgoogle-api-services-cloudresourcemanagerv1-rev6-1.22.0
com.google.apisgoogle-api-services-dataflowv1b3-rev221-1.22.0
com.google.apisgoogle-api-services-pubsubv1-rev10-1.22.0
com.google.apisgoogle-api-services-storagev1-rev71-1.22.0
com.google.authgoogle-auth-library-appengine0.7.0
com.google.authgoogle-auth-library-credentials0.7.1
com.google.authgoogle-auth-library-oauth2-http0.7.1
com.google.autoauto-common0.3
com.google.auto.serviceauto-service1.0-rc2
com.google.cloud.bigdataossgcsio1.4.5
com.google.cloud.bigdataossutil1.4.5
com.google.cloud.bigtablebigtable-client-core1.0.0
com.google.cloud.bigtablebigtable-protos1.0.0-pre3
com.google.cloud.datastoredatastore-v1-proto-client1.4.0
com.google.cloud.datastoredatastore-v1-protos1.3.0
com.google.cloudgoogle-cloud-core1.0.2
com.google.cloudgoogle-cloud-core-grpc1.2.0
com.google.cloudgoogle-cloud-spanner0.20.0b-beta
com.google.code.findbugsjsr3053.0.1
com.google.code.gsongson2.7
com.google.guavaguava20.0
com.google.http-clientgoogle-http-client1.22.0
com.google.http-clientgoogle-http-client-jackson1.22.0
com.google.http-clientgoogle-http-client-jackson21.22.0
com.google.http-clientgoogle-http-client-protobuf1.20.0
com.google.instrumentationinstrumentation-api0.3.0
com.google.oauth-clientgoogle-oauth-client1.22.0
com.google.oauth-clientgoogle-oauth-client-java61.20.0
com.google.protobuf.nanoprotobuf-javanano3.0.0-alpha-5
com.google.protobufprotobuf-java3.2.0
com.google.protobufprotobuf-java-util3.2.0
commons-codeccommons-codec1.3
commons-loggingcommons-logging1.2
com.squareup.okhttpokhttp2.5.0
com.squareup.okiookio1.6.0
com.thoughtworks.paranamerparanamer2.7
io.dropwizard.metricsmetrics-core3.1.2
io.grpcgrpc-all1.2.0
io.grpcgrpc-auth1.2.0
io.grpcgrpc-context1.2.0
io.grpcgrpc-core1.2.0
io.grpcgrpc-netty1.2.0
io.grpcgrpc-okhttp1.2.0
io.grpcgrpc-protobuf1.2.0
io.grpcgrpc-protobuf-lite1.2.0
io.grpcgrpc-protobuf-nano1.2.0
io.grpcgrpc-stub1.2.0
io.nettynetty-buffer4.1.8.Final
io.nettynetty-codec4.1.8.Final
io.nettynetty-codec-http24.1.8.Final
io.nettynetty-codec-http4.1.8.Final
io.nettynetty-codec-socks4.1.8.Final
io.nettynetty-common4.1.8.Final
io.nettynetty-handler4.1.8.Final
io.nettynetty-handler-proxy4.1.8.Final
io.nettynetty-resolver4.1.8.Final
io.nettynetty-tcnative-boringssl-static1.1.33.Fork18
io.nettynetty-transport4.1.8.Final
io.nettynetty-transport-native-epolllinux-x86_64
io.opencensusopencensus-api0.7.0
io.opencensusopencensus-contrib-grpc-util0.7.0
javax.servletjavax.servlet-api3.1.0
joda-timejoda-time2.4
org.apache.avroavro1.8.2
org.apache.beambeam-model-fn-execution2.4.0
org.apache.beambeam-model-job-management2.4.0
org.apache.beambeam-model-pipeline2.4.0
org.apache.beambeam-runners-core-construction-java2.4.0
org.apache.beambeam-runners-core-java2.4.0
org.apache.beambeam-runners-google-cloud-dataflow-java2.4.0
org.apache.beambeam-runners-java-fn-execution2.4.0
org.apache.beambeam-sdks-java-core2.4.0
org.apache.beambeam-sdks-java-extensions-google-cloud-platform-core2.4.0
org.apache.beambeam-sdks-java-extensions-protobuf2.4.0
org.apache.beambeam-sdks-java-io-google-cloud-platform2.4.0
org.apache.commonscommons-compress1.8.1
org.apache.httpcomponentshttpclient4.0.1
org.apache.httpcomponentshttpcore4.0.1
org.codehaus.jacksonjackson-core-asl1.9.13
org.codehaus.jacksonjackson-mapper-asl1.9.13
org.eclipse.jettyjetty-http9.2.10.v20150310
org.eclipse.jettyjetty-io9.2.10.v20150310
org.eclipse.jettyjetty-security9.2.10.v20150310
org.eclipse.jettyjetty-server9.2.10.v20150310
org.eclipse.jettyjetty-servlet9.2.10.v20150310
org.eclipse.jettyjetty-util9.2.10.v20150310
org.jsonjson20160810
org.slf4jslf4j-api1.7.25
org.slf4jslf4j-jdk141.7.25
org.threetenthreetenbp1.3.3
org.tukaanixz1.5
org.xerial.snappysnappy-java1.1.4

Additionally, a Cloud Dataflow worker starts with the following logging JARs prepended to the classpath.

GroupIdArtifactIdVersion
org.apache.logging.log4jlog4j-to-slf4j2.5
org.slf4jjcl-over-slf4j1.7.14
org.slf4jlog4j-over-slf4j1.7.14
org.slf4jslf4j-api1.7.14
org.slf4jslf4j-jdk141.7.14

SDK for Python 2.x

Version 2.4.0

Cloud Dataflow SDK for Python 2.4.0 has the following dependencies.

PackageVersion
avro>=1.8.1, <2.0.0
crcmod>=1.7, <2.0
dill0.2.6
futures>=3.1.1, <4.0.0
google-apitools>=0.5.18, <=0.5.20
google-cloud-pubsub0.26.0
google-cloud-bigquery0.25.0
googledatastore7.0.1
grpcio>=1.0, <2.0
hdfs>=2.1.0, <3.0.0
httplib2>=0.8, <0.10
mock>=1.0.1, <3.0.0
nose>=1.0
oauth2client>=2.0.1, <5.0.0
proto-google-cloud-datastore-v1>=0.90.0, <=0.90.4
protobuf>=3.5.0.post1, <4
pyhamcrest>=1.9, <2.0
pyvcf>=0.6.8, <0.7.0
pyyaml>=3.12, <4.0.0
six>=1.9, <1.12
typing>=3.6.0, <3.7.0

Workers have the following packages already installed.

PackageVersion
avro1.8.2
beautifulsoup44.5.1
bs40.0.1
crcmod1.7
cython0.27.3
dill0.2.6
futures3.1.1
grpcio1.3.0
google-apitools0.5.20
google-cloud-bigquery0.25.0
google-cloud-pubsub0.26.0
google-python-cloud-debugger1.9.1
googledatastore7.0.1
guppy0.1.10
hdfs30.3.0
httplib20.9.2
mock2.0.0
nltk3.2.1
nose1.3.7
numpy1.13.3
oauth2client4.1.2
pandas0.18.1
pillow3.4.1
proto-google-cloud-datastore-v10.90.4
protobuf3.5.1
protorpc0.11.1
python-gflags3.0.6
python-snappy0.5.1
pyvcf0.6.8
pyyaml3.12
requests2.18.4
scikit-learn0.19.1
scipy1.0.0
six1.10.0
tensorflow1.6.0
typing3.6.1
wheel0.29.0

SDK for Java 1.x

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

Version 1.9.1

Cloud Dataflow SDK for Java 1.9.1 has the following compile and runtime dependencies.

GroupIdArtifactIdVersion
com.fasterxml.jackson.corejackson-annotations2.7.0
com.fasterxml.jackson.corejackson-core2.7.0
com.fasterxml.jackson.corejackson-databind2.7.0
com.google.api-clientgoogle-api-client-jackson21.22.0
com.google.api-clientgoogle-api-client-java61.22.0
com.google.api-clientgoogle-api-client1.22.0
com.google.api.grpcgrpc-core-proto0.0.3
com.google.api.grpcgrpc-pubsub-v10.0.2
com.google.apisgoogle-api-services-bigqueryv2-rev295-1.22.0
com.google.apisgoogle-api-services-clouddebuggerv2-rev8-1.22.0
com.google.apisgoogle-api-services-dataflowv1b3-rev36-1.22.0
com.google.apisgoogle-api-services-datastore-protobufv1beta2-rev1-4.0.0
com.google.apisgoogle-api-services-pubsubv1-rev10-1.22.0
com.google.apisgoogle-api-services-storagev1-rev71-1.22.0
com.google.authgoogle-auth-library-credentials0.4.0
com.google.authgoogle-auth-library-oauth2-http0.4.0
com.google.auto.serviceauto-service1.0-rc2
com.google.autoauto-common0.3
com.google.cloud.bigdataossgcsio1.4.5
com.google.cloud.bigdataossutil1.4.5
com.google.cloud.bigtablebigtable-protos0.3.0
com.google.cloud.datastoredatastore-v1-proto-client1.1.0
com.google.cloud.datastoredatastore-v1-protos1.0.1
com.google.code.findbugsjsr3053.0.1
com.google.http-clientgoogle-http-client-jackson1.22.0
com.google.http-clientgoogle-http-client-jackson21.22.0
com.google.http-clientgoogle-http-client-protobuf1.22.0
com.google.oauth-clientgoogle-oauth-client-java61.22.0
com.google.oauth-clientgoogle-oauth-client1.22.0
com.google.protobuf.nanoprotobuf-javanano3.0.0-alpha-5
com.google.protobufprotobuf-java3.0.0-beta-1
com.squareup.okhttpokhttp2.5.0
com.squareup.okiookio1.6.0
com.thoughtworks.paranamerparanamer2.3
commons-codeccommons-codec1.3
commons-loggingcommons-logging1.1.1
io.grpcgrpc-all0.13.1
io.grpcgrpc-auth0.13.1
io.grpcgrpc-core0.13.1
io.grpcgrpc-netty0.13.1
io.grpcgrpc-okhttp0.13.1
io.grpcgrpc-protobuf-nano0.13.1
io.grpcgrpc-protobuf0.13.1
io.grpcgrpc-stub0.13.1
io.nettynetty-buffer4.1.0.CR1
io.nettynetty-codec-http4.1.0.CR1
io.nettynetty-codec-http24.1.0.CR1
io.nettynetty-codec4.1.0.CR1
io.nettynetty-common4.1.0.CR1
io.nettynetty-handler4.1.0.CR1
io.nettynetty-resolver4.1.0.CR1
io.nettynetty-transport4.1.0.CR1
joda-timejoda-time2.4
org.apache.avroavro1.7.7
org.apache.commonscommons-compress1.9
org.apache.httpcomponentshttpclient4.0.1
org.apache.httpcomponentshttpcore4.0.1
org.codehaus.jacksonjackson-core-asl1.9.13
org.codehaus.jacksonjackson-mapper-asl1.9.13
org.codehaus.woodstoxstax2-api3.1.4
org.codehaus.woodstoxwoodstox-core-asl4.4.1
org.slf4jslf4j-api1.7.14
org.tukaanixz1.5
org.xerial.snappysnappy-java1.1.2.1

Additionally, a Cloud Dataflow worker starts with the following logging JARs prepended to the classpath.

GroupIdArtifactIdVersion
org.apache.logging.log4jlog4j-to-slf4j2.5
org.slf4jjcl-over-slf4j1.7.14
org.slf4jlog4j-over-slf4j1.7.14
org.slf4jslf4j-api1.7.14
org.slf4jslf4j-jdk141.7.14
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow Documentation