SDK and Worker Dependencies

The Apache Beam SDKs and the Cloud Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

Apache Beam SDKs

SDK for Java

Version 2.6.0

Apache Beam SDK for Java 2.6.0 has the following compile and runtime dependencies.

GroupIdArtifactIdVersion
activemq-amqp org.apache.activemq 5.13.1
activemq-broker org.apache.activemq 5.13.1
activemq-client org.apache.activemq 5.13.1
activemq-jaas org.apache.activemq 5.13.1
activemq-junit org.apache.activemq.tooling 5.13.1
activemq-kahadb-store org.apache.activemq 5.13.1
activemq-mqtt org.apache.activemq 5.13.1
annotations com.google.code.findbugs 3.0.1
apache-rat-tasks org.apache.rat 0.12
apex-common org.apache.apex 3.7.0
apex-engine org.apache.apex 3.7.0
args4j args4j 2.33
avro org.apache.avro 1.8.2
bigtable-client-core com.google.cloud.bigtable 1.0.0
bigtable-protos com.google.cloud.bigtable 1.0.0-pre3
byte-buddy net.bytebuddy 1.8.11
commons-compress org.apache.commons 1.16.1
commons-csv org.apache.commons 1.4
commons-io commons-io 2.5
commons-lang commons-lang 2.6
commons-lang3 org.apache.commons 3.6
commons-math3 org.apache.commons 3.6.1
creadur-rat-gradle gradle.plugin.org.nosphere.apache 0.3.1
datastore-v1-proto-client com.google.cloud.datastore 1.4.0
datastore-v1-protos com.google.cloud.datastore 1.3.0
embedded-redis com.github.kstyrc 0.6
fmpp-maven-plugin com.googlecode.fmpp-maven-plugin 1.0
gax-grpc com.google.api 0.20.0
gcsio com.google.cloud.bigdataoss 1.4.5
gearpump-core_2.11 org.apache.gearpump 0.8.4
gearpump-streaming_2.11 org.apache.gearpump 0.8.4
geronimo-jms_1.1_spec org.apache.geronimo.specs 1.1.1
gogradle gradle.plugin.com.github.blindpirate 0.10
google-api-client com.google.api-client 1.23.0
google-api-services-bigquery com.google.apis v2-rev374-1.23.0
google-api-services-clouddebugger com.google.apis v2-rev233-1.23.0
google-api-services-cloudresourcemanager com.google.apis v1-rev477-1.23.0
google-api-services-dataflow com.google.apis v1b3-rev221-1.23.0
google-api-services-pubsub com.google.apis v1-rev382-1.23.0
google-api-services-storage com.google.apis v1-rev124-1.23.0
google-auth-library-credentials com.google.auth 0.7.1
google-auth-library-oauth2-http com.google.auth 0.7.1
google-cloud-core com.google.cloud 1.0.2
google-cloud-core-grpc com.google.cloud 1.2.0
google-cloud-dataflow-java-proto-library-all com.google.cloud.dataflow 0.5.160304
google-cloud-spanner com.google.cloud 0.20.0b-beta
google-http-client com.google.http-client 1.23.0
google-http-client-jackson2 com.google.http-client 1.23.0
grpc-all io.grpc 1.2.0
grpc-auth io.grpc 1.2.0
grpc-core io.grpc 1.2.0
grpc-google-cloud-pubsub-v1 com.google.api.grpc 0.1.18
grpc-netty io.grpc 1.2.0
grpc-stub io.grpc 1.2.0
guava-testlib com.google.guava 20.0
hadoop-client org.apache.hadoop 2.7.3
hadoop-common org.apache.hadoop 2.7.3
hadoop-hdfs org.apache.hadoop 2.7.3
hadoop-mapreduce-client-core org.apache.hadoop 2.7.3
hadoop-minicluster org.apache.hadoop 2.7.3
hamcrest-core org.hamcrest 1.3
hamcrest-library org.hamcrest 1.3
hawtbuf org.fusesource.hawtbuf 1.11
jackson-annotations com.fasterxml.jackson.core 2.9.5
jackson-core com.fasterxml.jackson.core 2.9.5
jackson-databind com.fasterxml.jackson.core 2.9.5
jackson-dataformat-cbor com.fasterxml.jackson.dataformat 2.9.5
jackson-dataformat-yaml com.fasterxml.jackson.dataformat 2.9.5
jackson-datatype-joda com.fasterxml.jackson.datatype 2.9.5
jackson-module-scala_2.11 com.fasterxml.jackson.module 2.9.5
javacc-gradle-plugin ca.coglinc 2.4.0
jaxb-api javax.xml.bind 2.2.12
jedis redis.clients 2.9.0
joda-time joda-time 2.4
junit junit 4.12
junit-quickcheck-core com.pholser 0.8
kafka-clients org.apache.kafka 1.0.0
kafka_2.11 org.apache.kafka 1.0.0
log4j-api org.apache.logging.log4j 2.6.2
log4j-core org.apache.logging.log4j 2.6.2
malhar-library org.apache.apex 3.4.0
mockito-core org.mockito 1.9.5
mqtt-client org.fusesource.mqtt-client 1.14
netty-handler io.netty 4.1.8.Final
postgresql org.postgresql 42.2.2
proto-google-cloud-pubsub-v1 com.google.api.grpc 0.1.18
proto-google-cloud-spanner-admin-database-v1 com.google.api.grpc 0.1.9
samza-api org.apache.samza 0.14.1
samza-core_2.11 org.apache.samza 0.14.1
samza-kafka_2.11 org.apache.samza 0.14.1
samza-kv-inmemory_2.11 org.apache.samza 0.14.1
samza-kv-rocksdb_2.11 org.apache.samza 0.14.1
samza-kv_2.11 org.apache.samza 0.14.1
shadow com.github.jengelman.gradle.plugins 2.0.4
slf4j-simple org.slf4j 1.7.25
snappy-java org.xerial.snappy 1.1.4
spark-core_2.11 org.apache.spark 2.3.1
spark-network-common_2.11 org.apache.spark 2.3.1
spark-streaming_2.11 org.apache.spark 2.3.1
sqlline sqlline 1.4.0
stax2-api org.codehaus.woodstox 3.1.4
t-digest com.tdunning 3.2
tika-core org.apache.tika 1.18
tika-parsers org.apache.tika 1.18
util com.google.cloud.bigdataoss 1.4.5
visteg cz.malohlava 1.0.3
woodstox-core-asl org.codehaus.woodstox 4.4.1
amazon-kinesis-client com.amazonaws 1.8.8
amazon-kinesis-producer com.amazonaws 0.12.8
assertj-core org.assertj 2.5.0
auto-service com.google.auto.service 1.0-rc2
auto-value com.google.auto.value 1.5.3
avatica-core org.apache.calcite.avatica 1.11.0
aws-java-sdk-cloudwatch com.amazonaws 1.11.255
aws-java-sdk-core com.amazonaws 1.11.319
aws-java-sdk-kinesis com.amazonaws 1.11.255
aws-java-sdk-s3 com.amazonaws 1.11.319
bndlib biz.aQute 1.43.0
build-scan-plugin com.gradle 1.13.1
calcite-core org.apache.calcite 1.16.0
calcite-linq4j org.apache.calcite 1.16.0
cassandra-all org.apache.cassandra 3.9
cassandra-driver-core com.datastax.cassandra 3.5.0
cassandra-driver-mapping com.datastax.cassandra 3.5.0
commons-cli commons-cli 1.2
commons-codec commons-codec 1.9
commons-dbcp2 org.apache.commons 2.1.1
config com.typesafe 1.3.0
de.flapdoodle.embed.mongo de.flapdoodle.embed 1.50.1
de.flapdoodle.embed.process de.flapdoodle.embed 1.50.1
derby org.apache.derby 10.12.1.1
derbyclient org.apache.derby 10.12.1.1
derbynet org.apache.derby 10.12.1.1
elasticsearch org.elasticsearch 5.6.3
elasticsearch-hadoop org.elasticsearch 5.0.0
elasticsearch-rest-client org.elasticsearch.client 5.6.3
error_prone_annotations com.google.errorprone 2.1.2
fastjson com.alibaba 1.2.12
flink-clients_2.11 org.apache.flink 1.5.0
flink-core org.apache.flink 1.5.0
flink-java org.apache.flink 1.5.0
flink-metrics-core org.apache.flink 1.5.0
flink-runtime_2.11 org.apache.flink 1.5.0
flink-streaming-java_2.11 org.apache.flink 1.5.0
flink-test-utils_2.11 org.apache.flink 1.5.0
framework org.elasticsearch.test 5.6.3
freemarker org.freemarker 2.3.25-incubating
gradle-apt-plugin net.ltgt.gradle 0.13
gradle-avro-plugin com.commercehub.gradle.plugin 0.11.0
gradle-docker gradle.plugin.com.palantir.gradle.docker 0.13.0
gradle-errorprone-plugin net.ltgt.gradle 0.0.13
gradle-offline-dependencies-plugin gradle.plugin.io.pry.gradle.offline_dependencies 0.3
gradle-release net.researchgate 2.6.0
gradle-versions-plugin com.github.ben-manes 0.17.0
groovy-all org.codehaus.groovy 2.4.13
grpc-context io.grpc 1.12.0
grpc-protobuf io.grpc 1.12.0
grpc-testing io.grpc 1.12.0
gson com.google.code.gson 2.7
guava com.google.guava 20.0
hbase-common org.apache.hbase 1.2.6
hbase-hadoop-compat org.apache.hbase 1.2.6
hbase-hadoop2-compat org.apache.hbase 1.2.6
hbase-server org.apache.hbase 1.2.6
hbase-shaded-client org.apache.hbase 1.2.6
hbase-shaded-server org.apache.hbase 1.2.6
hive-cli org.apache.hive 2.1.0
hive-common org.apache.hive 2.1.0
hive-exec org.apache.hive 2.1.0
hive-hcatalog-core org.apache.hive.hcatalog 2.1.0
httpasyncclient org.apache.httpcomponents 4.1.2
httpclient org.apache.httpcomponents 4.5.2
httpcore org.apache.httpcomponents 4.4.5
httpcore-nio org.apache.httpcomponents 4.4.5
javacc net.java.dev.javacc 4.0
jline jline 2.14.6
jna net.java.dev.jna 4.1.0
kryo com.esotericsoftware.kryo 2.21
metrics-core io.dropwizard.metrics 3.1.2
mongo-java-driver org.mongodb 3.2.2
netty-all io.netty 4.1.17.Final
netty-tcnative-boringssl-static io.netty 2.0.7.Final
netty-transport-native-epoll io.netty 4.1.8.Final
opencensus-api io.opencensus 0.11.0
opencensus-contrib-grpc-metrics io.opencensus 0.11.0
parquet-avro org.apache.parquet 1.10.0
parquet-common org.apache.parquet 1.10.0
parquet-hadoop org.apache.parquet 1.10.0
propdeps-plugin io.spring.gradle 0.0.9.RELEASE
proto-google-common-protos com.google.api.grpc 1.0.0
protobuf-gradle-plugin com.google.protobuf 0.8.5
protobuf-java com.google.protobuf 3.5.1
protobuf-java-util com.google.protobuf 3.5.1
protoc com.google.protobuf 3.5.1
protoc-gen-grpc-java io.grpc 1.2.0
proton-j org.apache.qpid 0.13.1
randomizedtesting-runner com.carrotsearch.randomizedtesting 2.5.0
scala-library org.scala-lang 2.11.8
slf4j-api org.slf4j 1.7.25
slf4j-jdk14 org.slf4j 1.7.25
solr-core org.apache.solr 5.5.4
solr-solrj org.apache.solr 5.5.4
solr-test-framework org.apache.solr 5.5.4
spotless-plugin-gradle com.diffplug.spotless 3.6.0
spring-expression org.springframework 4.3.5.RELEASE
stream com.clearspring.analytics 2.9.5
transport org.elasticsearch.client 5.0.0
transport-netty4-client org.elasticsearch.plugin 5.6.3
xz org.tukaani 1.5
zookeeper org.apache.zookeeper 3.4.11
google-auth-library-credentials com.google.auth 0.9.1
grpc-auth io.grpc 1.12.0
grpc-core io.grpc 1.12.0
grpc-netty io.grpc 1.12.0
grpc-stub io.grpc 1.12.0

Workers have the following packages already installed.

GroupIdArtifactIdVersion
com.google.api-client google-api-client 1.23.0
com.google.oauth-client google-oauth-client 1.23.0
com.google.http-client google-http-client-jackson2 1.23.0
com.google.apis google-api-services-dataflow v1b3-rev221-1.23.0
org.apache.beam beam-model-pipeline 2.6.0
org.apache.beam beam-model-fn-execution 2.6.0
org.apache.beam beam-sdks-java-extensions-google-cloud-platform-core 2.6.0
com.google.auth google-auth-library-oauth2-http 0.7.1
com.google.cloud.bigdataoss gcsio 1.4.5
com.google.api-client google-api-client-java6 1.20.0
com.google.api-client google-api-client-jackson2 1.20.0
com.google.oauth-client google-oauth-client-java6 1.20.0
com.google.cloud.bigdataoss util 1.4.5
com.google.apis google-api-services-cloudresourcemanager v1-rev477-1.23.0
com.google.apis google-api-services-storage v1-rev124-1.23.0
com.google.auth google-auth-library-credentials 0.7.1
org.apache.beam beam-sdks-java-core 2.6.0
org.xerial.snappy snappy-java 1.1.4
org.tukaani xz 1.5
org.apache.beam beam-sdks-java-io-google-cloud-platform 2.6.0
org.apache.beam beam-sdks-java-extensions-protobuf 2.6.0
com.google.apis google-api-services-bigquery v2-rev374-1.23.0
com.google.api gax-grpc 0.20.0
com.google.api gax 1.3.1
org.threeten threetenbp 1.3.3
com.google.cloud google-cloud-core-grpc 1.2.0
com.google.apis google-api-services-pubsub v1-rev382-1.23.0
com.google.api.grpc grpc-google-cloud-pubsub-v1 0.1.18
com.google.cloud.datastore datastore-v1-proto-client 1.4.0
com.google.http-client google-http-client-protobuf 1.20.0
com.google.cloud.datastore datastore-v1-protos 1.3.0
com.google.api.grpc grpc-google-common-protos 0.1.0
com.google.cloud google-cloud-core 1.0.2
org.json json 20160810
com.google.cloud google-cloud-spanner 0.20.0b-beta
com.google.api.grpc proto-google-cloud-spanner-v1 0.1.11b
com.google.api.grpc proto-google-cloud-spanner-admin-instance-v1 0.1.11
com.google.api.grpc grpc-google-cloud-spanner-v1 0.1.11b
com.google.api.grpc grpc-google-cloud-spanner-admin-database-v1 0.1.11
com.google.api.grpc grpc-google-cloud-spanner-admin-instance-v1 0.1.11
com.google.api.grpc grpc-google-longrunning-v1 0.1.11
com.google.api.grpc proto-google-longrunning-v1 0.1.11
com.google.cloud.bigtable bigtable-protos 1.0.0-pre3
com.google.cloud.bigtable bigtable-client-core 1.0.0
commons-logging commons-logging 1.2
com.google.auth google-auth-library-appengine 0.7.0
io.opencensus opencensus-contrib-grpc-util 0.7.0
io.dropwizard.metrics metrics-core 3.1.2
com.google.api.grpc proto-google-cloud-spanner-admin-database-v1 0.1.9
com.google.api.grpc proto-google-common-protos 0.1.9
io.grpc grpc-all 1.2.0
io.grpc grpc-okhttp 1.2.0
com.squareup.okhttp okhttp 2.5.0
com.squareup.okio okio 1.6.0
io.grpc grpc-protobuf-nano 1.2.0
com.google.protobuf.nano protobuf-javanano 3.0.0-alpha-5
org.apache.beam beam-runners-core-construction-java 2.6.0
org.apache.beam beam-model-job-management 2.6.0
org.apache.beam beam-runners-core-java 2.6.0
org.apache.beam beam-runners-java-fn-execution 2.6.0
org.apache.beam beam-vendor-sdks-java-extensions-protobuf 2.6.0
org.apache.beam beam-sdks-java-fn-execution 2.6.0
org.apache.beam beam-runners-google-cloud-dataflow-java 2.6.0
com.google.apis google-api-services-clouddebugger v2-rev233-1.23.0
com.fasterxml.jackson.core jackson-annotations 2.9.5
com.fasterxml.jackson.core jackson-databind 2.9.5
com.fasterxml.jackson.core jackson-core 2.9.5
joda-time joda-time 2.4
org.apache.avro avro 1.8.2
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
com.thoughtworks.paranamer paranamer 2.7
org.apache.commons commons-compress 1.8.1
org.slf4j slf4j-api 1.7.25
com.google.code.findbugs jsr305 3.0.1
com.google.auto.service auto-service 1.0-rc2
com.google.auto auto-common 0.3
com.google.auto.value auto-value 1.5.3
com.google.http-client google-http-client 1.23.0
org.apache.httpcomponents httpclient 4.0.1
org.apache.httpcomponents httpcore 4.0.1
commons-codec commons-codec 1.3
com.google.http-client google-http-client-jackson 1.23.0
com.google.api.grpc proto-google-cloud-pubsub-v1 0.1.18
com.google.api api-common 1.1.0
com.google.api.grpc proto-google-iam-v1 0.1.18
com.google.guava guava 20.0
com.google.guava guava-testlib 20.0
com.google.errorprone error_prone_annotations 2.0.15
com.google.protobuf protobuf-java 3.2.0
io.grpc grpc-core 1.2.0
io.grpc grpc-context 1.2.0
com.google.instrumentation instrumentation-api 0.3.0
io.grpc grpc-stub 1.2.0
io.grpc grpc-netty 1.2.0
io.netty netty-codec-http2 4.1.8.Final
io.netty netty-codec-http 4.1.8.Final
io.netty netty-handler-proxy 4.1.8.Final
io.netty netty-codec-socks 4.1.8.Final
io.grpc grpc-protobuf 1.2.0
com.google.protobuf protobuf-java-util 3.2.0
com.google.code.gson gson 2.7
io.grpc grpc-protobuf-lite 1.2.0
io.grpc grpc-auth 1.2.0
io.netty netty-handler 4.1.8.Final
io.netty netty-buffer 4.1.8.Final
io.netty netty-transport 4.1.8.Final
io.netty netty-resolver 4.1.8.Final
io.netty netty-codec 4.1.8.Final
io.netty netty-common 4.1.8.Final
io.netty netty-transport-native-epoll linux-x86_64
io.netty netty-tcnative-boringssl-static 1.1.33.Fork18
org.conscrypt conscrypt-openjdk linux-x86_64
org.eclipse.jetty jetty-server 9.2.10.v20150310
org.eclipse.jetty jetty-http 9.2.10.v20150310
org.eclipse.jetty jetty-util 9.2.10.v20150310
org.eclipse.jetty jetty-io 9.2.10.v20150310
org.eclipse.jetty jetty-servlet 9.2.10.v20150310
org.eclipse.jetty jetty-security 9.2.10.v20150310
javax.servlet javax.servlet-api 3.1.0
org.slf4j slf4j-jdk14 1.7.25
org.apache.beam beam-sdks-java-core 2.6.0
org.apache.beam beam-sdks-java-extensions-google-cloud-platform-core 2.6.0
org.apache.beam beam-runners-core-java 2.6.0
org.apache.beam beam-runners-direct-java 2.6.0
args4j args4j 2.33
org.hamcrest hamcrest-all 1.3
junit junit 4.12
org.hamcrest hamcrest-core 1.3
org.mockito mockito-all 1.10.19
io.opencensus opencensus-api 0.12.2

SDK for Python

Version 2.6.0

Apache Beam SDK for Python 2.5.0 has the following compile and runtime dependencies.

PackageVersion
avro&>=1.8.1, <2.0.0
crcmod>=1.7, <2.0
dill>=0.2.6,<=0.2.8.2
fastavro0.19.7
future>=0.16.0, <1.0.0
futures>=3.1.1, <4.0.0
google-apitools>=0.5.18, <=0.5.20
google-cloud-pubsub0.26.0
google-cloud-bigquery0.25.0
googledatastore7.0.1
grpcio>=1.8, <2.0
hdfs>=2.1.0, <3.0.0
httplib2>=0.8, <0.11.3
mock>=1.0.1, <3.0.0
nose>=1.3.7
oauth2client>=2.0.1, <5.0.0
proto-google-cloud-datastore-v1>=0.90.0, <=0.90.4
proto-google-cloud-pubsub-v10.15.4
protobuf>=3.5.0.post1, <4
pyhamcrest>=1.9, <2.0
pytz>=2018.3, <=2018.4
pyvcf>=0.6.8, <0.7.0
pyyaml>=3.12, <4.0.0
six>=1.9, <1.12
typing>=3.6.0, <3.7.0

Workers have the following packages already installed.

PackageVersion
avro1.8.2
beautifulsoup44.5.1
bs40.0.1
crcmod1.7
cython0.28.1
dill0.2.6
future0.16.0
futures3.1.1
grpcio1.10.0
google-apitools0.5.20
google-cloud-bigquery0.25.0
google-cloud-pubsub0.26.0
google-python-cloud-debugger1.9.1
googledatastore7.0.1
guppy0.1.10
hdfs2.1.0
httplib20.9.2
mock2.0.0
nltk3.2.1
nose1.3.7
numpy1.13.3
oauth2client4.1.2
pandas0.18.1
pillow3.4.1
proto-google-cloud-datastore-v10.90.4
proto-google-cloud-pubsub-v10.15.4
protobuf3.5.1
protorpc0.11.1
python-gflags3.0.6
python-snappy0.5.1
pytz2018.4
pyvcf0.6.8
pyyaml3.12
requests2.18.4
scikit-learn0.19.1
scipy1.0.0
six1.10.0
tensorflow1.8.0
typing3.6.1
wheel0.29.0

Cloud Dataflow SDKs

SDK for Java

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

SDK for Python

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow Documentation