SDK and Worker Dependencies

The Apache Beam SDKs and Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

This page contains dependency and worker package information for Apache Beam and Dataflow SDK releases:

Apache Beam 2.x SDKs

SDK for Java

Note: Dependency information for Apache Beam SDKs for Java is listed on the Apache Beam SDK for Java dependencies page.

SDK for Python

Note: Dependency information for Apache Beam SDKs for Python is listed on the Apache Beam SDK for Python dependencies page.

Version 2.16.0

Dataflow workers have the following packages already installed.

Python 2.7.13

PackageVersion
absl-py0.8.1
astor0.8.0
atomicwrites1.3.0
attrs19.2.0
avro1.8.2
backports.functools-lru-cache1.5
backports.weakref1.0.post1
beautifulsoup44.7.1
bs40.0.1
cachetools3.1.1
certifi2019.9.11
chardet3.0.4
configparser4.0.2
contextlib20.6.0
crcmod1.7
Cython0.29.10
dill0.3.0
docopt0.6.2
enum341.1.6
fastavro0.21.24
fasteners0.15
funcsigs1.0.2
functools323.2.3.post2
future0.17.1
futures3.2.0
gast0.2.2
google-api-core1.14.3
google-api-python-client1.7.11
google-apitools0.5.28
google-auth1.6.3
google-auth-httplib20.0.3
google-cloud-bigquery1.17.0
google-cloud-bigtable1.0.0
google-cloud-core1.0.3
google-cloud-datastore1.7.4
google-cloud-pubsub1.0.0
google-pasta0.1.7
google-python-cloud-debugger1.9
google-resumable-media0.4.1
googleapis-common-protos1.6.0
googledatastore7.0.2
grpc-google-iam-v10.12.3
grpcio1.23.0
guppy0.1.10
h5py2.10.0
hdfs2.5.8
httplib20.12.0
idna2.8
importlib-metadata0.23
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.1.1
mock2.0.0
monotonic1.5
more-itertools5.0.0
nltk3.4.3
nose1.3.7
nose-xunitmp0.4.1
numpy1.16.5
oauth2client3.0.0
opt-einsum2.3.2
pandas0.24.2
pathlib22.3.5
pbr5.4.3
Pillow6.0.0
pip19.2.3
pluggy0.13.0
proto-google-cloud-datastore-v10.90.4
protobuf3.9.1
protorpc0.12.0
py1.8.0
pyarrow0.14.1
pyasn10.4.7
pyasn1-modules0.2.6
pydot1.4.1
pymongo3.8.0
pyparsing2.4.2
pytest4.4.0
python-dateutil2.8.0
python-gflags3.0.6
python-snappy0.5.4
pytz2019.2
PyVCF0.6.8
PyYAML3.13
requests2.22.0
rsa4.0
scandir1.10.0
scikit-learn0.20.3
scipy1.2.2
setuptools41.4.0
singledispatch3.4.0.3
six1.12.0
soupsieve1.9.4
tensorboard1.15.0
tensorflow-cpu1.15.0rc3
tensorflow-estimator1.15.1
termcolor1.1.0
typing3.6.6
uritemplate3.0.0
urllib31.25.6
Werkzeug0.16.0
wheel0.33.6
wrapt1.11.2
zipp0.6.0

Python 3.5.7

PackageVersion
absl-py0.8.1
astor0.8.0
atomicwrites1.3.0
attrs19.2.0
avro-python31.8.2
beautifulsoup44.7.1
bs40.0.1
cachetools3.1.1
certifi2019.9.11
chardet3.0.4
crcmod1.7
Cython0.29.10
dill0.3.0
docopt0.6.2
fastavro0.21.24
fasteners0.15
future0.17.1
gast0.2.2
google-api-core1.14.3
google-apitools0.5.28
google-auth1.6.3
google-cloud-bigquery1.17.0
google-cloud-bigtable1.0.0
google-cloud-core1.0.3
google-cloud-datastore1.7.4
google-cloud-pubsub1.0.0
google-pasta0.1.7
google-resumable-media0.4.1
googleapis-common-protos1.6.0
grpc-google-iam-v10.12.3
grpcio1.23.0
h5py2.10.0
hdfs2.5.8
httplib20.12.0
idna2.8
importlib-metadata0.23
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.1.1
mock2.0.0
monotonic1.5
more-itertools7.2.0
nltk3.4.3
nose1.3.7
nose-xunitmp0.4.1
numpy1.17.2
oauth2client3.0.0
opt-einsum3.1.0
pandas0.24.2
pathlib22.3.5
pbr5.4.3
Pillow6.0.0
pip19.2.3
pluggy0.13.0
protobuf3.9.1
protorpc0.12.0
py1.8.0
pyarrow0.14.1
pyasn10.4.7
pyasn1-modules0.2.6
pydot1.4.1
pymongo3.8.0
pyparsing2.4.2
pytest4.4.0
python-dateutil2.8.0
python-gflags3.0.6
python-snappy0.5.4
pytz2019.2
PyYAML3.13
requests2.22.0
rsa4.0
scikit-learn0.20.3
scipy1.3.1
setuptools41.2.0
six1.12.0
soupsieve1.9.4
tensorboard1.15.0
tensorflow-cpu1.15.0rc3
tensorflow-estimator1.15.1
termcolor1.1.0
urllib31.25.6
Werkzeug0.16.0
wheel0.33.6
wrapt1.11.2
zipp0.6.0

Python 3.6.9

PackageVersion
absl-py0.8.1
astor0.8.0
atomicwrites1.3.0
attrs19.2.0
avro-python31.8.2
beautifulsoup44.7.1
bs40.0.1
cachetools3.1.1
certifi2019.9.11
chardet3.0.4
crcmod1.7
Cython0.29.10
dill0.3.0
docopt0.6.2
fastavro0.21.24
fasteners0.15
future0.17.1
gast0.2.2
google-api-core1.14.3
google-apitools0.5.28
google-auth1.6.3
google-cloud-bigquery1.17.0
google-cloud-bigtable1.0.0
google-cloud-core1.0.3
google-cloud-datastore1.7.4
google-cloud-pubsub1.0.0
google-pasta0.1.7
google-resumable-media0.4.1
googleapis-common-protos1.6.0
grpc-google-iam-v10.12.3
grpcio1.23.0
h5py2.10.0
hdfs2.5.8
httplib20.12.0
idna2.8
importlib-metadata0.23
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.1.1
mock2.0.0
monotonic1.5
more-itertools7.2.0
nltk3.4.3
nose1.3.7
nose-xunitmp0.4.1
numpy1.17.2
oauth2client3.0.0
opt-einsum3.1.0
pandas0.24.2
pbr5.4.3
Pillow6.0.0
pip19.2.3
pluggy0.13.0
protobuf3.9.1
protorpc0.12.0
py1.8.0
pyarrow0.14.1
pyasn10.4.7
pyasn1-modules0.2.6
pydot1.4.1
pymongo3.8.0
pyparsing2.4.2
pytest4.4.0
python-dateutil2.8.0
python-gflags3.0.6
python-snappy0.5.4
pytz2019.2
PyYAML3.13
requests2.22.0
rsa4.0
scikit-learn0.20.3
scipy1.3.1
setuptools41.2.0
six1.12.0
soupsieve1.9.4
tensorboard1.15.0
tensorflow-cpu1.15.0rc3
tensorflow-estimator1.15.1
termcolor1.1.0
urllib31.25.6
Werkzeug0.16.0
wheel0.33.6
wrapt1.11.2
zipp0.6.0

Python 3.7.4

PackageVersion
absl-py0.8.1
astor0.8.0
atomicwrites1.3.0
attrs19.2.0
avro-python31.8.2
beautifulsoup44.7.1
bs40.0.1
cachetools3.1.1
certifi2019.9.11
chardet3.0.4
crcmod1.7
Cython0.29.10
dill0.3.0
docopt0.6.2
fastavro0.21.24
fasteners0.15
future0.17.1
gast0.2.2
google-api-core1.14.3
google-apitools0.5.28
google-auth1.6.3
google-cloud-bigquery1.17.0
google-cloud-bigtable1.0.0
google-cloud-core1.0.3
google-cloud-datastore1.7.4
google-cloud-pubsub1.0.0
google-pasta0.1.7
google-resumable-media0.4.1
googleapis-common-protos1.6.0
grpc-google-iam-v10.12.3
grpcio1.23.0
h5py2.10.0
hdfs2.5.8
httplib20.12.0
idna2.8
importlib-metadata0.23
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.1.1
mock2.0.0
monotonic1.5
more-itertools7.2.0
nltk3.4.3
nose1.3.7
nose-xunitmp0.4.1
numpy1.17.2
oauth2client3.0.0
opt-einsum3.1.0
pandas0.24.2
pbr5.4.3
Pillow6.0.0
pip19.2.3
pluggy0.13.0
protobuf3.9.1
protorpc0.12.0
py1.8.0
pyarrow0.14.1
pyasn10.4.7
pyasn1-modules0.2.6
pydot1.4.1
pymongo3.8.0
pyparsing2.4.2
pytest4.4.0
python-dateutil2.8.0
python-gflags3.0.6
python-snappy0.5.4
pytz2019.2
PyYAML3.13
requests2.22.0
rsa4.0
scikit-learn0.20.3
scipy1.3.1
setuptools41.2.0
six1.12.0
soupsieve1.9.4
tensorboard1.15.0
tensorflow-cpu1.15.0rc3
tensorflow-estimator1.15.1
termcolor1.1.0
urllib31.25.6
Werkzeug0.16.0
wheel0.33.6
wrapt1.11.2
zipp0.6.0

Dataflow 2.x SDKs

SDK for Java

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

SDK for Python

Dataflow 1.x SDKs

SDK for Java

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow
Need help? Visit our support page.