SDK and Worker Dependencies

The Apache Beam SDKs and Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

This page contains dependency and worker package information for Apache Beam and Dataflow SDK releases:

Apache Beam 2.x SDKs

SDK for Java

Note: Dependency information for Apache Beam SDKs for Java is listed on the Apache Beam SDK for Java dependencies page.

SDK for Python

Note: Dependency information for Apache Beam SDKs for Python is listed on the Apache Beam SDK for Python dependencies page.

On October 7, 2020, Dataflow will stop supporting pipelines using Python 2. Read more information on the Python 2 support on Google Cloud page.

Version 2.23.0

Dataflow workers have the following packages already installed.

Python 2.7.18

PackageVersion
absl-py0.9.0
astor0.8.1
atomicwrites1.4.0
attrs19.3.0
avro1.8.2
backports.functools-lru-cache1.6.1
backports.weakref1.0.post1
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.6.20
chardet3.0.4
configparser4.0.2
contextlib20.6.0.post1
crcmod1.7
Cython0.29.20
dill0.3.1.1
docopt0.6.2
enum341.1.10
fastavro0.23.5
fasteners0.15
freezegun0.3.12
funcsigs1.0.2
functools323.2.3.post2
future0.18.2
futures3.3.0
gast0.2.2
google-api-core1.21.0
google-api-python-client1.9.3
google-apitools0.5.31
google-auth1.18.0
google-auth-httplib20.0.3
google-auth-oauthlib0.4.1
google-cloud-bigquery1.24.0
google-cloud-bigtable1.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-pasta0.2.0
google-python-cloud-debugger1.9
google-resumable-media0.5.1
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy0.1.11
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata1.7.0
Keras-Applications1.0.8
Keras-Preprocessing1.1.2
Markdown3.1.1
mock2.0.0
monotonic1.5
more-itertools5.0.0
nltk3.4.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.16.5
oauth2client3.0.0
oauthlib3.1.0
opt-einsum2.3.2
packaging20.4
pandas0.24.2
pathlib22.3.5
pbr5.4.5
Pillow6.2.2
pip20.1.1
pluggy0.13.1
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.16.0
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyVCF0.6.8
PyYAML5.3.1
requests2.24.0
requests-oauthlib1.3.0
rsa4.0
scandir1.10.0
scikit-learn0.20.4
scipy1.2.2
setuptools44.1.0
singledispatch3.4.0.3
six1.15.0
soupsieve1.9.6
tensorboard2.1.0
tensorflow2.1.0
tensorflow-estimator2.1.0
termcolor1.1.0
typing3.7.4.1
typing-extensions3.7.4.2
uritemplate3.0.1
urllib31.25.9
wcwidth0.2.5
Werkzeug1.0.1
wheel0.34.2
wrapt1.12.1
zipp1.2.0

Python 3.5.9

PackageVersion
absl-py0.9.0
astor0.8.1
atomicwrites1.4.0
attrs19.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.6.20
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.20
dill0.3.1.1
docopt0.6.2
fastavro0.23.5
fasteners0.15
freezegun0.3.12
future0.18.2
gast0.2.2
google-api-core1.21.0
google-apitools0.5.31
google-auth1.18.0
google-auth-oauthlib0.4.1
google-cloud-bigquery1.24.0
google-cloud-bigtable1.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-pasta0.2.0
google-resumable-media0.5.1
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.9
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata1.7.0
joblib0.14.1
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.2.2
mock2.0.0
monotonic1.5
more-itertools8.4.0
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.2.1
packaging20.4
pandas0.25.2
pathlib22.3.5
pbr5.4.5
Pillow7.1.2
pip20.1.1
pluggy0.13.1
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.6.8
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.21.3
scipy1.4.1
setuptools47.1.1
six1.15.0
soupsieve2.0.1
tensorboard2.1.1
tensorflow2.1.1
tensorflow-estimator2.1.0
termcolor1.1.0
tqdm4.47.0
typing-extensions3.7.4.2
urllib31.25.9
wcwidth0.2.5
Werkzeug1.0.1
wheel0.34.2
wrapt1.12.1
zipp1.2.0

Python 3.6.10

PackageVersion
absl-py0.9.0
astor0.8.1
atomicwrites1.4.0
attrs19.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.6.20
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.20
dill0.3.1.1
docopt0.6.2
fastavro0.23.5
fasteners0.15
freezegun0.3.12
future0.18.2
gast0.2.2
google-api-core1.21.0
google-api-python-client1.9.3
google-apitools0.5.31
google-auth1.18.0
google-auth-httplib20.0.3
google-auth-oauthlib0.4.1
google-cloud-bigquery1.24.0
google-cloud-bigtable1.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media0.5.1
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.9
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata1.7.0
joblib0.15.1
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.2.2
mock2.0.0
monotonic1.5
more-itertools8.4.0
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.2.1
packaging20.4
pandas0.25.2
pbr5.4.5
Pillow7.1.2
pip20.1.1
pluggy0.13.1
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.6.8
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.21.3
scipy1.4.1
setuptools47.1.1
six1.15.0
soupsieve2.0.1
tensorboard2.1.1
tensorflow2.1.1
tensorflow-estimator2.1.0
termcolor1.1.0
tqdm4.47.0
typing-extensions3.7.4.2
uritemplate3.0.1
urllib31.25.9
wcwidth0.2.5
Werkzeug1.0.1
wheel0.34.2
wrapt1.12.1
zipp3.1.0

Python 3.7.7

PackageVersion
absl-py0.9.0
astor0.8.1
atomicwrites1.4.0
attrs19.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.6.20
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.20
dill0.3.1.1
docopt0.6.2
fastavro0.23.5
fasteners0.15
freezegun0.3.12
future0.18.2
gast0.2.2
google-api-core1.21.0
google-api-python-client1.9.3
google-apitools0.5.31
google-auth1.18.0
google-auth-httplib20.0.3
google-auth-oauthlib0.4.1
google-cloud-bigquery1.24.0
google-cloud-bigtable1.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media0.5.1
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.9
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata1.7.0
joblib0.15.1
Keras-Applications1.0.8
Keras-Preprocessing1.1.0
Markdown3.2.2
mock2.0.0
monotonic1.5
more-itertools8.4.0
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.2.1
packaging20.4
pandas0.25.2
pbr5.4.5
Pillow7.1.2
pip20.1.1
pluggy0.13.1
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.6.8
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.21.3
scipy1.4.1
setuptools47.1.1
six1.15.0
soupsieve2.0.1
tensorboard2.1.1
tensorflow2.1.1
tensorflow-estimator2.1.0
termcolor1.1.0
tqdm4.47.0
typing-extensions3.7.4.2
uritemplate3.0.1
urllib31.25.9
wcwidth0.2.5
Werkzeug1.0.1
wheel0.34.2
wrapt1.12.1
zipp3.1.0

Python 3.8.3

PackageVersion
atomicwrites1.4.0
attrs19.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.6.20
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.20
dill0.3.1.1
docopt0.6.2
fastavro0.23.5
fasteners0.15
freezegun0.3.12
future0.18.2
google-api-core1.21.0
google-api-python-client1.9.3
google-apitools0.5.31
google-auth1.18.0
google-auth-httplib20.0.3
google-cloud-bigquery1.24.0
google-cloud-bigtable1.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-python-cloud-debugger2.15
google-resumable-media0.5.1
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.9
hdfs2.5.8
httplib20.17.3
idna2.10
joblib0.15.1
mock2.0.0
monotonic1.5
more-itertools8.4.0
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
packaging20.4
pandas0.25.2
pbr5.4.5
Pillow7.1.2
pip20.1.1
pluggy0.13.1
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.6.8
requests2.24.0
rsa4.6
scikit-learn0.21.3
scipy1.4.1
setuptools47.1.1
six1.15.0
soupsieve2.0.1
tqdm4.47.0
typing-extensions3.7.4.2
uritemplate3.0.1
urllib31.25.9
wcwidth0.2.5
wheel0.34.2

Dataflow 2.x SDKs

SDK for Java

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

SDK for Python

Dataflow 1.x SDKs

SDK for Java