SDK and Worker Dependencies

The Apache Beam SDKs and Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

This page contains dependency and worker package information for Apache Beam and Dataflow SDK releases:

Apache Beam 2.x SDKs

SDK for Java

Note: Dependency information for Apache Beam SDKs for Java is listed on the Apache Beam SDK for Java dependencies page.

SDK for Python

Version 2.28.0

Dataflow workers have the following packages already installed.

Python 3.6.12

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.12.5
cffi1.14.4
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dataclasses0.8
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.16
flatbuffers1.12
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.26.0
google-api-python-client1.12.8
google-apitools0.5.31
google-auth1.25.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.1.2
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.2.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.32.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata3.4.0
joblib1.0.0
Keras-Preprocessing1.1.2
libcst0.3.16
Markdown3.3.3
mmh32.5.1
mock2.0.0
more-itertools8.7.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.19.5
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.9
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip21.0.1
pluggy0.13.1
proto-plus1.13.0
protobuf3.12.2
protorpc0.12.0
py1.10.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.13
requests2.24.0
requests-oauthlib1.3.0
rsa4.7
scikit-learn0.22.1
scipy1.4.1
setuptools53.0.0
six1.15.0
soupsieve2.1
tensorboard2.4.1
tensorboard-plugin-wit1.8.0
tensorflow2.4.1
tensorflow-estimator2.4.0
termcolor1.1.0
tqdm4.56.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.36.2
wrapt1.12.1
zipp3.4.0

Python 3.7.9

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.12.5
cffi1.14.4
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.16
flatbuffers1.12
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.26.0
google-api-python-client1.12.8
google-apitools0.5.31
google-auth1.25.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.1.2
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.2.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.32.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata3.4.0
joblib1.0.0
Keras-Preprocessing1.1.2
libcst0.3.16
Markdown3.3.3
mmh32.5.1
mock2.0.0
more-itertools8.7.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.19.5
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.9
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip21.0.1
pluggy0.13.1
proto-plus1.13.0
protobuf3.12.2
protorpc0.12.0
py1.10.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.13
requests2.24.0
requests-oauthlib1.3.0
rsa4.7
scikit-learn0.22.1
scipy1.4.1
setuptools53.0.0
six1.15.0
soupsieve2.1
tensorboard2.4.1
tensorboard-plugin-wit1.8.0
tensorflow2.4.1
tensorflow-estimator2.4.0
termcolor1.1.0
tqdm4.56.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.36.2
wrapt1.12.1
zipp3.4.0

Python 3.8.7

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.12.5
cffi1.14.4
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.16
flatbuffers1.12
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.26.0
google-api-python-client1.12.8
google-apitools0.5.31
google-auth1.25.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.1.2
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.2.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.32.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
joblib1.0.0
Keras-Preprocessing1.1.2
libcst0.3.16
Markdown3.3.3
mmh32.5.1
mock2.0.0
more-itertools8.7.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.19.5
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.9
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip21.0.1
pluggy0.13.1
proto-plus1.13.0
protobuf3.12.2
protorpc0.12.0
py1.10.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.13
requests2.24.0
requests-oauthlib1.3.0
rsa4.7
scikit-learn0.22.1
scipy1.4.1
setuptools53.0.0
six1.15.0
soupsieve2.1
tensorboard2.4.1
tensorboard-plugin-wit1.8.0
tensorflow2.4.1
tensorflow-estimator2.4.0
termcolor1.1.0
tqdm4.56.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.36.2
wrapt1.12.1

Version 2.27.0

Dataflow workers have the following packages already installed.

Python 3.6.12

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dataclasses0.7
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.7.9

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.8.6

PackageVersion
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
requests2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1

Dataflow 2.x SDKs

SDK for Java

To determine if your JAR is using a conflicting version, inspect the dependency tree of your project. You can generate the dependency tree with various tools, such as Maven.

Avoid specifying "latest" in your pom.xml for the libraries in the following table.

SDK for Python

Dataflow 1.x SDKs

SDK for Java