SDK and Worker Dependencies

The Apache Beam SDKs and Dataflow workers depend on common third-party components which then import additional dependencies. Version collisions can result in unexpected behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution.

This page contains dependency and worker package information for Apache Beam and Dataflow SDK releases:

Apache Beam 2.x SDKs

SDK for Java

SDK for Python

Version 2.40.0

Dataflow workers have the following packages already installed.

Python 3.7.13

PackageVersion
absl-py1.1.0
astunparse1.6.3
atomicwrites1.4.0
attrs21.4.0
avro-python31.8.2
beautifulsoup44.11.1
bs40.0.1
cachetools4.2.4
certifi2022.6.15
charset-normalizer2.0.12
click8.1.3
cloudpickle2.1.0
crcmod1.7
Cython0.29.30
dill0.3.1.1
docopt0.6.2
fastavro1.5.1
fasteners0.17.3
flatbuffers2.0
future0.18.2
gast0.5.3
google-api-core1.31.6
google-api-python-client2.51.0
google-apitools0.5.31
google-auth1.35.0
google-auth-httplib20.1.0
google-auth-oauthlib0.4.6
google-cloud-bigquery2.34.4
google-cloud-bigquery-storage2.13.2
google-cloud-bigtable1.7.2
google-cloud-core1.7.2
google-cloud-datastore1.15.5
google-cloud-dlp3.7.1
google-cloud-language1.3.2
google-cloud-profiler3.1.0
google-cloud-pubsub2.13.0
google-cloud-pubsublite1.4.2
google-cloud-recommendations-ai0.2.0
google-cloud-spanner1.19.3
google-cloud-videointelligence1.16.3
google-cloud-vision1.0.2
google-crc32c1.3.0
google-pasta0.2.0
google-python-cloud-debugger2.18
google-resumable-media2.3.3
googleapis-common-protos1.56.3
grpc-google-iam-v10.12.4
grpcio1.47.0
grpcio-gcp0.2.2
grpcio-status1.47.0
guppy33.1.2
h5py3.7.0
hdfs2.7.0
httplib20.20.4
idna3.3
importlib-metadata4.11.4
joblib1.1.0
keras2.8.0
Keras-Preprocessing1.1.2
libclang14.0.1
Markdown3.3.7
mmh33.0.0
more-itertools8.13.0
nltk3.7
nose1.3.7
numpy1.21.6
oauth2client4.1.3
oauthlib3.2.0
opt-einsum3.3.0
orjson3.7.2
overrides6.1.0
packaging21.3
pandas1.3.5
Pillow7.2.0
pip22.1.2
pluggy0.13.1
proto-plus1.20.6
protobuf3.19.4
py1.11.0
pyarrow7.0.0
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.2
pymongo3.12.3
pyparsing3.0.9
pytest4.6.11
python-dateutil2.8.2
python-snappy0.6.1
pytz2022.1
PyYAML6.0
regex2022.6.2
requests2.28.0
requests-oauthlib1.3.1
rsa4.8
scikit-learn1.0.2
scipy1.7.3
setuptools57.5.0
six1.16.0
soupsieve2.3.2.post1
tensorboard2.8.0
tensorboard-data-server0.6.1
tensorboard-plugin-wit1.8.1
tensorflow2.8.0
tensorflow-io-gcs-filesystem0.26.0
termcolor1.1.0
tf-estimator-nightly2.8.0.dev2021122109
threadpoolctl3.1.0
tqdm4.64.0
typing-utils0.1.0
typing_extensions4.2.0
uritemplate4.1.1
urllib31.26.9
wcwidth0.2.5
Werkzeug2.1.2
wheel0.37.1
wrapt1.14.1
zipp3.8.0

Python 3.8.13

PackageVersion
absl-py1.1.0
astunparse1.6.3
atomicwrites1.4.0
attrs21.4.0
avro-python31.8.2
beautifulsoup44.11.1
bs40.0.1
cachetools4.2.4
certifi2022.6.15
charset-normalizer2.0.12
click8.1.3
cloudpickle2.1.0
crcmod1.7
Cython0.29.30
dill0.3.1.1
docopt0.6.2
fastavro1.5.1
fasteners0.17.3
flatbuffers2.0
future0.18.2
gast0.5.3
google-api-core1.31.6
google-api-python-client2.51.0
google-apitools0.5.31
google-auth1.35.0
google-auth-httplib20.1.0
google-auth-oauthlib0.4.6
google-cloud-bigquery2.34.4
google-cloud-bigquery-storage2.13.2
google-cloud-bigtable1.7.2
google-cloud-core1.7.2
google-cloud-datastore1.15.5
google-cloud-dlp3.7.1
google-cloud-language1.3.2
google-cloud-profiler3.1.0
google-cloud-pubsub2.13.0
google-cloud-pubsublite1.4.2
google-cloud-recommendations-ai0.2.0
google-cloud-spanner1.19.3
google-cloud-videointelligence1.16.3
google-cloud-vision1.0.2
google-crc32c1.3.0
google-pasta0.2.0
google-python-cloud-debugger2.18
google-resumable-media2.3.3
googleapis-common-protos1.56.3
grpc-google-iam-v10.12.4
grpcio1.47.0
grpcio-gcp0.2.2
grpcio-status1.47.0
guppy33.1.2
h5py3.7.0
hdfs2.7.0
httplib20.20.4
idna3.3
importlib-metadata4.11.4
joblib1.1.0
keras2.8.0
Keras-Preprocessing1.1.2
libclang14.0.1
Markdown3.3.7
mmh33.0.0
more-itertools8.13.0
nltk3.7
nose1.3.7
numpy1.22.3
oauth2client4.1.3
oauthlib3.2.0
opt-einsum3.3.0
orjson3.7.2
overrides6.1.0
packaging21.3
pandas1.4.0
Pillow7.2.0
pip22.1.2
pluggy0.13.1
proto-plus1.20.6
protobuf3.19.4
py1.11.0
pyarrow7.0.0
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.2
pymongo3.12.3
pyparsing3.0.9
pytest4.6.11
python-dateutil2.8.2
python-snappy0.6.1
pytz2022.1
PyYAML6.0
regex2022.6.2
requests2.28.0
requests-oauthlib1.3.1
rsa4.8
scikit-learn1.0.2
scipy1.7.3
setuptools57.5.0
six1.16.0
soupsieve2.3.2.post1
tensorboard2.8.0
tensorboard-data-server0.6.1
tensorboard-plugin-wit1.8.1
tensorflow2.8.0
tensorflow-io-gcs-filesystem0.26.0
termcolor1.1.0
tf-estimator-nightly2.8.0.dev2021122109
threadpoolctl3.1.0
tqdm4.64.0
typing-utils0.1.0
typing_extensions4.2.0
uritemplate4.1.1
urllib31.26.9
wcwidth0.2.5
Werkzeug2.1.2
wheel0.37.1
wrapt1.14.1
zipp3.8.0

Python 3.9.12

PackageVersion
absl-py1.1.0
astunparse1.6.3
atomicwrites1.4.0
attrs21.4.0
avro-python31.8.2
beautifulsoup44.11.1
bs40.0.1
cachetools4.2.4
certifi2022.6.15
charset-normalizer2.0.12
click8.1.3
cloudpickle2.1.0
crcmod1.7
Cython0.29.30
dill0.3.1.1
docopt0.6.2
fastavro1.5.1
fasteners0.17.3
flatbuffers2.0
future0.18.2
gast0.5.3
google-api-core1.31.6
google-api-python-client2.51.0
google-apitools0.5.31
google-auth1.35.0
google-auth-httplib20.1.0
google-auth-oauthlib0.4.6
google-cloud-bigquery2.34.4
google-cloud-bigquery-storage2.13.2
google-cloud-bigtable1.7.2
google-cloud-core1.7.2
google-cloud-datastore1.15.5
google-cloud-dlp3.7.1
google-cloud-language1.3.2
google-cloud-profiler3.1.0
google-cloud-pubsub2.13.0
google-cloud-pubsublite1.4.2
google-cloud-recommendations-ai0.2.0
google-cloud-spanner1.19.3
google-cloud-videointelligence1.16.3
google-cloud-vision1.0.2
google-crc32c1.3.0
google-pasta0.2.0
google-python-cloud-debugger2.18
google-resumable-media2.3.3
googleapis-common-protos1.56.3
grpc-google-iam-v10.12.4
grpcio1.47.0
grpcio-gcp0.2.2
grpcio-status1.47.0
guppy33.1.2
h5py3.7.0
hdfs2.7.0
httplib20.20.4
idna3.3
importlib-metadata4.11.4
joblib1.1.0
keras2.8.0
Keras-Preprocessing1.1.2
libclang14.0.1
Markdown3.3.7
mmh33.0.0
more-itertools8.13.0
nltk3.7
nose1.3.7
numpy1.22.3
oauth2client4.1.3
oauthlib3.2.0
opt-einsum3.3.0
orjson3.7.2
overrides6.1.0
packaging21.3
pandas1.4.0
Pillow7.2.0
pip22.1.2
pluggy0.13.1
proto-plus1.20.6
protobuf3.19.4
py1.11.0
pyarrow7.0.0
pyasn10.4.8
pyasn1-modules0.2.8
pydot1.4.2
pymongo3.12.3
pyparsing3.0.9
pytest4.6.11
python-dateutil2.8.2
python-snappy0.6.1
pytz2022.1
PyYAML6.0
regex2022.6.2
requests2.28.0
requests-oauthlib1.3.1
rsa4.8
scikit-learn1.0.2
scipy1.7.3
setuptools58.1.0
six1.16.0
soupsieve2.3.2.post1
tensorboard2.8.0
tensorboard-data-server0.6.1
tensorboard-plugin-wit1.8.1
tensorflow2.8.0
tensorflow-io-gcs-filesystem0.26.0
termcolor1.1.0
tf-estimator-nightly2.8.0.dev2021122109
threadpoolctl3.1.0
tqdm4.64.0
typing-utils0.1.0
typing_extensions4.2.0
uritemplate4.1.1
urllib31.26.9
wcwidth0.2.5
Werkzeug2.1.2
wheel0.37.1
wrapt1.14.1
zipp3.8.0