SDK 和工作器依赖项

Apache Beam SDK 和 Dataflow 工作器依赖于常见的第三方组件,而这些组件又会导入其他依赖项。版本冲突可能导致服务出现意外行为。如果您在代码中使用下列任一软件包,请注意有些库并不向前兼容,因此您可能需要固定使用(执行过程中将遇到的)所列版本。

本页面包含 Apache Beam 和 Dataflow SDK 版本的依赖项和工作器软件包信息:

Apache Beam 2.x SDK

Java 版 SDK

注意:有关 Java 版 Apache Beam SDK 的依赖项信息列在 Java 版 Apache Beam SDK 依赖项页面上。

Python 版 SDK

2.27.0 版

Dataflow 工作器已安装下列软件包。

Python 3.6.12

软件包版本
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dataclasses0.7
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
请求2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.7.9

软件包版本
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
请求2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.8.6

套餐版本
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3.5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
regex2020.11.11
请求2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
wheel0.35.1
wrapt1.12.1

Dataflow 2.x SDK

Java 版 SDK

要确定您的 JAR 是否使用了存在冲突的版本,请检查您项目的依赖关系树。您可以使用各种工具(如 Maven)来生成依赖关系树。

避免在 pom.xml 文件中针对下表中所列的库指定“latest”。

Python 版 SDK

Dataflow 1.x SDK

Java 版 SDK