Dependências de SDK e worker

Os SDKs do Apache Beam e os workers do Dataflow dependem de componentes comuns de terceiros que importam outras dependências. Os conflitos de versão podem resultar em um comportamento inesperado no serviço. Se você estiver usando algum desses pacotes no código, saiba que algumas bibliotecas não são compatíveis com versões futuras e talvez seja necessário manter as versões listadas que estarão no escopo durante a execução.

Esta página contém informações sobre pacotes de dependência e worker para as versões do SDK do Apache Beam e do Dataflow:

SDKs 2.x do Apache Beam

SDK para Java

Observação: as informações sobre dependência de SDKs do Apache Beam para Java estão na respectiva página (em inglês).

SDK para Python

Versão 2.27.0

Os workers do Dataflow contam com os pacotes a seguir já instalados.

Python 3.6.12

PacoteVersão
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dataclasses0,7
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3,5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
Regex2020.11.11
de solicitações2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
roda0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.7.9

PacoteVersão
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
importlib-metadata2.0.0
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3,5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
Regex2020.11.11
de solicitações2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
roda0.35.1
wrapt1.12.1
zipp3.4.0

Python 3.8.6

PacoteVersão
absl-py0.11.0
astunparse1.6.3
atomicwrites1.4.0
attrs20.3.0
avro-python31.8.2
beautifulsoup44.9.1
bs40.0.1
cachetools3.1.1
certifi2020.11.8
cffi1.14.3
chardet3.0.4
click7.1.2
crcmod1.7
Cython0.29.21
dill0.3.1.1
docopt0.6.2
fastavro0.24.2
fasteners0.15
freezegun0.3.15
future0.18.2
gast0.3.3
google-api-core1.23.0
google-api-python-client1.12.5
google-apitools0.5.31
google-auth1.23.0
google-auth-httplib20.0.4
google-auth-oauthlib0.4.2
google-cloud-bigquery1.26.1
google-cloud-bigtable1.0.0
google-cloud-build2.0.0
google-cloud-core1.1.0
google-cloud-datastore1.7.4
google-cloud-dlp0.13.0
google-cloud-language1.3.0
google-cloud-pubsub1.0.2
google-cloud-spanner1.13.0
google-cloud-videointelligence1.13.0
google-cloud-vision0.42.0
google-crc32c1.0.0
google-pasta0.2.0
google-python-cloud-debugger2.15
google-resumable-media1.1.0
googleapis-common-protos1.52.0
grpc-google-iam-v10.12.3
grpcio1.30.0
grpcio-gcp0.2.2
guppy33.0.10
h5py2.10.0
hdfs2.5.8
httplib20.17.3
idna2.10
joblib0.17.0
Keras-Preprocessing1.1.2
libcst0.3.13
Markdown3.3.3
mmh32.5.1
mock2.0.0
monotonic1.5
more-itertools8.6.0
mypy-extensions0.4.3
nltk3,5
nose1.3.7
nose-xunitmp0.4.1
numpy1.18.4
oauth2client3.0.0
oauthlib3.1.0
opt-einsum3.3.0
packaging20.4
pandas1.1.4
pbr5.5.1
Pillow7.2.0
pip20.2.4
pluggy0.13.1
proto-plus1.11.0
protobuf3.12.2
protorpc0.12.0
py1.9.0
pyarrow0.17.1
pyasn10.4.8
pyasn1-modules0.2.8
pycparser2.20
pydot1.4.1
pymongo3.10.1
pyparsing2.4.7
pytest4.6.11
python-dateutil2.8.1
python-gflags3.1.2
python-snappy0.5.4
pytz2020.1
PyYAML5.3.1
Regex2020.11.11
de solicitações2.24.0
requests-oauthlib1.3.0
rsa4.6
scikit-learn0.22.1
scipy1.4.1
setuptools50.3.2
six1.15.0
soupsieve2.0.1
tensorboard2.4.0
tensorboard-plugin-wit1.7.0
tensorflow2.3.0
tensorflow-estimator2.3.0
termcolor1.1.0
tqdm4.51.0
typing-extensions3.7.4.2
typing-inspect0.6.0
uritemplate3.0.1
urllib31.25.11
wcwidth0.2.5
Werkzeug1.0.1
roda0.35.1
wrapt1.12.1

SDKs do Dataflow 2.x

SDK para Java

Para determinar se o JAR está usando uma versão conflitante, inspecione a árvore de dependências do projeto. Ela pode ser gerada com várias ferramentas, como o Maven (em inglês).

Evite especificar "latest" no seu pom.xml para as bibliotecas na tabela a seguir.

SDK para Python

SDKs do Dataflow 1.x

SDK para Java