Job driver logs
See Dataproc job logs in Logging for information on configuring and viewing Dataproc on GKE Spark job driver logs.
Cloud Logging
Enable Cloud Operations for GKE to view, search, filter, and archive Dataproc on GKE logs in Cloud Logging.
You can search for Spark driver and executor logs in Cloud Logging using the following labels:
- dataproc.googleapis.com/cluster-uuid = Dataproc cluster UUID
- dataproc.googleapis.com/job-uuid = Dataproc job UUID
- dataproc.googleapis.com/role
Sample queries:
labels."k8s-pod/dataproc_googleapis_com/cluster-uuid"="Dataproc cluster UUID"
labels."k8s-pod/dataproc_googleapis_com/job-uuid"="Dataproc job UUID"
labels."k8s-pod/dataproc_googleapis_com/role"="driver"
For more inforation
- See Cloud Logging for information on accessing Dataproc on GKE job and cluster logs in Logging.