Dataproc on GKE logging

Stay organized with collections Save and categorize content based on your preferences.

Job driver logs

See Dataproc job logs in Logging for information on configuring and viewing Dataproc on GKE Spark job driver logs.

Cloud Logging

Enable Cloud Operations for GKE to view, search, filter, and archive Dataproc on GKE logs in Cloud Logging.

You can search for Spark driver and executor logs in Cloud Logging using the following labels:

  • dataproc.googleapis.com/cluster-uuid = Dataproc cluster UUID
  • dataproc.googleapis.com/job-uuid = Dataproc job UUID
  • dataproc.googleapis.com/role

Sample queries:

labels."k8s-pod/dataproc_googleapis_com/cluster-uuid"="Dataproc cluster UUID"
labels."k8s-pod/dataproc_googleapis_com/job-uuid"="Dataproc job UUID"
labels."k8s-pod/dataproc_googleapis_com/role"="driver"

For more inforation

  • See Cloud Logging for information on accessing Dataproc on GKE job and cluster logs in Logging.