Secara default, Dataproc Serverless mengaktifkan pengumpulan metrik Spark yang tersedia, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark.
Properti pengumpulan metrik Spark
Anda dapat menggunakan properti yang tercantum di bagian ini untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark yang tersedia.
Properti | Deskripsi |
---|---|
spark.dataproc.driver.metrics |
Gunakan untuk menonaktifkan atau mengganti Metrik pengemudi Spark. |
spark.dataproc.executor.metrics |
Gunakan untuk menonaktifkan atau mengganti metrik eksekutor Spark. |
spark.dataproc.system.metrics |
Gunakan untuk menonaktifkan Metrik sistem Spark. |
Contoh gcloud CLI:
Nonaktifkan pengumpulan metrik driver Spark:
gcloud dataproc batches submit spark \ --properties spark.dataproc.driver.metrics="" \ --region=region \ other args ...
Ganti pengumpulan metrik driver default Spark agar hanya mengumpulkan metrik
BlockManager:disk.diskSpaceUsed_MB
danDAGScheduler:stage.failedStages
:gcloud dataproc batches submit spark \ --properties=spark.dataproc.driver.metrics="BlockManager:disk.diskSpaceUsed_MB,DAGScheduler:stage.failedStages" \ --region=region \ other args ...
Metrik Spark yang tersedia
Dataproc Serverless mengumpulkan metrik Spark yang tercantum di bagian ini, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulannya.
custom.googleapis.com/METRIC_EXPLORER_NAME
.
Metrik driver Spark
Metrik | Nama Metrics Explorer |
---|---|
BlockManager:disk.diskSpaceUsed_MB | spark/driver/BlockManager/disk/diskSpaceUsed_MB |
BlockManager:memory.maxMem_MB | spark/driver/BlockManager/memory/maxMem_MB |
BlockManager:memory.memUsed_MB | spark/driver/BlockManager/memory/memUsed_MB |
DAGScheduler:job.activeJobs | spark/driver/DAGScheduler/job/activeJobs |
DAGScheduler:job.allJobs | spark/driver/DAGScheduler/job/allJobs |
DAGScheduler:messageProcessingTime | spark/driver/DAGScheduler/messageProcessingTime |
DAGScheduler:stage.failedStages | spark/driver/DAGScheduler/stage/failedStages |
DAGScheduler:stage.runningStages | spark/driver/DAGScheduler/stage/runningStages |
DAGScheduler:stage.waitingStages | spark/driver/DAGScheduler/stage/waitingStages |
Metrik eksekutor Spark
Metrik | Nama Metrics Explorer |
---|---|
ExecutorAllocationManager:executors.numberExecutorsDecommissionUnfinished | spark/driver/ExecutorAllocationManager/executors/numberExecutorsDecommissionUnfinished |
ExecutorAllocationManager:executors.numberExecutorsExitedUntunggal | spark/driver/ExecutorAllocationManager/executors/numberExecutorsExitedUnexpectedly |
ExecutorAllocationManager:executors.numberExecutorsGracefilledDecommissioned | spark/driver/ExecutorAllocationManager/executors/numberExecutorsGracefullyDecommissioned |
ExecutorAllocationManager:executors.numberExecutorsKilledByDriver | spark/driver/ExecutorAllocationManager/executors/numberExecutorsKilledByDriver |
LiveListenerBus:queue.executorManagement.listenerProcessingTime | spark/driver/LiveListenerBus/queue/executorManagement/listenerProcessingTime |
executor:bytesRead | spark/executor/bytesRead |
executor:bytesWritten | spark/executor/bytesWritten |
executor:cpuTime | spark/executor/cpuTime |
executor:diskBytesSpilled | spark/executor/diskBytesSpilled |
executor:jvmGCTime | spark/executor/jvmGCTime |
executor:memoryBytesSpilled | spark/executor/memoryBytesSpilled |
executor:recordsRead | spark/executor/recordsRead |
executor:recordsWritten | spark/executor/recordsWritten |
executor:runTime | spark/executor/runTime |
executor:shuffleFetchWaitTime | spark/executor/shuffleFetchWaitTime |
executor:shuffleRecordsRead | spark/executor/shuffleRecordsRead |
executor:shuffleRecordsWritten | spark/executor/shuffleRecordsWritten |
executor:shuffleRemoteBytesReadToDisk | spark/executor/shuffleRemoteBytesReadToDisk |
executor:shuffleWriteTime | spark/executor/shuffleWriteTime |
executor:succeededTasks | spark/executor/succeededTasks |
ExecutorMetrics:MajorGCTime | spark/executor/ExecutorMetrics/MajorGCTime |
ExecutorMetrics:MinorGCTime | spark/executor/ExecutorMetrics/MinorGCTime |
Metrik sistem
Metrik | Nama Metric Explorer |
---|---|
agent:uptime | agent/uptime |
{i>cpu:utilization<i} | cpu/pemakaian |
disk:byte_digunakan | disk/byte_digunakan |
disk:persen_digunakan | disk/persen_digunakan |
memori:byte_digunakan | memori/byte_digunakan |
memori:persen_digunakan | memori/persen_digunakan |
network:tcp_connections | network/tcp_connections |
Melihat metrik Spark
Untuk melihat Metrik batch, klik ID batch pada halaman Batches Dataproc di konsol Google Cloud untuk membuka halaman Detail batch, yang menampilkan grafik metrik untuk beban kerja batch di bagian tab Monitoring.