Dokumen ini memberikan informasi tentang metrik Spark. Secara default, Dataproc Serverless mengaktifkan pengumpulan metrik Spark yang tersedia, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark.
Untuk properti tambahan yang dapat Anda tetapkan saat mengirimkan workload batch Spark Dataproc Serverless, lihat Properti Spark
Properti pengumpulan metrik Spark
Anda dapat menggunakan properti yang tercantum di bagian ini untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark yang tersedia.
Properti | Deskripsi |
---|---|
spark.dataproc.driver.metrics |
Gunakan untuk menonaktifkan atau mengganti metrik driver Spark. |
spark.dataproc.executor.metrics |
Gunakan untuk menonaktifkan atau mengganti metrik eksekutor Spark. |
spark.dataproc.system.metrics |
Gunakan untuk menonaktifkan metrik sistem Spark. |
Contoh gcloud CLI:
Menonaktifkan pengumpulan metrik driver Spark:
gcloud dataproc batches submit spark \ --properties spark.dataproc.driver.metrics="" \ --region=region \ other args ...
Ganti pengumpulan metrik driver default Spark untuk hanya mengumpulkan metrik
BlockManager:disk.diskSpaceUsed_MB
danDAGScheduler:stage.failedStages
:gcloud dataproc batches submit spark \ --properties=^~^spark.dataproc.driver.metrics="BlockManager:disk.diskSpaceUsed_MB,DAGScheduler:stage.failedStages" \ --region=region \ other args ...
Metrik Spark yang tersedia
Dataproc Serverless mengumpulkan metrik Spark yang tercantum di bagian ini, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulannya.
custom.googleapis.com/METRIC_EXPLORER_NAME
.
Metrik pemicu Spark
Metrik | Nama Metrics Explorer |
---|---|
BlockManager:disk.diskSpaceUsed_MB | spark/driver/BlockManager/disk/diskSpaceUsed_MB |
BlockManager:memory.maxMem_MB | spark/driver/BlockManager/memory/maxMem_MB |
BlockManager:memory.memUsed_MB | spark/driver/BlockManager/memory/memUsed_MB |
DAGScheduler:job.activeJobs | spark/driver/DAGScheduler/job/activeJobs |
DAGScheduler:job.allJobs | spark/driver/DAGScheduler/job/allJobs |
DAGScheduler:messageProcessingTime | spark/driver/DAGScheduler/messageProcessingTime |
DAGScheduler:stage.failedStages | spark/driver/DAGScheduler/stage/failedStages |
DAGScheduler:stage.runningStages | spark/driver/DAGScheduler/stage/runningStages |
DAGScheduler:stage.waitingStages | spark/driver/DAGScheduler/stage/waitingStages |
Metrik eksekutor Spark
Metrik | Nama Metrics Explorer |
---|---|
ExecutorAllocationManager:executors.numberExecutorsDecommissionUnfinished | spark/driver/ExecutorAllocationManager/executors/numberExecutorsDecommissionUnfinished |
ExecutorAllocationManager:executors.numberExecutorsExitedUnexpectedly | spark/driver/ExecutorAllocationManager/executors/numberExecutorsExitedUnexpectedly |
ExecutorAllocationManager:executors.numberExecutorsGracefullyDecommissioned | spark/driver/ExecutorAllocationManager/executors/numberExecutorsGracefullyDecommissioned |
ExecutorAllocationManager:executors.numberExecutorsKilledByDriver | spark/driver/ExecutorAllocationManager/executors/numberExecutorsKilledByDriver |
LiveListenerBus:queue.executorManagement.listenerProcessingTime | spark/driver/LiveListenerBus/queue/executorManagement/listenerProcessingTime |
executor:bytesRead | spark/executor/bytesRead |
executor:bytesWritten | spark/executor/bytesWritten |
executor:cpuTime | spark/executor/cpuTime |
executor:diskBytesSpilled | spark/executor/diskBytesSpilled |
executor:jvmGCTime | spark/executor/jvmGCTime |
executor:memoryBytesSpilled | spark/executor/memoryBytesSpilled |
executor:recordsRead | spark/executor/recordsRead |
executor:recordsWritten | spark/executor/recordsWritten |
executor:runTime | spark/executor/runTime |
executor:shuffleFetchWaitTime | spark/executor/shuffleFetchWaitTime |
executor:shuffleRecordsRead | spark/executor/shuffleRecordsRead |
executor:shuffleRecordsWritten | spark/executor/shuffleRecordsWritten |
executor:shuffleRemoteBytesReadToDisk | spark/executor/shuffleRemoteBytesReadToDisk |
executor:shuffleWriteTime | spark/executor/shuffleWriteTime |
executor:succeededTasks | spark/executor/succeededTasks |
ExecutorMetrics:MajorGCTime | spark/executor/ExecutorMetrics/MajorGCTime |
ExecutorMetrics:MinorGCTime | spark/executor/ExecutorMetrics/MinorGCTime |
Metrik sistem
Metrik | Nama Metrics Explorer |
---|---|
agent:uptime | agent/uptime |
cpu:utilization | cpu/pemakaian |
disk:bytes_used | disk/bytes_used |
disk:percent_used | disk/percent_used |
memory:bytes_used | memory/bytes_used |
memory:percent_used | memory/percent_used |
network:tcp_connections | network/tcp_connections |
Melihat metrik Spark
Untuk melihat metrik Batch, klik ID batch di halaman Batches Dataproc di konsol Google Cloud untuk membuka halaman Details batch, yang menampilkan grafik metrik untuk workload batch di bagian Monitoring tab.
Lihat Cloud Monitoring Dataproc untuk mengetahui informasi tambahan tentang cara melihat metrik yang dikumpulkan.