Metrik Spark

Secara default, Dataproc Serverless mengaktifkan pengumpulan metrik Spark yang tersedia, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark.

Properti pengumpulan metrik Spark

Anda dapat menggunakan properti yang tercantum di bagian ini untuk menonaktifkan atau mengganti pengumpulan satu atau beberapa metrik Spark yang tersedia.

Properti	Deskripsi
`spark.dataproc.driver.metrics`	Gunakan untuk menonaktifkan atau mengganti Metrik pengemudi Spark.
`spark.dataproc.executor.metrics`	Gunakan untuk menonaktifkan atau mengganti metrik eksekutor Spark.
`spark.dataproc.system.metrics`	Gunakan untuk menonaktifkan Metrik sistem Spark.

Contoh gcloud CLI:

Nonaktifkan pengumpulan metrik driver Spark:

gcloud dataproc batches submit spark \
    --properties spark.dataproc.driver.metrics="" \
    --region=region \
    other args ...

Ganti pengumpulan metrik driver default Spark agar hanya mengumpulkan metrik BlockManager:disk.diskSpaceUsed_MB dan DAGScheduler:stage.failedStages:

gcloud dataproc batches submit spark \
    --properties=spark.dataproc.driver.metrics="BlockManager:disk.diskSpaceUsed_MB,DAGScheduler:stage.failedStages" \
    --region=region \
    other args ...

Metrik Spark yang tersedia

Dataproc Serverless mengumpulkan metrik Spark yang tercantum di bagian ini, kecuali jika Anda menggunakan properti pengumpulan metrik Spark untuk menonaktifkan atau mengganti pengumpulannya.

custom.googleapis.com/METRIC_EXPLORER_NAME.

Metrik driver Spark

Metrik	Nama Metrics Explorer
BlockManager:disk.diskSpaceUsed_MB	spark/driver/BlockManager/disk/diskSpaceUsed_MB
BlockManager:memory.maxMem_MB	spark/driver/BlockManager/memory/maxMem_MB
BlockManager:memory.memUsed_MB	spark/driver/BlockManager/memory/memUsed_MB
DAGScheduler:job.activeJobs	spark/driver/DAGScheduler/job/activeJobs
DAGScheduler:job.allJobs	spark/driver/DAGScheduler/job/allJobs
DAGScheduler:messageProcessingTime	spark/driver/DAGScheduler/messageProcessingTime
DAGScheduler:stage.failedStages	spark/driver/DAGScheduler/stage/failedStages
DAGScheduler:stage.runningStages	spark/driver/DAGScheduler/stage/runningStages
DAGScheduler:stage.waitingStages	spark/driver/DAGScheduler/stage/waitingStages

Metrik eksekutor Spark

Metrik	Nama Metrics Explorer
ExecutorAllocationManager:executors.numberExecutorsDecommissionUnfinished	spark/driver/ExecutorAllocationManager/executors/numberExecutorsDecommissionUnfinished
ExecutorAllocationManager:executors.numberExecutorsExitedUntunggal	spark/driver/ExecutorAllocationManager/executors/numberExecutorsExitedUnexpectedly
ExecutorAllocationManager:executors.numberExecutorsGracefilledDecommissioned	spark/driver/ExecutorAllocationManager/executors/numberExecutorsGracefullyDecommissioned
ExecutorAllocationManager:executors.numberExecutorsKilledByDriver	spark/driver/ExecutorAllocationManager/executors/numberExecutorsKilledByDriver
LiveListenerBus:queue.executorManagement.listenerProcessingTime	spark/driver/LiveListenerBus/queue/executorManagement/listenerProcessingTime
executor:bytesRead	spark/executor/bytesRead
executor:bytesWritten	spark/executor/bytesWritten
executor:cpuTime	spark/executor/cpuTime
executor:diskBytesSpilled	spark/executor/diskBytesSpilled
executor:jvmGCTime	spark/executor/jvmGCTime
executor:memoryBytesSpilled	spark/executor/memoryBytesSpilled
executor:recordsRead	spark/executor/recordsRead
executor:recordsWritten	spark/executor/recordsWritten
executor:runTime	spark/executor/runTime
executor:shuffleFetchWaitTime	spark/executor/shuffleFetchWaitTime
executor:shuffleRecordsRead	spark/executor/shuffleRecordsRead
executor:shuffleRecordsWritten	spark/executor/shuffleRecordsWritten
executor:shuffleRemoteBytesReadToDisk	spark/executor/shuffleRemoteBytesReadToDisk
executor:shuffleWriteTime	spark/executor/shuffleWriteTime
executor:succeededTasks	spark/executor/succeededTasks
ExecutorMetrics:MajorGCTime	spark/executor/ExecutorMetrics/MajorGCTime
ExecutorMetrics:MinorGCTime	spark/executor/ExecutorMetrics/MinorGCTime

Metrik sistem

Metrik	Nama Metric Explorer
agent:uptime	agent/uptime
{i>cpu:utilization<i}	cpu/pemakaian
disk:byte_digunakan	disk/byte_digunakan
disk:persen_digunakan	disk/persen_digunakan
memori:byte_digunakan	memori/byte_digunakan
memori:persen_digunakan	memori/persen_digunakan
network:tcp_connections	network/tcp_connections

Melihat metrik Spark

Untuk melihat Metrik batch, klik ID batch pada halaman Batches Dataproc di konsol Google Cloud untuk membuka halaman Detail batch, yang menampilkan grafik metrik untuk beban kerja batch di bagian tab Monitoring.