집계 메서드를 선택하여 측정항목을 확인합니다. 즉, 각 Ray 프로세스의 CPU 사용률을 보여주는 집계되지 않은 측정항목을 보도록 선택할 수 있습니다.
GCM 대시보드
Vertex AI 기반 Ray용 Grafana 대시보드를 가져오려면 Cloud Monitoring 대시보드의 자체 Grafana 대시보드 가져오기 가이드라인을 따릅니다.
Grafana 대시보드 JSON 파일만 있으면 됩니다. OSS Ray는 기본 대시보드 Grafana JSON 파일을 제공하여 이 수동 설정을 지원합니다.
사용자 소유 Grafana에서 측정항목 모니터링
Grafana 서버가 이미 실행 중인 경우 모든 Vertex AI 기반 Ray 클러스터 Prometheus 측정항목을 기존 Grafana 서버로 내보내는 방법도 있습니다. 이렇게 하려면 GMP Grafana를 사용하여 쿼리 안내를 따르세요. 이렇게 하면 새 Grafana 데이터 소스를 기존 Grafana 서버에 추가하고 데이터 소스 동기화를 사용하여 새 Grafana Prometheus 데이터 소스를 Vertex AI 기반 Ray 측정항목에 동기화할 수 있습니다.
데이터 소스 동기화를 사용하여 새로 추가된 Grafana 데이터 소스를 구성하고 인증하는 것이 중요합니다. Grafana 데이터 소스 구성 및 인증에 안내된 단계를 따릅니다.
동기화가 완료되면 Vertex AI 기반 Ray 측정항목을 기반으로 필요한 대시보드를 만들고 추가할 수 있습니다.
기본적으로 Vertex AI 기반 Ray 측정항목 수집이 사용 설정됩니다.
Vertex AI SDK for Python을 사용하여 이를 사용 중지하는 방법은 다음과 같습니다.
[[["이해하기 쉬움","easyToUnderstand","thumb-up"],["문제가 해결됨","solvedMyProblem","thumb-up"],["기타","otherUp","thumb-up"]],[["이해하기 어려움","hardToUnderstand","thumb-down"],["잘못된 정보 또는 샘플 코드","incorrectInformationOrSampleCode","thumb-down"],["필요한 정보/샘플이 없음","missingTheInformationSamplesINeed","thumb-down"],["번역 문제","translationIssue","thumb-down"],["기타","otherDown","thumb-down"]],["최종 업데이트: 2025-09-04(UTC)"],[],[],null,["# Monitor your Ray cluster on Vertex AI\n\nThis page covers how to view the tracking logs associated with your\nRay clusters and monitor the Ray on Vertex AI metrics. Guidance\nfor debugging Ray clusters is also provided.\n\nView logs\n---------\n\nWhen you perform tasks with your Ray cluster on Vertex AI,\ntracking logs are automatically generated and stored in both Cloud Logging\nand [open source Ray dashboard](https://docs.ray.io/en/latest/ray-observability/getting-started.html#logs-view). This section describes how to access\nthe generated logs through the Google Cloud console.\nBefore you begin, make sure to read the [Ray on Vertex AI overview](/vertex-ai/docs/open-source/ray-on-vertex-ai/overview) and [set up](/vertex-ai/docs/open-source/ray-on-vertex-ai/set-up) all the prerequisite tools you need. \n\n### Ray OSS dashboard\n\nYou can view the open source Ray log files through the Ray OSS dashboard:\n\n1. In the Google Cloud console, go to the Ray on Vertex AI page.\n\n [Go to the Ray on Vertex AI page](https://console.cloud.google.com/vertex-ai/ray)\n2. In the row for the cluster you created, select more_vert\n **more actions** menu.\n\n3. Select the Ray OSS dashboard link.\n The dashboard opens in another tab.\n\n4. Navigate to the **Logs** view in the top right corner in the menu:\n\n5. Click each node to see the log files associated with that node.\n\n### Cloud Logging console\n\n1. In the Google Cloud console, go to the **Logs Explorer** page:\n\n [Go to **Logs Explorer**](https://console.cloud.google.com/logs/query)\n\n \u003cbr /\u003e\n\n If you use the search bar to find this page, then select the result whose subheading is\n **Logging**.\n2. Select an existing Google Cloud project, folder, or organization.\n\n3. To display all Ray logs, enter the following query into the query-editor\n field, and then click **Run query**:\n\n ```\n resource.labels.task_name=\"ray-cluster-logs\"\n ```\n4. To narrow down the logs to a specific Ray cluster, add the following line\n to the query and then click **Run query**:\n\n ```\n labels.\"ml.googleapis.com/ray_cluster_id\"=CLUSTER_NAME\n ```\n\n Replace \u003cvar translate=\"no\"\u003eCLUSTER_NAME\u003c/var\u003e with the name for your Ray cluster. In the Google Cloud console go to **Vertex AI** \\\u003e **Ray on Vertex AI** where you see a list of cluster names in each region.\n5. To further narrow down the logs to a specific log file like `raylet.out`,\n click the name of the log under **Log fields** -\\\u003e **Log name**.\n\n6. You can group similar log entries together:\n\n 1. In the **Query results**, click a log entry to expand the log.\n\n 2. In the `jsonPayload`, click the `tailed_path` value. A drop-down menu\n appears.\n\n 3. Click **Show matching entries**.\n\nDisable logs\n------------\n\nBy default, Ray on Vertex AI Cloud Logging is enabled.\n\n- To disable the export of Ray logs to Cloud Logging, use the following\n Vertex AI SDK for Python command:\n\n vertex_ray.create_ray_cluster(..., enable_logging=False, ...)\n\nYou can view the Ray log files on the Ray dashboard even if the\nRay on Vertex AI Cloud Logging feature is disabled.\n\nMonitor metrics\n---------------\n\nYou can view the Ray on Vertex AI metrics in different ways using\n[Google Cloud Monitoring (GCM)](/monitoring).\nAlternatively, you can export the metrics from GCM to your own Grafana server.\n| **Note:** See [Google Cloud Managed Service for Prometheus (GMP)](/stackdriver/docs/managed-prometheus) for [pricing](/stackdriver/docs/managed-prometheus/cost-controls) and [data storage](/stackdriver/docs/managed-prometheus#gmp-data-storage) information.\n\n### Monitor Metrics in GCM\n\nThere are two ways you can view the Ray on Vertex AI metrics in GCM.\n\n- Use the direct view under **Metrics Explorer**.\n- Import the Grafana dashboard.\n\n### **Metrics Explorer**\n\n\nTo use the direct view under **Metrics Explorer**, follow these steps:\n\n1. Go to the Google Cloud Monitoring console.\n2. Under [**Explore**](http://console.cloud.google.com/monitoring/metrics-explorer) select **Metrics explorer**.\n3. Under **Active Resources** , select **Prometheus Target** . **Active Metric Categories** appears.\n4. Select **Ray**.\n\n A list of metrics appears:\n5. Select the metrics you want to monitor. For example:\n 1. Choose the cpu utilization percentage as a monitored metric: \n\n 2. Select a filter. For example, select cluster: \n Use the cluster ID to only monitor the above metrics for a specific cluster. To locate your cluster ID, follow these steps:\n 1. In the Google Cloud console, go to the **Ray** page.\n\n [Go to Ray](https://console.cloud.google.com/vertex-ai/ray)\n 2. Be sure you're in the project you want to create the experiment in. \n 3. Under **Name** a list of cluster IDs appears.\n\n 3. Select the **Aggregation** method to view the metrics. That is, you can choose to view unaggregated metrics, which show each Ray process's CPU utilization: \n\n\u003cbr /\u003e\n\n### **GCM** dashboard\n\n\nTo import a Grafana dashboard for Ray on Vertex AI follow the guidelines on the\ncloud monitoring dashboard,\n[Import your own grafana dashboard](https://cloud.google.com/monitoring/dashboards/import-grafana-dashboards).\n\n\nAll you need is a Grafana dashboard JSON file. OSS Ray supports this\n[manual setup](https://docs.ray.io/en/releases-2.5.1/cluster/metrics.html?highlight=simplist#recommended-use-ray-dashboard-with-embedded-grafana-visualizations)\nby providing the default dashboard Grafana JSON file.\n\n\u003cbr /\u003e\n\n### Monitor metrics\n\nfrom user-owned Grafana\n\nIf you already have a Grafana server running, then there's also a way to export\nall the Ray cluster on Vertex AI Prometheus metrics to your existing\nGrafana server. To do so, follow the GMP\n[Query using Grafana](/stackdriver/docs/managed-prometheus/query#begin)\nguidance. This lets you add a new Grafana data source to your existing Grafana\nserver and use the data source syncer to sync the new Grafana Prometheus data\nsource to Ray on Vertex AI metrics.\n\nIt's important that you configure and authenticate the newly added Grafana\ndata source using the data source syncer. Follow the steps provided in\n[Configure and authenticate the Grafana data source](/stackdriver/docs/managed-prometheus/query#grafana-oauth).\n\nOnce synced, you can create and add any dashboard you need based on the\nRay on Vertex AI metrics.\n\nBy default, the Ray on Vertex AI metrics collections are enabled.\nHere's how to disable them using Vertex AI SDK for Python: \n\n```python\nvertex_ray.create_ray_cluster(..., enable_metrics_collection=False, ...)\n```\n\nDebug Ray clusters\n------------------\n\nTo debug Ray clusters, use the **Head node interactive shell**:\n**Note:** Only use the interactive shell for debugging purposes or other advanced operations not supported in other ways. It's **not recommended** for normal operations like running workloads. \n\n### Google Cloud console\n\n\nTo access the **Head node interactive shell**, do the following:\n\n1. In the Google Cloud console, go to the **Ray on Vertex AI** page. \n [Go to Ray on Vertex AI](https://console.cloud.google.com/vertex-ai/ray)\n2. Be sure you're in the correct project. \n3. Select the cluster you want to examine. **Basic info** section appears.\n4. In the **Access links** section, click the link for **Head node interactive shell**. The head node interactive shell appears.\n5. Follow the instructions outlined in [Monitor and debug training with an interactive shell](/vertex-ai/docs/training/monitor-debug-interactive-shell).\n\nWhat's next\n-----------\n\n- [Delete a Ray cluster](/vertex-ai/docs/open-source/ray-on-vertex-ai/delete-cluster)"]]
클러스터 ID를 사용하여 특정 클러스터에 대한 위의 측정항목만 모니터링합니다. 클러스터 ID를 찾으려면 다음 단계를 따르세요.
