Monitor Dataplex logs

Dataplex job logs can be viewed, searched, filtered, and archived in Cloud Logging.

See Google Cloud Observability pricing to understand your costs.
See Logs retention periods for information on logging retention.
See Exclusion filters to disable all logs or exclude logs from Logging.
See Routing and storage overview to route logs from Logging to Cloud Storage, BigQuery, or Pub/Sub.

Access Dataplex service logs in Logging

Dataplex publishes the following service logs to Cloud Logging.

Log Type	Log Name	logName query	Log Description
Data scan event logs	`dataplex.googleapis.com/data_scan`	`logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_scan)`	Event logs for data scan jobs indicating job state, results, and statistics
Data quality scan rule result logs	`dataplex.googleapis.com/data_quality_scan_rule_result`	`logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result)`	Results of data quality scan rules in a data quality job
Discovery logs	`dataplex.googleapis.com/discovery`	`logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery)`	Discovery progress and updates over Assets in a Zone
Process logs	`dataplex.googleapis.com/process`	`logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess)`	Job runs resulting from data processing Tasks

Query data scan event logs

When you use Dataplex to create and run a data scan, a data scan event log is produced in Logging for the resulting job.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

In the Google Cloud console, go to Cloud Logging > Logs (Logs Explorer):

Go to Logs Explorer
In the Logs Explorer view, find the Query tab.
Click the Resource drop-down.
Select Cloud Dataplex DataScan. Click Apply.
Click the Log name drop-down.
In the Search log names field, enter dataplex.googleapis.com%2Fdata_scan. Select data_scan and click Apply.
Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:
```
resource.labels.location="LOCATION"
resource.labels.datascan_id="DATA_SCAN_ID"
```
Click Run query.

gcloud

The Google Cloud SDK has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/DataScan" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_scan AND
    resource.labels.location=LOCATION AND
    resource.labels.datascan_id=DATA_SCAN_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

Query data quality scan rule result logs

When you use Dataplex to create and run a data quality scan, a data quality scan rule result log is produced in Logging for the resulting job.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

In the Google Cloud console, go to Cloud Logging > Logs (Logs Explorer):

Go to Logs Explorer
In the Logs Explorer view, find the Query tab.
Click the Resource drop-down.
Select Cloud Dataplex DataScan. Click Apply.
Click the Log name drop-down.
In the Search log names field, enter dataplex.googleapis.com%2Fdata_quality_scan_rule_result. Select data_quality_scan_rule_result and click Apply.
Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:
```
resource.labels.location="LOCATION"
resource.labels.datascan_id="DATA_SCAN_ID"
```
Click Run query.

gcloud

The Google Cloud SDK has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/DataScan" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result AND
    resource.labels.location=LOCATION AND
    resource.labels.datascan_id=DATA_SCAN_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

Query discovery logs

When you use Dataplex to discover data in assets, a discovery log is produced in Logging.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

In the Google Cloud console, go to Cloud Logging > Logs (Logs Explorer):

Go to Logs Explorer
In the Logs Explorer view, find the Query tab.
Click the Resource drop-down.
Select Cloud Dataplex Zone. Click Add.
Click the Log name drop-down.
Type dataplex.googleapis.com%2Fdiscovery. Select discovery and click Add.

(Optional) Filter the logs to a specific asset by adding the following filters in the log query:

resource.labels.location="LOCATION"
resource.labels.lake_id="LAKE_ID"
resource.labels.zone_id="ZONE_ID"
jsonPayload.assetId="ASSET_ID"

Click Run query.

gcloud

The Google Cloud CLI has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/Zone" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery AND
    resource.labels.location=LOCATION AND
    resource.labels.lake_id=LAKE_ID AND
    resource.labels.zone_id=ZONE_ID AND
    jsonPayload.assetId=ASSET_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

Query process logs

When you use Dataplex to schedule and run Tasks, a process log is produced in Logging for the resulting job.

You can access Logging using the Logging console, the gcloud logging command, or the Logging API.

Console

In the Google Cloud console, go to Cloud Logging > Logs (Logs Explorer):

Go to Logs Explorer
In the Logs Explorer view, find the Query tab.
Click the Resource drop-down.
Select Cloud Dataplex Task. Click Add.
Click the Log name drop-down.
Type dataplex.googleapis.com%2Fprocess. Select process and click Add.

(Optional) Filter the logs to a specific task by adding the following filters in the log query:

resource.labels.location="LOCATION"
resource.labels.lake_id="LAKE_ID"
resource.labels.task_id="TASK_ID"

Click Run query.

gcloud

The Google Cloud SDK has a group of commands, gcloud logging, that provide a command-line interface to the Cloud Logging API. To read your log entries, run the following command.

gcloud logging read \
    'resource.type="dataplex.googleapis.com/Task" AND
    logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess AND
    resource.labels.location=LOCATION AND
    resource.labels.lake_id=LAKE_ID AND
    resource.labels.task_id=TASK_ID'
    --limit 10

REST

Follow the Logging API instructions to list log entries (entries.list) by using the APIs Explorer.

What's next

Learn more about Cloud Logging.
Learn about Dataplex monitoring.