Dataplex job logs can be viewed, searched, filtered, and archived in Cloud Logging.
See Google Cloud Observability pricing to understand your costs.
See Logs retention periods for information on logging retention.
See Exclusion filters to disable all logs or exclude logs from Logging.
See Routing and storage overview to route logs from Logging to Cloud Storage, BigQuery, or Pub/Sub.
Access Dataplex service logs in Logging
Dataplex publishes the following service logs to Cloud Logging.
Log type | Log name | logName query |
Log description |
---|---|---|---|
Data scan event logs | dataplex.googleapis.com/data_scan |
logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_scan) |
Event logs for data scan jobs indicating job state, results, and statistics |
Data quality scan rule result logs | dataplex.googleapis.com/data_quality_scan_rule_result |
logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result) |
Results of data quality scan rules in a data quality job |
Discovery logs | dataplex.googleapis.com/discovery |
logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery) |
Discovery progress and updates over assets in a zone |
Metadata job logs | dataplex.googleapis.com/metadata_job |
logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fmetadata_job) |
Logs about metadata import jobs and import items in the metadata import file |
Process logs | dataplex.googleapis.com/process |
logName=(projects/$PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess) |
Job runs resulting from data processing tasks |
You can access Logging by using the
Logs Explorer in the
Google Cloud console, the
gcloud logging
commands, or
the Logging API.
Query data scan event logs
When you use Dataplex to create and run a data scan, a data scan event log is produced in Logging for the resulting job.
Console
In the Google Cloud console, go to the Logs explorer page.
In the Logs Explorer view, find the Query tab.
Click the Resource menu.
Select Cloud Dataplex DataScan. Click Apply.
Click the Log name menu.
In the Search log names field, enter
dataplex.googleapis.com%2Fdata_scan
. Select data_scan and click Apply.Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:
resource.labels.location="LOCATION" resource.labels.datascan_id="DATA_SCAN_ID"
Click Run query.
gcloud
To read your data scan event log entries, use the
gcloud logging read
command
with the following query:
gcloud logging read \ 'resource.type="dataplex.googleapis.com/DataScan" AND logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_scan AND resource.labels.location=LOCATION AND resource.labels.datascan_id=DATA_SCAN_ID' --limit 10
REST
To list log entries, use the
entries.list
method.
Query data quality scan rule result logs
When you use Dataplex to create and run a data quality scan, a data quality scan rule result log is produced in Logging for the resulting job.
Console
In the Google Cloud console, go to the Logs explorer page.
In the Logs Explorer view, find the Query tab.
Click the Resource menu.
Select Cloud Dataplex DataScan. Click Apply.
Click the Log name menu.
In the Search log names field, enter
dataplex.googleapis.com%2Fdata_quality_scan_rule_result
. Select data_quality_scan_rule_result and click Apply.Optional: Filter the logs to a specific data scan ID or location by adding the following filters in the log query:
resource.labels.location="LOCATION" resource.labels.datascan_id="DATA_SCAN_ID"
Click Run query.
gcloud
To read your data quality scan rule result log entries, use the
gcloud logging read
command
with the following query:
gcloud logging read \ 'resource.type="dataplex.googleapis.com/DataScan" AND logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdata_quality_scan_rule_result AND resource.labels.location=LOCATION AND resource.labels.datascan_id=DATA_SCAN_ID' --limit 10
REST
To list log entries, use the
entries.list
method.
Query discovery logs
When you use Dataplex to discover data in assets, a discovery log is produced in Logging.
Console
In the Google Cloud console, go to the Logs explorer page.
In the Logs Explorer view, find the Query tab.
Click the Resource menu.
Select Cloud Dataplex Zone. Click Apply.
Click the Log name menu.
In the Search log names field, enter
dataplex.googleapis.com%2Fdiscovery
. Select discovery and click Apply.Optional: Filter the logs to a specific asset by adding the following filters in the log query:
resource.labels.location="LOCATION" resource.labels.lake_id="LAKE_ID" resource.labels.zone_id="ZONE_ID" jsonPayload.assetId="ASSET_ID"
Click Run query.
gcloud
To read your discovery log entries, use the
gcloud logging read
command
with the following query:
gcloud logging read \ 'resource.type="dataplex.googleapis.com/Zone" AND logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fdiscovery AND resource.labels.location=LOCATION AND resource.labels.lake_id=LAKE_ID AND resource.labels.zone_id=ZONE_ID AND jsonPayload.assetId=ASSET_ID' --limit 10
REST
To list log entries, use the
entries.list
method.
Query metadata job logs
When you run a metadata import job, metadata job logs are produced in Logging.
Console
In the Google Cloud console, go to the Logs explorer page.
In the Logs Explorer view, find the Query tab.
Click the Resource menu.
Select Cloud Dataplex Metadata Job.
Optional: To filter the logs to a specific location or metadata job ID, select a location or job ID.
Click Apply.
Click the Log name menu.
Type
dataplex.googleapis.com%2Fmetadata_job
and then select metadata_job.Click Apply.
gcloud
To read your metadata job log entries, use the
gcloud logging read
command
with the following query:
gcloud logging read \ 'resource.type="dataplex.googleapis.com/MetadataJob" AND logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fmetadata_job AND resource.labels.location=LOCATION AND resource.labels.metadata_job_id=METADATA_JOB_ID --limit 10
REST
To list log entries, use the
entries.list
method.
Query process logs
When you use Dataplex to schedule and run tasks, a process log is produced in Logging for the resulting job.
Console
In the Google Cloud console, go to the Logs explorer page.
In the Logs Explorer view, find the Query tab.
Click the Resource menu.
Select Cloud Dataplex Task. Click Apply.
Click the Log name menu.
In the Search log names field, enter
dataplex.googleapis.com%2Fprocess
. Select process and click Apply.Optional: Filter the logs to a specific task by adding the following filters in the log query:
resource.labels.location="LOCATION" resource.labels.lake_id="LAKE_ID" resource.labels.task_id="TASK_ID"
Click Run query.
gcloud
To read your process log entries, use the
gcloud logging read
command
with the following query:
gcloud logging read \ 'resource.type="dataplex.googleapis.com/Task" AND logName=projects/PROJECT_ID/logs/dataplex.googleapis.com%2Fprocess AND resource.labels.location=LOCATION AND resource.labels.lake_id=LAKE_ID AND resource.labels.task_id=TASK_ID' --limit 10
REST
To list log entries, use the
entries.list
method.
What's next
- Learn more about Cloud Logging.
- Learn about Dataplex monitoring.