This document describes the current file system transfer log format.
View transfer logs
File system transfers produce detailed transfer logs that you can use to verify the results of your transfer job. Each job produces a collection of transfer logs that are stored in the transer's Cloud Storage bucket: the source bucket, destination bucket, or intermediate bucket, depending on the transfer direction.
Logs are produced while the transfer job is running. The complete logs are typically available within 15 minutes of job completion.
View logs in the Cloud Storage bucket
Transfer logs are stored in the transfer's Cloud Storage bucket at the following path:
bucket-name/storage-transfer/logs/transferJobs/job-name/transferOperations/operation-name
where:
bucket-name
is the name of the Cloud Storage bucket involved in this transfer.job-name
is the job name, as displayed in the job list.operation-name
is the name of the individual transfer operation, comprised as the IS08601 timestamp and generated ID.
Logs are aggregated and stored as objects. Each batch of logs is named by its creation time. For example:
my
bucket/storage-transfer/logs/transferOperations/job1/2019-10-19T10_52_56.519081644-07_00.log
Run BigQuery queries on transfer logs
To run BigQuery queries on your transfer logs:
Example queries
Display the number of files that attempted transfer and whether they failed or succeeded
select ActionStatus, count(*) as num_files from big-query-table where Action="TRANSFER" group by 1;
Where big-query-table
is the name of the
BigQuery table that contains the transfer log.
Display all files that failed to transfer
select Src_File_Path from big-query-table where Action="TRANSFER" and ActionStatus="FAILED";
Where big-query-table
is the name of the
BigQuery table that contains the transfer log.
Display checksum and timestamp for each file that successfully transferred
select Timestamp, Action, ActionStatus, Src_File_Path, Src_File_Size, Src_File_Crc32C, Dst_Gcs_BucketName, Dst_Gcs_ObjectName, Dst_Gcs_Size, Dst_Gcs_Crc32C, Dst_Gcs_Md5 from big-query-table where Action="TRANSFER" and ActionStatus="SUCCEEDED";
Where big-query-table
is the name of the
BigQuery table that contains the transfer log.
Display all error information for directories that failed to transfer
select FailureDetails_ErrorType, FailureDetails_GrpcCode, FailureDetails_Message from big-query-table where Action="FIND" and ActionStatus="FAILED";
Where big-query-table
is the name of the
BigQuery table that contains the transfer log.
Format description
File system transfer logs are saved in tab-separated values (TSV) format, and contain a header row with field names. The order that the fields are displayed are generally stable, but aren't guaranteed, and must be inferred from the header row.
Each line represents a single log record, that is a status update on part of the overall transfer operation.
The following table describes the fields in the log file:
Log field | Description |
---|---|
Timestamp | ISO 8601-compliant timestamp when we recorded the event. |
Operation Name | The fully-qualified operation name. |
Action | Describes the action of this particular task. One of the following:
|
ActionStatus | The high-level status of the action. One of the following:
|
FailureDetails.ErrorType | A string that represents the type of error encountered. For example, FILE_NOT_FOUND. Populated only if an error was encountered. |
FailureDetails.GrpcCode | A string value of the Google RPC code. For example, FAILED_PRECONDITION. Populated only if an error was encountered. |
FailureDetails.Message | A human-readable error message for the failure. Populated only if an error was encountered. |
Src.Type | String that describes the source's storage system type. This is always ON_PREM. |
Src.File.Path | Path of the transferred on-premises file. |
Src.File.LastModified | The POSIX modified time (mtime) of the on-premises file. |
Src.FileSize | The file size in bytes. |
Src.File.Crc32C | The CRC32C checksum of the file's contents. |
Dst.Type | String that describes the destination storage system type. This is always
GCS . |
Dst.Gcs.BucketName | The destination Cloud Storage bucket name. |
Dst.Gcs.ObjectName | The object prefix for the destination object. |
Dst.Gcs.LastModified | The POSIX modified time (mtime) of the destination object. |
Dst.Gcs.Size | The object size in bytes. |
Dst.Gcs.Crc23C | The CRC32C checksum of the object's contents. |
Dst.Md5 | The MD5 checksum of the final object in Cloud Storage. |
Example log entry
The following is an example of two lines of log output, the header and a data line:
Timestamp OperationName Action ActionStatus FailureDetails.ErrorType FailureDetails.GrpcCode FailureDetails.Message Src.Type Src.File.Path Src.File.LastModified Src.File.Size Src.File.Crc32C Dst.Type Dst.Gcs.BucketName Dst.Gcs.ObjectName Dst.Gcs.LastModified Dst.Gcs.Size Dst.Gcs.Crc32C Dst.Gcs.Md5 2019-10-18T21:06:17Z transferOperations/agent-manual_transferJobs/OPI1494275376193533620_0000000001571432448 TRANSFER SUCCEEDED ON_PREM_FILE /mnt/disks/disk-1/directory1/file.txt 1561409931 406 3089075447 GCS_OBJECT destination-gcs-bucket file.txt 1571432776 406 3089075447 FDjya7dWbd0OrgnZ7g9ZJw==