On-premises transfer log format

This document describes the current Transfer service for on-premises data transfer log format.

Format description

Transfer service for on-premises data logs are saved in tab-separated values (TSV) format, and contain a header row with field names. The order that the fields are displayed are generally stable, but aren't guaranteed, and must be inferred from the header row.

Each line represents a single log record, that is a status update on part of the overall transfer operation.

The following table describes the fields in the log file:

Log field Description
Timestamp ISO 8601-compliant timestamp when we recorded the event.
Operation Name The fully-qualified operation name.
Action Describes the action of this particular task. One of the following:
  • FIND: Finding work to do, such as listing files on-premises
  • TRANSFER: Uploading files
  • DELETE: Deleting files at destination
ActionStatus The high-level status of the action. One of the following:
  • SUCCEEDED: The action completed successfully.
  • FAILED: The action failed. Additional details about the failure are in the FailureDetails fields.
FailureDetails.ErrorType A string that represents the type of error encountered. For example, FILE_NOT_FOUND. Populated only if an error was encountered.
FailureDetails.GrpcCode A string value of the Google RPC code. For example, FAILED_PRECONDITION. Populated only if an error was encountered.
FailureDetails.Message A human-readable error message for the failure. Populated only if an error was encountered.
Src.Type String that describes the source's storage system type. This is always ON_PREM.
Src.File.Path Path of the transferred on-premises file.
Src.File.LastModified The POSIX modified time (mtime) of the on-premises file.
Src.FileSize The file size in bytes.
Src.File.Crc32C The CRC32C checksum of the file's contents.
Dst.Type String that describes the destination storage system type. This is always GCS.
Dst.Gcs.BucketName The destination Cloud Storage bucket name.
Dst.Gcs.ObjectName The object prefix for the destination object.
Dst.Gcs.LastModified The POSIX modified time (mtime) of the destination object.
Dst.Gcs.Size The object size in bytes.
Dst.Gcs.Crc23C The CRC32C checksum of the object's contents.
Dst.Md5 The MD5 checksum of the final object in Cloud Storage.

Example log entry

The following is an example of two lines of log output, the header and a data line:

Timestamp       OperationName   Action  ActionStatus    FailureDetails.ErrorType        FailureDetails.GrpcCode FailureDetails.Message     Src.Type        Src.File.Path   Src.File.LastModified   Src.File.Size   Src.File.Crc32C Dst.Type   Dst.Gcs.BucketName      Dst.Gcs.ObjectName      Dst.Gcs.LastModified    Dst.Gcs.Size    Dst.Gcs.Crc32C  Dst.Gcs.Md5
2019-10-18T21:06:17Z    transferOperations/agent-manual_transferJobs/OPI1494275376193533620_0000000001571432448     TRANSFER        SUCCEEDED                               ON_PREM_FILE    /mnt/disks/disk-1/directory1/file.txt      1561409931      406     3089075447      GCS_OBJECT      destination-gcs-bucket    file.txt  1571432776      406     3089075447      FDjya7dWbd0OrgnZ7g9ZJw==