Troubleshooting Transfer for on-premises

This document describes how to troubleshoot and resolve transfer and agent issues, and where to find agent logs to help you troubleshoot issues.

Troubleshooting transfer errors in Cloud Console

The following table describes transfer error messages, and how to resolve them:

Error message Error type What the error means How to resolve the error
Modified during transfer FILE_MODIFIED_FAILURE The source file was modified during the upload each time that Transfer service for on-premises data attempted to upload the source file. Prevent writes to the specified file during the next Transfer service for on-premises data operation.
Failed to transfer PRECONDITION_FAILURE The Cloud Storage object associated with the source file was modified each time that Transfer service for on-premises data attempted to upload the file. Prevent multiple transfer jobs from writing the same file to the same Cloud Storage bucket by using unique Cloud Storage object prefixes when you create transfer jobs.
Source directory not found SOURCE_DIR_NOT_FOUND Either the specified source path is incorrect, or the path is correct but not all agents have access to the path. Check the transfer job configuration and verify that:
File not found FILE_NOT_FOUND_FAILURE The source file was found, but deleted before it was transferred to Cloud Storage. If the file was mistakenly deleted, restore it so that the next transfer job can upload it.
Failed to find the destination bucket BUCKET_NOT_FOUND The destination bucket doesn't exist in Cloud Storage. Verify that the destination bucket spelling is correct and that it exists.
Failed to find an internal metadata object METADATA_OBJECT_NOT_FOUND_FAILURE Transfer service for on-premises data stores metadata in the destination bucket with the prefix storage-transfer. If the metadata files are deleted before their corresponding transfer operations complete, then this error is displayed. Avoid deleting objects with the prefix storage-transfer/ in the destination bucket until after all transfer jobs complete.
Failed due to invalid file name INVALID_FILE_NAME The path of a source file is invalid. Verify and fix the specified file path. Verify that the path uses characters that are supported by Cloud Storage
Failed due to permissions PERMISSION_FAILURE An agent didn't have sufficient permissions to perform an operation. There are two possibilities for this error:
  • An agent had insufficient Google Cloud permissions.
  • An agent was unable to read a file or directory due to insufficient permissions on the source file system.
Verify following:
  • Ensure that your agents have the following Cloud IAM roles:
    • roles/pubsub.editor
    • roles/storage.admin for all destination buckets
    Either of these roles can be granted to a service account that is used by the agents, or a user with these roles can use their default credentials when, installing agents.
  • Ensure that all paths on the source file system are readable by each agent.
Service lacked sufficient permissions SERVICE_PERMISSION_FAILURE Transfer service for on-premises data didn't have sufficient permissions to perform an operation. Transfer service for on-premises data uses the service account, cloud-ingest-dcp@cloud-ingest.iam.gserviceaccount.com to access resources. Verify that the service account has the following roles:
  • roles/pubsub.editor for the project.
  • roles/storage.admin for all destination buckets.
Agent unsupported AGENT_UNSUPPORTED_VERSION The agent version is no longer compatible with Transfer service for on-premises data. This is a temporary error, related to a bad agent update. If it occurs, do the following:
  1. Stop all of your agents.
  2. Pull the latest docker image by running: sudo docker pull gcr.io/cloud-ingest/tsop-agent
  3. Issue the Docker run command to start all of your agent containers.
If the issue persists, reach out to your support team.
Failed due to hash mismatch HASH_MISMATCH_FAILURE Each time Transfer service for on-premises data tried to upload this file, the uploaded bytes were corrupted. This resulted in the hash of the on-premises file not matching the hash of the resulting Cloud Storage object. This error may be caused by a number of potential issues. If you see a small percentage of hash mismatch failures (less than 1%) in a large transfer, retry the failed files. If you see a large percentage of hash mismatch failures (1% or greater), we recommend investigating potential memory, CPU, or other hardware failures on the agent machine.
Failed due to an unsupported file mode UNSUPPORTED_FILE_MODE Transfer service for on-premises data encountered a file with an unsupported mode, such as a device, socket, named pipe, or irregular file. Remove these special file types from the source directory.
Failed due to an error in the filesystem FILESYSTEM_ERROR An agent encountered a filesystem or operating system error when performing a filesystem operation such as read, seek, or stat. Read the failure description to understand which filesystem operation failed. Ensure the filesystem is accessible to the on-prem agent and responsive to basic file operations.
Failed due to an unknown error UNKNOWN_FAILURE An unexpected error occurred. Read the failure description. If the failure description does not contain sufficient information to resolve the issue, please contact support.

Viewing agent logs

Agent logs contain information relevant to agent processes, and can help you troubleshoot agent connection problems. If your agents are listed as connected in Google Cloud Console and you are experiencing transfer failures, see Viewing errors to view a sample of transfer errors. To view logs that contain a record of every file Storage Transfer Service considered during a transfer, see Viewing transfer logs.

By default, agent logs are stored in /tmp. You can change the location with the --log-dir=logs-directory command-line option.

The logs are named:

agent.hostname.username.log.log-level.timestamp

Where:

  • hostname - hostname the agent is running on.
  • username - username running the agent.
  • log-level is one of:
    • INFO - informational messages
    • ERROR - errors encountered during transfer, but that don't prevent the transfer job from continuing.
    • FATAL - errors encountered that prevent the transfer job from continuing.
  • timestamp - timestamp in YYYYMMDD-hhmmss.thread-id format.

The logs directory contains symlinks to the most recent logs for each of the priority levels:

  • agent.ERROR
  • agent.FATAL
  • agent.INFO

Troubleshooting agent errors

The following sections describe how to troubleshoot and resolve Transfer for on-premises agent errors:

Agents are not connected

If Transfer for on-premises agents are not displayed as connected within Google Cloud Console:

  1. Verify that agents can connect to Cloud Storage APIs and Pub/Sub APIs and there are no network or authentication issues:

    1. Run the following command from the same machine as the transfer agent to test the agent's connection to Cloud Storage APIs:

      gsutil cp test.txt gs://my-bucket

      Replace:

      my-bucket with the name of your Cloud Storage bucket.

    2. Run the following command from the same machine as the transfer agent to test the agent's connection to Pub/Sub APIs:

      gcloud pubsub topics list --project=project-id

      Replace:

      project-id with the name of your Google Cloud project name.

  2. If your project uses VPC Service Controls, view the agent logs for errors. If VPC Service Controls is misconfigured, the INFO agent logs will contain the following error:

    Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: id

    In this output:

Agents are connected but jobs fail

If agents are displayed as connected but transfer jobs fail, check the error details of the failed jobs.