Transfer Service for on-premises data Overview

This page describes Transfer Service for on-premises data, its requirements, and its features.

What is Transfer Service for on-premises data?

Transfer Service for on-premises data is a software service that enables you to transfer large amounts of data from your data center to a Cloud Storage bucket. It is well suited for customers that are moving billions of files and 100s of TB of data in a single transfer. It can scale to network connections in the 10s of Gbps.

Why should I use Transfer Service for on-premises data?

Transfer Service for on-premises data enables you to transfer large volumes of data without writing custom scripts or buying off-the-shelf solutions. Custom scripts can be:

  • Unreliable
  • Slow
  • Insecure
  • Difficult to maintain and troubleshoot

Off-the-shelf solutions can be costly to deploy.

Transfer Service for on-premises data is a scalable, reliable, and managed service that enables you to move your data without investing in engineering teams or buying transfer solutions. You install a Docker container containing the on-premises agent for Linux on your data center's computers, and Transfer Service for on-premises data coordinates the agents to transfer your data securely to Cloud Storage.

I have limited bandwidth. Can I use Transfer Service for on-premises data?

You can set a bandwidth limit used by your Google Cloud project to limit the rate at which the on-premises agents copy files to Google Cloud from your machines.

If you have limited bandwidth, you can still use Transfer Service for on-premises data. You can set a bandwidth limit for your Google Cloud project, which limits the rate that on-premises agents copy data to Google Cloud. The bandwidth limit is shared across all transfer jobs and their associated on-premises agents within your Google Cloud project.

How does it work?

The following is a high-level overview of how Transfer Service for on-premises data works:

  1. Install Docker and run a small piece of software, called an agent, in your private data center. The agent runs within a Docker container and has access to your locally mounted NFS data.

    See Installing and running the on-premises agent for more information.

  2. Complete Transfer for on-premises first-time setup.

  3. Start a Transfer Service for on-premises data transfer from the Google Cloud Console. You'll provide the NFS directory and a destination Cloud Storage bucket to transfer data to.

    See Creating a transfer job for more information.

  4. When the transfer starts, it recursively traverses through the given NFS directory and moves data it finds to your Cloud Storage bucket.

    Transferred data is checksummed, files with errors are re-tried, and data is sent via a secure connection. A record of the transfer's progress is written to log objects within your destination Cloud Storage bucket. You can track the progress of the transfer within the Google Cloud Console.

  5. When the transfer completes, you can view error samples within the Google Cloud Console. You can also review the transfer log for a catalog of files transferred and any errors.

What requirements does Transfer Service for on-premises data have?

To use Transfer for on-premises, you need:

  • A POSIX-compliant source.

  • Network connection that is 300Mbps or faster.

  • A Docker-supported 64-bit Linux server or virtual machine that can access the data you plan to transfer.

    Docker Community Edition, supports CentOs, Debian, Fedora, and Ubuntu operating systems.

    To use other Linux operating systems, see Docker Enterprise.

  • Complete Transfer for on-premises first-time setup.

Before you start a transfer, verify that:

  • TCP ports 80 (HTTP) and 443 (HTTPS) are open for outbound connections.
  • All agent processes within a single Google Cloud project have the same filesystem mounted at the same mount point.

What's next?

Start your transfer by completing first-time setup.