Transfer Appliance is a high capacity storage server that you lease from Google. You connect it to your network, load it with data, and then ship it to an upload facility where the data is uploaded to Google Cloud Storage. Transfer Appliance enables you to securely transfer up to a petabyte of data on a single appliance.
Transfer Appliance comes in two sizes, 100 terabytes (TB) and 480 TB. The 100 TB appliance can store from 100 TB up to potentially 200 TB, depending on the deduplication and compression ratio of your data. Similarly, the 480 TB appliance can store from 480 TB up to potentially 1 petabyte (PB). Both rackable and standalone appliances are available.
If you have a typical network bandwidth of 100 Mbps, a petabyte of data takes about 3 years to upload. With Transfer Appliance, you can receive the appliance and capture a petabyte of data in under 25 days, and that data can be accessible in Google Cloud Storage within another 25 days, all without consuming a single bit of outbound network bandwidth.
Apply to participate in the Transfer Appliance beta by filling out the Transfer Appliance beta application form. For more information on Transfer Appliance pricing, see Google Transfer Appliance Pricing.
When should I use Transfer Appliance?
We recommend using Transfer Appliance if your dataset meets the following conditions:
- If it would take more than one week to upload your data.
- If you have 60 TB or more data, regardless of the connection speed.
Refer to the following table to see the approximate upload time for your dataset, based on size and network speed.
Transfer Appliance workflow
When you receive Transfer Appliance, you must first configure it and connect it to your network. Once that is done, you use Transfer Appliance utilities to capture and stream data to Transfer Appliance. Transfer Appliance can capture any data that can be accessed as a file in a POSIX compliant file system, whether attached directly or mounted over network shares.
Before being stored on Transfer Appliance, all captured data is deduplicated, compressed and encrypted with an industry standard AES 256 algorithm using a password and passphrase that you specify. You must specify the same password and passphrase to rehydrate — unencrypt, uncompress, and reconstruct — your data once it is uploaded to Google Cloud Platform (GCP).
Once all of your data is loaded onto Transfer Appliance, you run data integrity checks to make sure everything transferred completely and with no corruption. Once those are completed, you ship Transfer Appliance back to Google. Once Transfer Appliance is received, the encrypted data is copied from Transfer Appliance to your Google Cloud Storage staging bucket. You receive an email from Google when all of your data is available and you can start the process of rehydrating it.
To rehydrate your data, you run the Transfer Appliance Rehydrator application and specify a Cloud Storage destination bucket. Once all of your data is rehydrated to the destination bucket, you run data integrity checks to make sure everything transferred completely and with no corruption.
After your use of Transfer Appliance, it is securely wiped and re-imaged.
You use data capture jobs to identify data on your network and stream it to Transfer Appliance. The options for capturing data are as follows:
- Workstation capture
- To perform workstation capture, you install a capture utility on a Windows or Linux workstation. You use that utility to select directories that contain the data to capture and then stream that data to Transfer Appliance. Workstation capture typically delivers the greatest performance.
- For workstation capture, you should use a workstation that is dedicated to the capture process. Using a computer that also serves other purposes can tax system resources and slow down the capture process.
- Appliance capture
- To perform appliance capture, you connect directly to an NFS share from the Transfer Appliance. You select directories on that share that contain the data to capture and then stream that data to Transfer Appliance. Appliance capture provides the simplest configuration experience by avoiding the need to install the capture utility on a workstation.
You can run multiple capture jobs simultaneously, for example running several workstation capture jobs as well as an appliance capture job at once. The number of parallel capture jobs is bounded only by the limit of your system resources and network bandwidth to handle them. We recommend running multiple capture jobs at once in order to transfer data faster.
The capture utility automatically spawns up to 8 parallel capture tasks for each capture job, with each task handling up to 1 terabyte (TB) of data. This helps optimize performance and bandwidth utilization. As each capture task completes, a new one is created, until all targeted data has been captured.
By default, a data capture job dynamically chooses a port range to use. Each data capture task in the job requires its own data streaming port and chooses one from the range between the starting port number and ([port number] + [data capture tasks] - 1). For example, if the data capture job starts at port 50555 and uses the default of 8 for data capture tasks, the data capture tasks would use ports 50555-50562.
If a capture job fails, it can be restarted. Capture jobs checkpoint their progress, so a restarted capture job resumes from the last known good point to capture remaining data.
Once Transfer Appliance is received at the Google upload facility, the data is copied from Transfer Appliance to your Cloud Storage staging bucket. The data uploaded to your staging bucket is still compressed, deduplicated and encrypted. You rehydrate the data to reverse this process and return your data to a useable state. As it is rehydrated, it is moved to the Cloud Storage destination bucket you created.
You perform data rehydration by using a Rehydrator instance, which is a virtual appliance that runs as a Google Compute Engine instance on GCP.
Supported operating systems
The Transfer Appliance capture utility is supported on the following Linux and Microsoft Windows operating systems.
- Ubuntu 12.04 LTS or 14.04 LTS
- RHEL 6.4, 6.8, 7.0 or 7.2
- CentOS 6.4, 6.8, 7.0 or 7.2
- Windows 2012 R2
- Windows 2012
- Windows 2008 R2
- Windows 2008-64 bit
- Windows 2008-32 bit
You can use Transfer Appliance with the latest stable version of the following browsers.
- Internet Explorer