Introducing Transfer Appliance in the EU for cloud data migration

We’re pleased to announce that Transfer Appliance is available in beta in the EU market today. Transfer Appliance is a high-capacity server that lets you move large amounts of data to Google Cloud Platform (GCP) quickly and securely. It’s recommended if you’re moving more than 20TB of data, or data that would take more than a week to upload.

You can request a Transfer Appliance directly from your GCP console. The service will be available in beta in the EU in a 100TB configuration with total usable capacity of 200TB. And it’ll soon be available in a 480TB configuration with a total usable capacity of a petabyte.

Moving HDFS clusters with Transfer Appliance  

Customers have been using Transfer Appliance to move everything from audio and satellite imagery archives to geographic and wind data. One popular use case is migrating Hadoop Distributed File System (HDFS) clusters to GCP.

We see lots of users run their powerful Apache Spark and Apache Hadoop clusters on GCP with Cloud Dataproc, a managed Spark and Hadoop service that allows you to create clusters quickly, then hand off cluster management to the service. Transfer Appliance is an easy way to migrate petabytes of data from on-premise HDFS clusters to GCP.

Earlier this year, we announced the ability to configure Transfer Appliance with one or more NFS volumes. This lets you push HDFS data to Transfer Appliance using Apache DistCp (also known as Distributed Copy)—an open source tool commonly used for intra/inter-cluster data copy. To copy HDFS data onto a Transfer Appliance, configure it with an NFS volume and mount it from the HDFS cluster. Then run DistCp with the mount point as the copy target. Once your data is copied to Transfer Appliance, ship it to us and we’ll load your data into Cloud Storage.

If you are interested in migrating your HDFS data to GCP, request an appliance here. Learn more about the benefits of moving HDFS to GCP here.

Using Transfer Appliance in production

EU customers such as Candour Creative, which helps their clients tell stories through films and photographs, wanted to take advantage of having their content readily available in the cloud. But Zac Crawley, Director at Candour, was facing some challenges with the move.

“Multiple physical backups of our data were taking up space and becoming costly,” Crawley says. “But when we looked at our network, we figured it would take a matter of months to move the 40TBs of large file data. Transfer Appliance reduced that time significantly.”

TerraLoupe, a provider of aerial imagery analytics, has been using Transfer Appliance to move fast-growing datasets quickly.

“At Terraloupe, we analyze aerial imagery with deep learning algorithms,” says Niklas Barkmeyer, Product Manager at TerraLoupe. “Depending on the geographic areas and image resolutions, these geospatial datasets are typically several terabytes large for single cities and can reach petabytes for entire countries or continents. Transfer Appliance was the fastest and most convenient way for us to get these terabytes to the cloud for analytics.”

Transfer Appliance can help you meet strict data protection requirements, if required.  The entire Transfer Appliance service, from shipping to data upload to GCP, is handled in the EU, so your data won’t leave the EU. You’ll be able to choose which region the data uploads to during the signup process. Be sure to review Google Cloud’s commitments to the General Data Protection Regulation, as well as your organization’s commitment to your customers regarding their data. The data you send with Transfer Appliance is encrypted at capture. You hold the encryption keys, and you decrypt your own data as you upload it to its final Cloud Storage destination.   

To learn more about Transfer Appliance, check out the documentation.