Requirements for file system transfers

Before you create a file system transfer job, you'll need to enable required APIs and install Docker.

If you receive errors while performing first-time setup, confirm that the user you logged in with has permissions required to perform the set up steps. In many cases, these permissions are not available to all users, and you may need to contact a project administrator for assistance.

System requirements

To use Storage Transfer Service for file system transfers, you need:

The transfer agent machine requirements are listed below:

  • A Docker-supported 64-bit Linux server or virtual machine that can access the data you plan to transfer. Docker Community Edition, supports CentOs, Debian, Fedora, and Ubuntu operating systems.

    To use other Linux operating systems, see Docker Enterprise.

  • At least 8 GB of memory per container and at least four CPUs to be most effective.

  • Any system-level or user-level constraints on the number of allowed process IDs, such as configured with ulimit -u, must account for the number of agents you plan to run. Each agent process can create up to many hundreds of threads. We recommend removing process ID limits when running transfer agents.

    Customers using Podman to create agent containers should specify a --pids-limit value high enough to accommodate Storage Transfer Service agent threads. The default --pids-limit value is 2048, which may cause issues when running more than one agent. We recommend setting --pids-limit=-1 to allow unlimited process IDs per container.

Scaling restrictions on jobs and agents

Storage Transfer Service has the following scale restrictions on transfer jobs and agents:

  • Fewer than one billion files per job
  • Up to 100 agents per agent pool
  • Bandwidth cap must be over 1MBps

In addition, the number of agents and agent pools per project is limited by the number of Pub/Sub subscriptions they consume:

  • Each project supports up to 10,000 subscriptions.
  • An agent pool requires 20 subscriptions.
  • Each agent requires 1 subscription.

For example, an agent pool containing 100 agents requires 120 subscriptions. Therefore, a project can support 83 agent pools of 100 agents (10,000 / 120 = 83.3).

Enable APIs

  1. Enable the Google Storage Transfer API.

    Enable the API

  2. Enable the Pub/Sub API.

    Enable the API

Install Docker

Install Docker Community Edition on a physical or virtual Linux machine:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo systemctl enable docker

If you encounter an installation error, see Troubleshooting.

Allow Docker to authenticate

To allow the Docker container to authenticate with your gcloud default credentials, create a Docker volume containing a file with your application default credentials by running the following command:

sudo docker run -ti --name gcloud-config google/cloud-sdk gcloud auth application-default login

What's next?