Managing Transfer for on-premises agents

Storage Transfer Service agents are applications running inside a Docker container, that coordinate with Storage Transfer Service to read data from POSIX filesystem sources, and/or write data to POSIX filesystem sinks.

If your transfer does not involve a POSIX filesystem, you do not need to set up agents.

This document describes how to administer Transfer service for on-premises data agents on your servers. All agents are assigned to an agent pool.

Install and run an on-premises agent

Important: Users that create an on-premises transfer job can choose any source directory that is accessible by the agent, regardless of the data access restrictions in place in your data center. To prevent unauthorized access to data, select users that are granted access to the transfer project and that are permitted to run the user agent in alignment with your organization's security and access policies.

To install and run an on-premises agent:

Cloud console

1. In the Cloud console, go to the Transfer service for on-premises data page.

   Go to Transfer service for on-premises data

2. Click Connection Settings.

3. Select the agent pool to which to add the new agent.

4. Click Install Agent.

5. Follow the instructions to install and run the agent.

   For more information about the agent's command-line options, see Agent command-line options.

gcloud CLI

To install one or more agents using the gcloud tool, follow these steps.

1. Follow the instructions to Configure the gcloud tool.

2. Run gcloud transfer agents install:

   gcloud transfer agents install --pool=POOL_NAME --count=1
   

   The tool walks you through any required steps to install the agent(s). This command installs one agent on your machine, mapped to the pool name specified as POOL_NAME, and authenticates the agent using your gcloud credentials. The pool name must exist, or an error is returned.

To run agents using a service account key, use the --creds-file option:

gcloud transfer agents install --pool=my-agent-pool --count=3 \
 --creds-file=/relative/path/to/service-account-key.json

For a full list of optional flags, run gcloud transfer agents install --help.

We recommend installing more than one agent for each machine. For more information about determining how many agents to run, see Maximizing transfer agent performance.

Confirm agent connections

After you install your on-premises agents, you can verify that they're connected to your Transfer service for on-premises data agent pool.

To confirm that your agents are connected:

1. In the Cloud console, go to the Transfer service for on-premises data page.

   Go to Transfer service for on-premises data

2. Click Connection Settings. Your agent pools are displayed, with the number of connected agents.

3. Select an agent pool to view details on connected agents.

If a new agent doesn't show up in the agent pool page within ten minutes of its creation, see Agents are not connected.

Monitor agent activity

You can use Cloud Monitoring alerts to monitor agent activity.

Monitoring is available along project, agent_pool, and agent_id dimensions.

Using this monitoring data, you can set up alerts to notify you of potential issues with your transfer. To do so, create an alert on either of the following Google Cloud metrics:

Metric name	What it describes	Suggested uses
storagetransfer.googleapis.com/agent/transferred_bytes_count	Measures how quickly a specific agent is moving data across all jobs that it services at a point in time.	Alert for dips in performance.
storagetransfer.googleapis.com/agent/connected	A boolean that is True for each agent that Google Cloud received a recent heartbeat message from.	Alert for failing agents Failing below a number of agents that you consider necessary for reasonable performance Signal an issue with agent machines

Stop an agent

To stop an agent, run docker stop on the agent's Docker container ID. To find the ID:

1. In the Cloud console, go to the Transfer service for on-premises data page.

   Go to Transfer service for on-premises data

2. Click Connection Settings.

3. Select the agent pool containing the agent to stop.

4. Select an agent from the list. Use the Filter field to search for prefixes, agent status, agent age, and more.

5. Click Stop agent. The docker stop command with the specific container ID is displayed.

6. Run the command on the machine on which the agent is running. A successful docker stop command returns the container ID.

Once stopped, the agent is shown in the agent pools list as Disconnected.

Delete an agent

To delete specific agents, list which agents are running on your machine:

docker container list --all --filter ancestor=gcr.io/cloud-ingest/tsop-agent

Then pass the agent IDs to transfer agents delete:

gcloud transfer agents delete --ids=id1,id2,…

To delete all agents running on the machine, use either the --all flag or the --uninstall flag. Both flags delete all agents on the machine; the --uninstall flag additionally uninstalls the agent Docker image.

gcloud transfer agents delete --all
gcloud transfer agents delete --uninstall