Storage Transfer Service agents are applications running inside a Docker container, that coordinate with Storage Transfer Service to read data from POSIX filesystem sources, and/or write data to POSIX filesystem sinks.
If your transfer does not involve a POSIX filesystem, you do not need to set up agents.
This document describes how to administer Transfer service for on-premises data agents on your servers. All agents are assigned to an agent pool.
Install and run an on-premises agent
To install and run an on-premises agent:
Cloud console
In the Cloud console, go to the Transfer service for on-premises data page.
Click Connection Settings.
Select the agent pool to which to add the new agent.
Click Install Agent.
Follow the instructions to install and run the agent.
For more information about the agent's command-line options, see Agent command-line options.
gcloud CLI
To install one or more agents using the gcloud
tool, follow these steps.
Follow the instructions to Configure the
gcloud
tool.Run
gcloud transfer agents install
:gcloud transfer agents install --pool=POOL_NAME --count=1
The tool walks you through any required steps to install the agent(s). This command installs one agent on your machine, mapped to the pool name specified as
POOL_NAME
, and authenticates the agent using your gcloud credentials. The pool name must exist, or an error is returned.
To run agents using a
service account key, use
the --creds-file
option:
gcloud transfer agents install --pool=my-agent-pool --count=3 \
--creds-file=/relative/path/to/service-account-key.json
For a full list of optional flags, run
gcloud transfer agents install --help
.
We recommend installing more than one agent for each machine. For more information about determining how many agents to run, see Maximizing transfer agent performance.
Confirm agent connections
After you install your on-premises agents, you can verify that they're connected to your Transfer service for on-premises data agent pool.
To confirm that your agents are connected:
In the Cloud console, go to the Transfer service for on-premises data page.
Click Connection Settings. Your agent pools are displayed, with the number of connected agents.
Select an agent pool to view details on connected agents.
If a new agent doesn't show up in the agent pool page within ten minutes of its creation, see Agents are not connected.
Monitor agent activity
You can use Cloud Monitoring alerts to monitor agent activity.
Monitoring is available along project
, agent_pool
, and agent_id
dimensions.
Using this monitoring data, you can set up alerts to notify you of potential issues with your transfer. To do so, create an alert on either of the following Google Cloud metrics:
Metric name | What it describes | Suggested uses |
---|---|---|
storagetransfer.googleapis.com/agent/transferred_bytes_count | Measures how quickly a specific agent is moving data across all jobs that it services at a point in time. | Alert for dips in performance. |
storagetransfer.googleapis.com/agent/connected | A boolean that is True for each agent that Google Cloud received a recent heartbeat message from. |
|
Stop an agent
To stop an agent, run docker stop
on the agent's Docker container ID. To find
the ID:
In the Cloud console, go to the Transfer service for on-premises data page.
Click Connection Settings.
Select the agent pool containing the agent to stop.
Select an agent from the list. Use the Filter field to search for prefixes, agent status, agent age, and more.
Click Stop agent. The
docker stop
command with the specific container ID is displayed.Run the command on the machine on which the agent is running. A successful
docker stop
command returns the container ID.
Once stopped, the agent is shown in the agent pools list as Disconnected.
Delete an agent
To delete specific agents, list which agents are running on your machine:
docker container list --all --filter ancestor=gcr.io/cloud-ingest/tsop-agent
Then pass the agent IDs to transfer agents delete
:
gcloud transfer agents delete --ids=id1,id2,…
To delete all agents running on the machine, use either the --all
flag
or the --uninstall
flag. Both flags delete all agents on the machine;
the --uninstall
flag additionally uninstalls the agent Docker image.
gcloud transfer agents delete --all
gcloud transfer agents delete --uninstall