An agent pool is a collection of agents that use the same configuration, with uniform access and visibility to your source and destination. Agent pools also provide control over transfer bandwidth limits.
All agents must belong to an agent pool.
This guide describes how to use agent pools.
Before you begin
You must have the correct permissions on your project to work with agent pools. Refer to Access control with IAM for details.
You can view your current permissions.
To use gcloud
commands,
install the gcloud command-line tool.
Create an agent pool
To create an agent pool:
Google Cloud console
In the Google Cloud console, go to the Agent pools page.
The Agent pools page is displayed, listing your existing agent pools.
Click Create another pool.
Name your pool, and optionally describe it.
You may choose to set a bandwidth limit that will apply to the pool as a whole. The specified bandwidth in MB/s will be split amongst all of the agents in the pool. See Manage network bandwidth for more information.
Click Create.
REST API
Use projects.agentPools.create:
POST https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools?agent_pool_id=AGENT_POOL_ID
Where:
PROJECT_ID
: The project ID that you're creating the agent pool in.AGENT_POOL_ID
: The agent pool ID that you are creating.
If an agent pool is stuck in the Creating
state for more than 30 minutes,
we recommend deleting the agent pool and creating it again.
Revoking required Storage Transfer Service permissions from a project while
an agent pool is in the Creating
state leads to incorrect service behavior.
gcloud CLI
To create an agent pool with the gcloud
command line tool, run
gcloud transfer agent-pools create
.
gcloud transfer agent-pools create AGENT_POOL
Where the following options are available:
AGENT_POOL is a unique, permanent identifier for this pool.
--no-async
blocks other tasks in your terminal until the pool has been created. If not included, pool creation runs asynchronously.--bandwidth-limit
defines how much of your bandwidth in MB/s to make available to this pool's agents. A bandwidth limit applies to all agents in a pool and can help prevent the pool's transfer workload from disrupting other operations that share your bandwidth. For example, enter '50' to set a bandwidth limit of 50 MB/s. By leaving this flag unspecified, this pool's agents will use all bandwidth available to them.--display-name
is a modifiable name to help you identify this pool. You can include details that might not fit in the pool's unique full resource name.
Assign agents to a pool
New agent pools are empty and must have agents assigned to them. An agent can only be assigned to an agent pool at the time of the agent's creation.
If an agent pool is not specified, the agent is assigned to the
transfer_service_default
pool.
To create an agent and assign it to a pool:
Google Cloud console
From the Agent pools page, select the pool to assign the agent(s) to.
Click Install agent. The Agent installation guide appears.
Follow the instructions to create the agent in this agent pool.
gcloud CLI
To use the gcloud
command-line tool to create an agent and assign it to a
pool, run gcloud transfer agents install
. The agent must
be installed on a machine with root access to the file system.
gcloud transfer agents install --pool=AGENT_POOL
To install agents to use with an S3-compatible source, provide access
credentials either as environment variables as the
values of AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
, or stored as
default credentials in your system's configuration files.
export AWS_ACCESS_KEY_ID=ID
export AWS_SECRET_ACCESS_KEY=SECRET
gcloud transfer agents install --pool=AGENT_POOL
See Manage transfer agents for more details.
For additional options, see Advanced agent options.
To manage agents, see Manage transfer agents.
Assign an agent pool to a job
Once your agent pool has been created, it can be assigned to a job during job creation or update.
Update an agent pool
You can update an agent pool's display name and bandwidth limit.
To update an agent pool:
Google Cloud console
In the Google Cloud console, go to the Agent pools page.
You can edit the following items:
The agent pool's display name: Click
Edit next to the current Agent pool name.The agent pool's bandwidth limit: Click Set bandwidth limit, and enter a new bandwidth limit in the Set bandwidth limit dialog. Click Set Limit to apply the new bandwidth limit. The bandwidth is shared across all agents in the pool.
REST API
To update an agent pool, use
projects.agentPools.patch
with a field mask
of the fields to update. The following agent pool fields can be updated:
For example, to update the displayName
to my-transfer
, you would provide
the field mask "displayName,my-transfer"
.
The following is an example patch request to update the display name:
PATCH https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools/AGENT_POOL:"displayName,NEW_DISPLAY_NAME"
Where:
PROJECT_ID
: The project ID that you're updating the agent pool in.AGENT_POOL
: The name of the agent pool that you're updating.NEW_DISPLAY_NAME
: The new display name for this agent pool.
The following is an example patch request to update the bandwidth limit:
PATCH https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools/AGENT_POOL:"bandwidthLimit,NEW_LIMIT"
Where:
PROJECT_ID
: The project ID that you're updating the agent pool in.AGENT_POOL
: The agent pool ID that you are updating.NEW_LIMIT
: The new bandwidth limit for this agent pool.
gcloud CLI
To use the gcloud
command-line tool to update an agent pool, run
gcloud transfer agent-pools update
.
gcloud transfer agent-pools update AGENT_POOL \
[--bandwidth-limit=NEW_BANDWIDTH_LIMIT] [--clear-bandwidth-limit] \
[--clear-display-name] [--display-name=NEW_DISPLAY_NAME]
The following argument is required:
- AGENT_POOL is the unique, permanent identifier for this pool.
AGENT_POOL
identifies the agent pool to update, and cannot be updated itself.
The following flags can be specified with this command:
--bandwidth-limit
can be updated, or removed by specifying--clear-bandwidth-limit
instead. See Create an agent pool for more details about bandwidth limits.--display-name
can be updated, or deleted with the--clear-display-name
flag instead.
Delete an agent pool
You can delete an agent pool that has no active agents and no active job runs.
To delete an agent pool:
Google Cloud console
In the Google Cloud console, go to the Agent pools page.
Click
Delete. Read the pop-up, then click Delete to confirm.If the delete button isn't active, you'll need to stop all agents and jobs that are associated with this pool.
REST API
Use projects.agentPools.delete:
DELETE https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools/AGENT_POOL
Where:
PROJECT_ID
: The project ID that you're deleting the agent pool from.AGENT_POOL
: The agent pool name.
gcloud CLI
To use the gcloud
command-line tool to delete an agent pool, run
gcloud transfer agent-pools delete
.
gcloud transfer agent-pools delete AGENT_POOL
The following argument is required:
- AGENT_POOL is the unique, permanent identifier for this pool.
Before you can delete a pool, all agents in the pool must be stopped, the pool's jobs must be disabled, and there must be no in-progress transfer operations for this pool.
To view all jobs associated with a pool, select your pool in the Google Cloud console and click the Transfer jobs tab.
Get an agent pool
To get an agent pool:
Google Cloud console
In the Google Cloud console, go to the Agent pools page.
The page displays a list of all agent pools associated with your project, and the following information for each agent pool:
- Agent pool name
- Connection status
- Number of connected agents
- Bandwidth limit, if set
- Number of associated transfer jobs
- The agent pool's display name
To view a specific agent pool, click the agent pool's Name.
The following actions are available from an agent pool's information page:
- Install agent: displays instructions for installing Transfer agents.
- Stop agent: Select an agent in the table, then click Stop Agent.
- List transfer jobs: Click Transfer Jobs to display the transfer jobs agents in this pool are connected to.
REST API
GET https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools/AGENT_POOL
Where:
PROJECT_ID
: The project ID that you're getting an agent pool of.AGENT_POOL
: The agent pool to retrieve.
gcloud CLI
To use the gcloud
command-line tool to retrieve an agent pool, run
gcloud transfer agent-pools describe
.
gcloud transfer agent-pools describe AGENT_POOL
The following argument is required:
- AGENT_POOL is the unique, permanent identifier for this pool.
List agent pools
To list your agent pools:
Google Cloud console
In the Google Cloud console, go to the Agent pools page.
All agent pools are displayed.
REST API
GET https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools
Where:
PROJECT_ID
: The project ID for which to list agent pools.
gcloud CLI
To use the gcloud
command-line tool to list agent pools, run
gcloud transfer agent-pools list
.
gcloud transfer agent-pools list \
[--limit=LIMIT] \
[--names=[NAMES,...]] \
[--page-size=PAGE_SIZE]
The following flags can be specified with this command:
--limit
defines the maximum number of agent pools to return.--names
specifies the names of the agent pools to list. Separate multiple names with commas; for example,--names=foo,bar
. If not specified, all agent pools in your current project are listed.--page-size
defines the batching behavior for this command. By default, thelist
command batches responses into 256 items, automatically fetching the next batch until all items are listed orlimit
is reached.
The command returns a list of matching agent pools and their configuration details. The following example shows a sample response.
name: projects/my-project-123/agentPools/my-first-pool
state: CREATED
---
bandwidthLimit:
limitMbps: '120'
displayName: Source A to destination Z.
name: projects/my-project-123/agentPools/a2z-pool
state: CREATED