Manage agent pools

An agent pool is a collection of agents that use the same configuration, with uniform access and visibility to your source and destination. Agent pools also provide control over transfer bandwidth limits.

All agents must belong to an agent pool.

This guide describes how to use agent pools.

Before you begin

You must have the correct permissions on your project to work with agent pools. Refer to Access control with IAM for details.

You can view your current permissions.

To use gcloud commands, install the gcloud command-line tool.

Create an agent pool

Don't include sensitive information such as personally identifiable information (PII) or security data in your agent pool name. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

To create an agent pool:

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

    The Agent pools page is displayed, listing your existing agent pools.

  2. Click Create another pool.

  3. Name your pool, and optionally describe it.

  4. You may choose to set a bandwidth limit that will apply to the pool as a whole. The specified bandwidth in MB/s will be split amongst all of the agents in the pool. See Manage network bandwidth for more information.

  5. Click Create.

REST API

Use projects.agentPools.create:

POST https://storagetransfer.googleapis.com/v1/projects/PROJECT_ID/agentPools?agent_pool_id=AGENT_POOL_ID

Where:

  • PROJECT_ID: The project ID that you're creating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are creating.

If an agent pool is stuck in the Creating state for more than 30 minutes, we recommend deleting the agent pool and creating it again.

Revoking required Storage Transfer Service permissions from a project while an agent pool is in the Creating state leads to incorrect service behavior.

gcloud CLI

To create an agent pool with the gcloud command line tool, run gcloud transfer agent-pools create.

gcloud transfer agent-pools create NAME \
  [--no-async] \
  [--bandwidth-limit=BANDWIDTH_LIMIT] \
  [--display-name=DISPLAY_NAME]

Where the following options are available:

  • NAME is a unique, permanent identifier for this pool.

  • --no-async blocks other tasks in your terminal until the pool has been created. If not included, pool creation runs asynchronously.

  • --bandwidth-limit defines how much of your bandwidth in MB/s to make available to this pool's agents. A bandwidth limit applies to all agents in a pool and can help prevent the pool's transfer workload from disrupting other operations that share your bandwidth. For example, enter '50' to set a bandwidth limit of 50 MB/s. By leaving this flag unspecified, this pool's agents will use all bandwidth available to them.

  • --display-name is a modifiable name to help you identify this pool. You can include details that might not fit in the pool's unique full resource name.

Assign agents to a pool

New agent pools are empty and must have agents assigned to them. An agent can only be assigned to an agent pool at the time of the agent's creation.

If an agent pool is not specified, the agent is assigned to the transfer_service_default pool.

To create an agent and assign it to a pool:

Google Cloud console

  1. From the Agent pools page, select the pool to assign the agent(s) to.

  2. Click Install agent. The Agent installation guide appears.

  3. Follow the instructions to create the agent in this agent pool.

gcloud CLI

To use the gcloud command-line tool to create an agent and assign it to a pool, run gcloud transfer agents install. The agent must be installed on a machine with root access to the file system.

gcloud transfer agents install --pool=AGENT_POOL

To install agents to use with an S3-compatible source, provide access credentials either as environment variables as the values of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or stored as default credentials in your system's configuration files.

export AWS_ACCESS_KEY_ID=ID
export AWS_SECRET_ACCESS_KEY=SECRET
gcloud transfer agents install --pool=POOL_NAME

See Manage transfer agents for more details.

For additional options, see Advanced agent options.

To manage agents, see Manage transfer agents.

Assign an agent pool to a job

Once your agent pool has been created, it can be assigned to a job during job creation or update.

Update an agent pool

You can update an agent pool's display name and bandwidth limit.

To update an agent pool:

Google Cloud console

In the Google Cloud console, go to the Agent pools page.

Go to Agent pools

You can edit the following items:

  • The agent pool's display name: Click Edit next to the current Agent pool name.

  • The agent pool's bandwidth limit: Click Set bandwidth limit, and enter a new bandwidth limit in the Set bandwidth limit dialog. Click Set Limit to apply the new bandwidth limit. The bandwidth is shared across all agents in the pool.

REST API

To update an agent pool, use projects.agentPools.patch with a field mask of the fields to update. The following agent pool fields can be updated:

For example, to update the displayName to my-transfer, you would provide the field mask "displayName,my-transfer".

The following is an example patch request to update the display name:

PATCH https://storagetransfer.googleapis.com/v1/projects/{PROJECT_ID}/agentPools/{AGENT_POOL_ID}:"displayName,NEW_NAME"

Where:

  • PROJECT_ID: The project ID that you're updating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are updating.
  • NEW_NAME: The new display name for this agent pool.

The following is an example patch request to update the bandwidth limit:

PATCH https://storagetransfer.googleapis.com/v1/projects/{PROJECT_ID}/agentPools/{AGENT_POOL_ID}:"bandwidthLimit,NEW_LIMIT"

Where:

  • PROJECT_ID: The project ID that you're updating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are updating.
  • NEW_LIMIT: The new bandwidth limit for this agent pool.

gcloud CLI

To use the gcloud command-line tool to update an agent pool, run gcloud transfer agent-pools update.

gcloud transfer agent-pools update NAME \
  [--bandwidth-limit=NEW_BANDWIDTH_LIMIT] [--clear-bandwidth-limit] \
  [--clear-display-name] [--display-name=NEW_DISPLAY_NAME]

The following argument is required:

  • NAME is the unique, permanent identifier for this pool. NAME identifies the agent pool to update, and can not be updated itself.

The following flags can be specified with this command:

  • --bandwidth-limit can be updated, or removed by specifying --clear-bandwidth-limit instead. See Create an agent pool for more details about bandwidth limits.

  • --display-name can be updated, or deleted with the --clear-display-name flag instead.

Delete an agent pool

You can delete an agent pool that has no active agents and no active job runs.

To delete an agent pool:

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

  2. Click Delete. Read the pop-up, then click Delete to confirm.

    If the delete button isn't active, you'll need to stop all agents and jobs that are associated with this pool.

REST API

Use projects.agentPools.delete:

DELETE https://storagetransfer.googleapis.com/v1/{name=PROJECT_ID/*/agentPools/}

Where:

  • PROJECT_ID: The project ID that you're deleting the agent pool from.

gcloud CLI

To use the gcloud command-line tool to delete an agent pool, run gcloud transfer agent-pools delete.

gcloud transfer agent-pools delete NAME

The following argument is required:

  • NAME is the unique, permanent identifier for this pool.

Before you can delete a pool, all agents in the pool must be stopped, the pool's jobs must be disabled, and there must be no in-progress transfer operations for this pool.

To view all jobs associated with a pool, select your pool in the Google Cloud console and click the Transfer jobs tab.

Get an agent pool

To get an agent pool:

Google Cloud console

In the Google Cloud console, go to the Agent pools page.

Go to Agent pools

The page displays a list of all agent pools associated with your project, and the following information for each agent pool:

  • Agent pool name
  • Connection status
  • Number of connected agents
  • Bandwidth limit, if set
  • Number of associated transfer jobs
  • The agent pool's display name

To view a specific agent pool, click the agent pool's Name.

The following actions are available from an agent pool's information page:

  • Install agent: displays instructions for installing Transfer agents.
  • Stop agent: Select an agent in the table, then click Stop Agent.
  • List transfer jobs: Click Transfer Jobs to display the transfer jobs agents in this pool are connected to.

REST API

Use projects.agentPools.get:

GET https://storagetransfer.googleapis.com/v1/{name=PROJECT_ID/*/agentPools/}

Where:

  • PROJECT_ID: The project ID that you're getting an agent pool of.

gcloud CLI

To use the gcloud command-line tool to retrieve an agent pool, run gcloud transfer agent-pools describe.

gcloud transfer agent-pools describe NAME

The following argument is required:

  • NAME is the unique, permanent identifier for this pool.

List agent pools

To list your agent pools:

Google Cloud console

In the Google Cloud console, go to the Agent pools page.

Go to Agent pools

All agent pools are displayed.

REST API

Use projects.agentPools.list:

GET https://storagetransfer.googleapis.com/v1/projects/{project_id=PROJECT_ID}/agentPools

Where:

  • PROJECT_ID: The project ID that you're getting an agent pool of.

gcloud CLI

To use the gcloud command-line tool to list agent pools, run gcloud transfer agent-pools list.

gcloud transfer agent-pools list \
  [--limit=LIMIT] \
  [--names=[NAMES,...]] \
  [--page-size=PAGE_SIZE]

The following flags can be specified with this command:

  • --limit defines the maximum number of agent pools to return.

  • --names specifies the names of the agent pools to list. Separate multiple names with commas; for example, --names=foo,bar. If not specified, all agent pools in your current project are listed.

  • --page-size defines the batching behavior for this command. By default, the list command batches responses into 256 items, automatically fetching the next batch until all items are listed or limit is reached.

The command returns a list of matching agent pools and their configuration details. The following example shows a sample response.

name: projects/my-project-123/agentPools/my-first-pool
state: CREATED
---
bandwidthLimit:
  limitMbps: '120'
displayName: Source A to destination Z.
name: projects/my-project-123/agentPools/a2z-pool
state: CREATED