Managing agent pools

Agents are pieces of software used by Transfer service for on-premises data to transfer your data to Cloud Storage. An agent pool is a collection of agents that use the same configuration, with uniform access and visibility to your source and destination.

This guide describes how to use Transfer service for on-premises data agent pools.

Before you begin

You must have certain permissions on your project to work with agent pools. Refer to Access control with IAM for details on permissions and roles.

You can view your current permissions.

Create an agent pool

To create an agent pool:

Cloud Console

  1. In the Cloud Console, go to the Transfer service for on-premises data page.

    Go to Transfer service for on-premises data

  2. Click Connection settings.

    The Agent pools page is displayed, listing your existing agent pools. All projects include a default pool named transfer_service_default.

  3. Click Create another pool.

  4. Name your pool, and optionally describe it.

  5. You may choose to set a bandwidth limit that will apply to the pool as a whole. The specified bandwidth in MB/s will be split amongst all of the agents in the pool.

  6. Click Create.

REST API

Use projects.agentPools.create:

POST https://storagetransfer.googleapis.com/v1/projects/{PROJECT_ID}/agentPools/{agent_pool_id=AGENT_POOL_ID}:create

Where:

  • PROJECT_ID: The project ID that you're creating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are creating.

If an agent pool is stuck in the Creating state for more than 30 minutes, we recommend deleting the agent pool and creating it again.

Revoking required Transfer for on-premises permissions from a project while an agent pool is in the Creating state leads to incorrect service behavior.

Assign agents to a pool

New agent pools are empty and must have agents assigned to them. An agent can only be assigned to an agent pool at the time of the agent's creation, using the --agent-pool field.

If --agent-pool is not specified, the agent is assigned to the transfer_service_default pool.

To create an agent and assign it to a pool:

  1. From the Agent pools page, select the pool to assign the agent(s) to.

  2. Click Install agent. The Agent installation guide appears.

  3. Follow the instructions to create the agent in this agent pool.

For additional options, see Advanced agent setup.

To manage agents, see Administering Transfer for on-premises agents.

Assign an agent pool to a job

Once your agent pool has been created, it can be assigned to a job during job creation or update.

Update an agent pool

You can update an agent pool's display name and bandwidth limit.

To update an agent pool:

Cloud Console

In the Cloud Console, go to the Agent pools page.

Go to Agent pools

You can edit the following items:

  • The agent pool's display name: Click Edit next to the current Agent pool name.

  • The agent pool's bandwidth limit: Click Set bandwidth limit, and enter a new bandwidth limit in the Set bandwidth limit dialog. Click Set Limit to apply the new bandwidth limit. The bandwidth is shared across all agents in the pool.

REST API

To update an agent pool, use projects.agentPools.patch with a field mask of the fields to update. The following agent pool fields can be updated:

For example, to update the displayName to my-transfer, you would provide the field mask "displayName,my-transfer".

The following is an example patch request to update the display name:

PATCH https://storagetransfer.googleapis.com/v1/projects/{PROJECT_ID}/agentPools/{AGENT_POOL_ID}:"displayName,NEW_NAME"

Where:

  • PROJECT_ID: The project ID that you're updating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are updating.
  • NEW_NAME: The new display name for this agent pool.

The following is an example patch request to update the bandwidth limit:

PATCH https://storagetransfer.googleapis.com/v1/projects/{PROJECT_ID}/agentPools/{AGENT_POOL_ID}:"bandwidthLimit,NEW_LIMIT"

Where:

  • PROJECT_ID: The project ID that you're updating the agent pool in.
  • AGENT_POOL_ID: The agent pool ID that you are updating.
  • NEW_LIMIT: The new bandwidth limit for this agent pool.

Delete an agent pool

You can delete an agent pool that has no active agents and no active job runs.

To delete an agent pool:

Cloud Console

  1. In the Cloud Console, go to the Agent pools page.

    Go to Agent pools

  2. Click Delete. Read the pop-up, then click Delete to confirm.

    If the delete button isn't active, you'll need to stop all agents and jobs that are associated with this pool.

REST API

Use projects.agentPools.delete:

DELETE https://storagetransfer.googleapis.com/v1/{name=PROJECT_ID/*/agentPools/}

Where:

  • PROJECT_ID: The project ID that you're deleting the agent pool from.

Get an agent pool

To get an agent pool:

Cloud Console

In the Cloud Console, go to the Agent pools page.

Go to Agent pools

The page displays a list of all agent pools associated with your project, and the following information for each agent pool:

  • Agent pool name
  • Connection status
  • Number of connected agents
  • Bandwidth limit, if set
  • Number of associated transfer jobs
  • The agent pool's display name

To view a specific agent pool, click the agent pool's Name.

The following actions are available from an agent pool's information page:

  • Install agent: displays instructions for installing Transfer for on-premises agents.
  • Stop agent: Select an agent in the table, then click Stop Agent.
  • List transfer jobs: Click Transfer Jobs to display the transfer jobs agents in this pool are connected to.

REST API

Use projects.agentPools.get:

GET https://storagetransfer.googleapis.com/v1/{name=PROJECT_ID/*/agentPools/}

Where:

  • PROJECT_ID: The project ID that you're getting an agent pool of.

List agent pools

To list your agent pools:

Cloud Console

In the Cloud Console, go to the Agent pools page.

Go to Agent pools

All agent pools are displayed.

REST API

Use projects.agentPools.list:

GET https://storagetransfer.googleapis.com/v1/projects/{project_id=PROJECT_ID}/agentPools

Where:

  • PROJECT_ID: The project ID that you're getting an agent pool of.