Setting up Transfer for on-premises

The first time you create a Transfer service for on-premises data job, you'll need to enable required APIs and ensure correct permissions are granted.

If you receive errors while performing first-time setup, confirm that the user you logged in with has permissions required to perform the set up steps. In many cases, these permissions are not available to all users, and you may need to contact a project administrator for assistance.

To do the first-time setup:

  1. Enable Pub/Sub API:

    1. Go to the API Library page in the Google Cloud Console.

    Go to the API Library Page

    1. In the Search box, enter Pub/Sub API.

    2. Select Pub/Sub API

      The Pub/Sub API page is displayed.

    3. Click Enable.

      The Pub/Sub API overview is displayed.

  2. Use the Google Cloud project administrator, a user with resourcemanager.projects.setIamPolicy privileges, to grant Identity and Access Management permissions or roles to:
    • Transfer for on-premises administrator (admin) accounts — superuser accounts supporting colleagues that perform transfers. Admins manage Transfer for on-premises agents and set bandwidth usage limits.
    • Transfer for on-premises user accounts — accounts used to create and execute transfers. These accounts typically don't have access to delete transfer jobs.
    • The Transfer for on-premises service account — the service account used by Transfer for on-premises to perform transfers.
    • Transfer for on-premises agent identity - the identity used to run the Transfer for on-premises agent. This can be either a service account or a user account that sets up the on-premises agents.

    The Google Cloud project administrator account is necessary only to set up transfer users and grant the Transfer for on-premises service account required permissions. It isn't required to start transfer jobs.

    For more information on granting Identity and Access Management permissions, see Granting, changing, and revoking access to resources.

    1. To set up a Transfer for on-premises admin account, assign the following IAM permissions and roles to the account:
      Role / Permission What it does Notes
      resourcemanager.projects.getIamPolicy This permission is used to confirm that the Transfer for on-premises service account has the required permissions for a transfer.
      roles/storagetransfer.admin Enables administrative actions in the transfer project, such as project set up and agent monitoring. For a detailed listing of permissions granted, see Predefined roles.
    2. To set up a Transfer for on-premises user account, assign the following permissions and roles to the account:
      Role / Permission What it does Notes
      resourcemanager.projects.getIamPolicy Used to confirm that the Transfer for on-premises service account has the required Pub/Sub permissions for a transfer.
      roles/storagetransfer.user Enables the user to create, get, update, and list transfers. For a detailed listing of permissions granted, see Predefined roles.
      roles/storage.objectAdmin Enables the user to create, update, and delete Cloud Storage objects as part of a transfer. Must be granted for every Cloud Storage bucket this account will use in transfers.

      For a detailed listing of permissions granted, see Predefined roles.
    3. To allow the Transfer for on-premises service account to access resources needed to complete transfers, assign the following roles, or equivalent permissions, to the Transfer for on-premises service account cloud-ingest-dcp@cloud-ingest-prod.iam.gserviceaccount.com:
      Role / Permission What it does Notes
      roles/storage.objectCreator Enables Transfer for on-premises to create transfer logs in the destination Cloud Storage bucket. Grant to all Cloud Storage buckets used in a transfer. If appropriate for your situation, you can grant the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions these roles grant, see Predefined roles.
      roles/storage.objectViewer Enables Transfer for on-premises to determine if a file has already been uploaded to Cloud Storage.
      roles/pubsub.editor Enables Transfer for on-premises to automatically create and modify Pub/Sub topics to communicate from Google Cloud to Transfer for on-premises agents. Apply the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions this role grants, see Roles.
      storage.buckets.get This permission enables reading Cloud Storage bucket metadata.
    4. To set up a Transfer for on-premises agent service account or user account that will run the Transfer for on-premises agents, assign the following permissions and roles:
      Role / Permission What it does Notes
      roles/storage.objectAdmin Enables Transfer for on-premises agents to create, update, and delete Cloud Storage objects as part of a transfer. Grant to all Cloud Storage buckets used in a transfer. If appropriate for your situation, you can grant the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions this role grants, see Roles.
      roles/pubsub.publisher Enables Transfer for on-premises agents to share information with Google Cloud using Pub/Sub topics. For a detailed listing of the permissions this role grants, see Roles.
      roles/pubsub.subscriber Enables Google Cloud to share information with Transfer for on-premises agents using Pub/Sub topics. For a detailed listing of the permissions this role grants, see Roles.
      pubsub.subscriptions.create This permission enables Transfer for on-premises agents to create Pub/Sub subscriptions to the Pub/Sub topic used to communicate between Google Cloud and Transfer for on-premises agents.
      pubsub.subscriptions.delete This permission enables Transfer for on-premises agents that exit gracefully to clean up any Pub/Sub subscriptions they create.
  3. Install and run on-premises agents on each of your machines.

What's next?

Create a transfer job.