Before you begin

The first time you create a Transfer service for on-premises data job, you'll need to enable required APIs and ensure correct permissions are granted.

If you receive errors while performing first-time setup, confirm that the user you logged in with has permissions required to perform the set up steps. In many cases, these permissions are not available to all users, and you may need to contact a project administrator for assistance.

To do the first-time setup:

  1. Enable the Google Storage Transfer API.

    Enable the API

    1. Confirm the project you are enabling the API for, then click Next.

    2. Click Enable.

  2. Enable the Pub/Sub API.

    Enable the API

    1. Confirm the project you are enabling the API for, then click Next.

    2. Click Enable.

  3. Use the Google Cloud project administrator, a user with resourcemanager.projects.setIamPolicy privileges, to grant Identity and Access Management (IAM permissions or roles to:
    • Transfer for on-premises administrator (admin) accounts — superuser accounts supporting colleagues that perform transfers. Admins manage Transfer for on-premises agents and set bandwidth usage limits.
    • Transfer for on-premises user accounts — accounts used to create and execute transfers. These accounts typically don't have access to delete transfer jobs.
    • The Transfer for on-premises service account — the Google-managed service account used by Transfer for on-premises to perform transfers.
    • Transfer for on-premises agent identity - the identity used to run the Transfer for on-premises agent. This can be either a service account or a user account that sets up the on-premises agents.

    The Google Cloud project administrator account is necessary only to set up transfer users and grant the Transfer for on-premises service account required permissions. It isn't required to start transfer jobs.

    For more information on granting IAM roles, see Granting, changing, and revoking access to resources.

    If you're interested in creating custom roles for Transfer for on-premises, see IAM permissions for Transfer for on-premises and Understanding IAM custom roles.

    1. To set up a Transfer for on-premises admin account, assign the following IAM permissions and roles to the account:
      Role / Permission What it does Notes
      resourcemanager.projects.getIamPolicy This permission is used to confirm that the Transfer for on-premises service account has the required permissions for a transfer.
      Storage Transfer Admin (roles/storagetransfer.admin) Enables administrative actions in the transfer project, such as project set up and agent monitoring. For a detailed listing of permissions granted, see Storage Transfer Service Predefined roles.
    2. To set up a Transfer for on-premises user account, assign the following permissions and roles to the account:
      Role / Permission What it does Notes
      resourcemanager.projects.getIamPolicy Used to confirm that the Transfer for on-premises service account has the required Pub/Sub permissions for a transfer.
      Storage Transfer User (roles/storagetransfer.user) Enables the user to create, get, update, and list transfers. For a detailed listing of permissions granted, see Storage Transfer Service Predefined roles.
      Storage Object Admin (roles/storage.objectAdmin) Enables the user to create, update, and delete Cloud Storage objects as part of a transfer. Must be granted for every Cloud Storage bucket this account uses in transfers.

      For a detailed listing of permissions granted, see Cloud Storage Predefined roles.
    3. Transfer for on-premises uses a Google-managed service account to move your data. The service account's format is typically project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com. To determine your specific PROJECT_NUMBER, use the googleServiceAccounts.get API call.

      To allow the Transfer for on-premises service account to access resources needed to complete transfers, assign the following roles, or equivalent permissions, to the Transfer for on-premises service account:

      Role / Permission What it does Notes
      Storage Object Creator (roles/storage.objectCreator) Enables Transfer for on-premises to create transfer logs in the Cloud Storage bucket connected to this transfer. Grant to all Cloud Storage buckets used in a transfer. If appropriate for your situation, you can grant the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions these roles grant, see Cloud Storage Predefined roles.
      Storage Object Viewer (roles/storage.objectViewer) Enables Transfer for on-premises to determine if a file has already been transferred to or from Cloud Storage.
      Pub/Sub Editor (roles/pubsub.editor) Enables Transfer for on-premises to automatically create and modify Pub/Sub topics to communicate from Google Cloud to Transfer for on-premises agents. Apply the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions this role grants, see Pub/Sub Roles.
      Storage Legacy Bucket Reader (roles/storage.legacyBucketReader) Enables Transfer for on-premises to read Cloud Storage bucket metadata. Grant to each Cloud Storage bucket used in a transfer.
    4. To set up a Transfer for on-premises agent service account or user account running the Transfer for on-premises agents, assign the following permissions and roles:
      Role / Permission What it does Notes
      Storage Object Admin (roles/storage.objectAdmin) Enables Transfer for on-premises agents to create, update, and delete Cloud Storage objects as part of a transfer. Grant to all Cloud Storage buckets used in a transfer. If appropriate for your situation, you can grant the role on a project-level to the project that Transfer for on-premises is running from.

      For a detailed listing of the permissions this role grants, see Cloud Storage Predefined roles.
      Pub/Sub Publisher (roles/pubsub.publisher) Enables Transfer for on-premises agents to share information with Google Cloud using Pub/Sub topics. For a detailed listing of the permissions this role grants, see Pub/Sub Roles.
      Pub/Sub Subscriber (roles/pubsub.subscriber) Enables Google Cloud to share information with Transfer for on-premises agents using Pub/Sub topics. For a detailed listing of the permissions this role grants, see Pub/Sub Roles.
      Pub/Sub Editor (roles/pubsub.editor) Enables Transfer for on-premises agents to create, get, and delete Pub/Sub subscriptions; enables agents to get the Pub/Sub topic used to communicate between Google Cloud and Transfer for on-premises agents. For a detailed listing of the permissions this role grants, see Pub/Sub Roles.
  4. Install Docker Community Edition on a physical or virtual Linux machine by doing the following:

    1. To install Docker Community Edition, run the following commands:

       curl -fsSL https://get.docker.com -o get-docker.sh
      
       sudo sh get-docker.sh
      
       sudo systemctl enable docker
      

      If you encounter installation error, see Troubleshooting.

    2. Enable this machine's agents to authenticate to Google Cloud.

      To enable authentication, run the following commands to save your machine's default gcloud credentials in a Docker volume:

      sudo docker run -ti --name gcloud-config \
          gcr.io/google.com/cloudsdktool/cloud-sdk \
          gcloud auth application-default login
      
  5. Start the agent by running the following command:

    sudo docker run -d --ulimit memlock=64000000 --rm \
    --volumes-from gcloud-config \
    -v /:/transfer_root \
    gcr.io/cloud-ingest/tsop-agent:latest \
    --enable-mount-directory \
    --project-id=PROJECT_ID \
    --hostname=$(hostname)
    

    Replace PROJECT_ID with the ID of your project.

What's next?