Using the Transfer Appliance Cloud Setup Application

This topic describes configuring Google Cloud permissions and Cloud Storage using the Transfer Appliance Cloud Setup Application.

The Transfer Appliance Cloud Setup Application prompts you for information, such as your transfer session ID, destination Cloud Storage bucket and Cloud Key Management Service (Cloud KMS) preferences. Using the information you provide, the Transfer Appliance Cloud Setup Application configures your Google Cloud permissions, preferred Cloud Storage bucket, and Cloud KMS key for your transfer.

Before you begin

Ensure that you have the following:

  • The name of the project used for the transfer.

  • The session ID from the Transfer Appliance Team email titled Google Transfer Appliance Access Credentials. You'll complete the form in this email, which the Transfer Appliance Team team uses to configure your transfer.

  • The correct IAM roles.

    If you are the project owner, roles/owner is sufficient.

    If you don't have roles/owner and are creating a Cloud Storage bucket and Cloud KMS key through the Transfer Appliance Cloud Setup Application, then you should have the roles/storage.admin and roles/cloudkms.admin roles for the Google Cloud project.

    If you don't have roles/owner and are using an existing Cloud Storage bucket and Cloud KMS key, then you should have the roles/storage.admin on the Cloud Storage bucket and roles/cloudkms.admin on the Cloud KMS key you are using for the transfer.

    To view IAM roles that your principals have for a project and its resources, do the following:

    1. In the Cloud Console, go to the IAM page.

      Go to the IAM page

    2. The page displays all the principals that have IAM roles on your project.

  • The service agent that is tied to the Transfer service for on-premises data service. This service agent is listed in an email titled Google Transfer Appliance Prepare Destination Bucket, and looks similar to the following example:

    project-IDENTIFIER@storage-transfer-service.gserviceaccount.com

    In this example, IDENTIFIER is a generated number specific to this particular project.

    We use the Transfer service for on-premises data service to transfer data from the appliance to your Cloud Storage bucket.

Grant permissions to the service agent

To grant permissions to the service agent, do the following:

Google Cloud Console

  1. In the Google Cloud Console, go to the Cloud Storage Browser page.

    Go to Browser

  2. Click the Bucket overflow menu () associated with the bucket to which you are granting a principal a role.

  3. Choose Edit bucket permissions.

  4. Click the + Add members button.

  5. In the New members field, enter the service agent. It looks similar to the following example:

    project-IDENTIFIER@storage-transfer-service.gserviceaccount.com

    In this example, IDENTIFIER is a generated number specific to this particular project.

  6. From the Select a role drop-down menu, select the Storage Admin role.

    The roles you select appear in the pane with a short description of the permissions they grant.

  7. Click Save.

Command line

Use the gsutil iam ch command:

gsutil iam ch \
serviceAccount:project-IDENTIFIER@storage-transfer-service.gserviceaccount.com:storageAdmin \
gs://BUCKET_NAME

In this example:

  • IDENTIFIER: A generated number specific to this particular project.
  • BUCKET_NAME: The name of the bucket you're using.

Download the Transfer Appliance Cloud Setup Application

To download the Transfer Appliance Cloud Setup Application:

  1. Open the Cloud Console Dashboard.

    Open the Cloud Console Dashboard

  2. Verify that the name of the project used for the transfer is displayed in the project selector. The project selector tells you what project you are currently working in.

    Selecting a Google Cloud project from the project selector

    If you don't see the name of the project you are using for the transfer, click the project selector, then select the correct project.

  3. Click Activate Cloud Shell.

    Starting devshell from the menu bar.

  4. In Cloud Shell, use the wget command to download the Transfer Appliance Cloud Setup Application:

    wget https://storage.googleapis.com/transferappliance/cloudsetup/ta_cloudsetup_x86_64-linux
    

Run the Transfer Appliance Cloud Setup Application

  1. In Cloud Shell, run the following command to start the Transfer Appliance Cloud Setup Application:

    chmod 0777 ta_cloudsetup_x86_64-linux && ./ta_cloudsetup_x86_64-linux
    
  2. The Transfer Appliance Cloud Setup Application prompts you for the following items:

    • The service agent that is tied to the Transfer service for on-premises data service. It looks similar to the following example:

      project-IDENTIFIER@storage-transfer-service.gserviceaccount.com

      In this example, IDENTIFIER is a generated number specific to this particular project.

    • The Session ID, from the Transfer Appliance Team email titled Google Transfer Appliance Access Credentials.

    • Whether you want to use an existing bucket:

      • If you select Yes, a list of buckets are displayed for you to choose from.

      • If you select No, you are prompted for a bucket name. A list of bucket locations are displayed for you to choose from.

    • [Optional] An object prefix. Without an object prefix, objects are transferred to Cloud Storage with the source's path, not including the root path, before the file name on the filesystem. For example, if you have the following files:

      • /source_root_path/file1.txt
      • /source_root_path/dirA/file2.txt
      • /source_root_path/dirA/dirB/file3.txt
      Then the object names in Cloud Storage are:
      • file1.txt
      • dirA/file2.txt
      • dirA/dirB/file3.txt
      The object prefix is added to the object's destination name in Cloud Storage, after the / character of the destination bucket name and before any path names that the object was transferred from, not including the source's root path. This can help you distinguish between objects transferred from other transfer jobs.

      The following table demonstrates several examples of object prefixes and their resulting object names in Cloud Storage, if the source object's path is /source_root_path/sub_folder_name/object_name:
      Prefix Destination object name
      None /destination_bucket/sub_folder_name/object_name
      prefix/ /destination_bucket/prefix/sub_folder_name/object_name

    • Whether you want to use a Google-managed or customer-managed Cloud KMS key:

      If you select a customer-managed Cloud KMS key, the Transfer Appliance Cloud Setup Application prompts if you prefer to use an existing Cloud KMS key:

      • If you select Yes, you are prompted for the Cloud KMS key ring location, the Cloud KMS key ring, and the Cloud KMS key used for this session.
      • If you select No, you are prompted for the Cloud KMS key ring location. The Transfer Appliance Cloud Setup Application creates the Cloud KMS key in the location you select.

    Sample session

    The following is a sample session:

    Please enter the session ID (The session ID is shared separately by the
    Transfer Appliance support team)
    CUS-1120-TEST
    
    [Optional] Please enter the Google-managed service account if it was
    shared by the Transfer Appliance support team:
    project-12345@storage-transfer-service.gserviceaccount.com
    
    Would you like to use an existing storage bucket where your data will
    arrive?
    
    Choose Yes or No:
        Yes
       ▸No
    
    Name your bucket. Pick a globally unique, permanent name. Naming guidelines:
    https://cloud.google.com/storage/docs/naming-buckets
    
    2021-05-transfer-set
    
    Please wait. Fetching the list of available locations, this can take a few
    seconds..
    
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Cloud Storage Location:
      ▸ asia-east1
        asia-east2
        asia-northeast1
        asia-northeast2
    ↓   asia-northeast3
    
    [Optional] Enter an object prefix or press enter to skip:
    
    test
    
    Please select the encryption key type
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Key Type:
        Google-managed encryption key
      ▸ Customer-managed encryption key
    
    Would you like to use an existing Cloud KMS key?
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Yes or No:
        Yes
       ▸No
    
    Please wait. Fetching the list of available KMS locations, this can take a few seconds..
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose key ring location:
      ▸ asia
        asia-east1
        asia-east2
        asia-northeast1
    ↓   asia-northeast2
    
    Cloud Setup is complete.
    
    Next Steps..
    
    Please complete the form received in the email titled Google Transfer
    Appliance Access Credentials sent by the Transfer Appliance Team with the
    following information:
    
    Session ID:
    CUS-1120-TEST
    
    Google Cloud Platform Project ID:
    customer-test
    
    Encryption:
    Customer-managed encryption key
    
    Google Cloud Storage Destination Bucket Name:
    2021-05-tranfsfer-set
    
    Google Cloud Storage Destination Object Bucket Prefix (optional):
    
    Online Enabled:
    No
    

    The Transfer Appliance Cloud Setup Application does the following when it completes:

    • Grants permissions to the Transfer Appliance service accounts used to transfer data to your Cloud Storage bucket.
    • Grants permission to the Transfer Appliance service accounts to access necessary Cloud KMS key data if you chose Customer Managed Key.
    • Displays the following information:

      • The Google Cloud cryptographic key resource name, if you chose to use a customer-managed Cloud KMS encryption key.
      • The Google Cloud Cloud Storage destination bucket name.
      • A Google Cloud Cloud Storage destination bucket prefix, if you supplied one.

      The information displayed is also stored within the home directory on Cloud Shell, named SESSION_ID-output.txt, where SESSION_ID is the session ID for this particular transfer.

      The names of the service accounts granted permission for this particular transfer are stored within the home directory on Cloud Shell, named cloudsetup.log.

  3. You'll receive an email from the Transfer Appliance Team, titled Google Transfer Appliance Prepare Permissions and Storage. Complete the form linked in that email with the following information:

    • The Google Cloud Platform project ID.
    • Select your choice for Encryption:
      • Google-managed encryption key, if you chose to have Google to manage your encryption key.
      • Customer-managed encryption key, if you chose to manage your own encryption key. Select the desired encryption key from the Select a customer-managed encryption-key drop-down menu.
    • The Google Cloud Cloud Storage destination bucket name used for this transfer.
    • Optional: An Google Cloud Cloud Storage Destination Bucket Prefix

Troubleshooting

Error 400: Service account does not exist

Issue:

Transfer Appliance Cloud Setup Application displays the following message:

Service account SESSION_ID@transfer-appliance-zimbru.iam.gserviceaccount.com
does not exist.

Where SESSION_ID is the session ID provided to Transfer Appliance Cloud Setup Application.

Solution:

Verify the session ID for your transfer. The session ID is unique to each transfer session and shared by the Transfer Appliance Team. If you haven't received a session ID, contact data-support@google.com.

Error: Listing KMS locations

Issue:

Transfer Appliance Cloud Setup Application displays the following message:

Error: listing kms locations

Solution:

Do the following within Cloud Shell:

  1. Re-authenticate by running gcloud auth login.

  2. Retry Transfer Appliance Cloud Setup Application.

If the error persists, contact the Transfer Appliance Team at data-support@google.com.

Error: Creating Cloud KMS key constraint error

Issue:

Transfer Appliance Cloud Setup Application displays a message similar to the following:

Error: creating cloud kms key violates constraint error: code = FailedPrecondition
desc= europe-west6 violates constraint 'constraints/gcp.resourceLocations' on
the resource 'projects/test/locations/europe-west6'

Solution:

Your Google Cloud project may have organization policies that disallow creating Cloud Key Management Service keys in certain locations. The following are possible solutions:

  • Choose a different location to create the Cloud Key Management Service key.
  • Update the organization policy to allow Cloud Key Management Service key creation in the location you desire.

For more information see Restricting Resource Locations.