Using the Transfer Appliance Cloud Setup Application

This topic describes configuring Google Cloud permissions and Cloud Storage using the Transfer Appliance Cloud Setup Application.

The Transfer Appliance Cloud Setup Application prompts you for information, such as your transfer session ID, destination Cloud Storage bucket and Cloud Key Management Service (Cloud KMS) preferences. Using the information you provide, the Transfer Appliance Cloud Setup Application configures your Google Cloud permissions, preferred Cloud Storage bucket, and Cloud KMS key for your transfer.

Before you begin

Ensure that you have the following:

  • The name of the project used for the transfer.

  • The session ID from the Transfer Appliance Team email titled Google Transfer Appliance Access Credentials. You'll complete the form in this email, which the Transfer Appliance Team team uses to configure your transfer.

  • The correct IAM roles.

    If you are the project owner, roles/owner is sufficient.

    If you don't have roles/owner and are creating a Cloud Storage bucket and Cloud KMS key through the Transfer Appliance Cloud Setup Application, then you should have the roles/storage.admin and roles/cloudkms.admin roles for the Google Cloud project.

    If you don't have roles/owner and are using an existing Cloud Storage bucket and Cloud KMS key, then you should have the roles/storage.admin on the Cloud Storage bucket and roles/cloudkms.admin on the Cloud KMS key you are using for the transfer.

    To view IAM roles that your members have for a project and its resources, do the following:

    1. In the Cloud Console, go to the IAM page.

      Go to the IAM page

    2. The page displays all the members in your project that have IAM roles on your project.

Downloading the Transfer Appliance Cloud Setup Application

  1. Open the Cloud Console Dashboard.

    Open the Cloud Console Dashboard

  2. Verify that the name of the project used for the transfer is displayed in the project selector. The project selector tells you what project you are currently working in.

    Selecting a Google Cloud project from the project selector

    If you don't see the name of the project you are using for the transfer, click the project selector, then select the correct project.

  3. Click Activate Cloud Shell.

    Starting devshell from the menu bar.

  4. In Cloud Shell, use the wget command to download the Transfer Appliance Cloud Setup Application:

    wget https://storage.googleapis.com/transferappliance/cloudsetup/ta_cloudsetup_x86_64-linux
    

Running the Transfer Appliance Cloud Setup Application

  1. In Cloud Shell, run the following command to start the Transfer Appliance Cloud Setup Application:

    chmod 0777 ta_cloudsetup_x86_64-linux && ./ta_cloudsetup_x86_64-linux
    
  2. The Transfer Appliance Cloud Setup Application prompts you for the following items:

    • The Session ID
    • Whether you want to use an existing bucket:
      • If you select Yes, a list of buckets are displayed for you to choose from.
      • If you select No, you are prompted for a bucket name. A list of bucket locations are displayed for you to choose from.
    • [Optional] An object prefix. Without an object prefix, objects are transferred to Cloud Storage with the source's path, not including the root path, before the file name on the filesystem. For example, if you have the following files:

      • /source_root_path/file1.txt
      • /source_root_path/dirA/file2.txt
      • /source_root_path/dirA/dirB/file3.txt
      Then the object names in Cloud Storage are:
      • file1.txt
      • dirA/file2.txt
      • dirA/dirB/file3.txt
      The object prefix is added to the object's destination name in Cloud Storage, after the / character of the destination bucket name and before any path names that the object was transferred from, not including the source's root path. This can help you distinguish between objects transferred from other transfer jobs.

      The following table demonstrates several examples of object prefixes and their resulting object names in Cloud Storage, if the source object's path is /source_root_path/sub_folder_name/object_name:
      Prefix Destination object name
      None /destination_bucket/sub_folder_name/object_name
      prefix/ /destination_bucket/prefix/sub_folder_name/object_name

    • Whether you want to use a Google-managed or customer-managed Cloud KMS key:

      If you select a customer-managed Cloud KMS key, the Transfer Appliance Cloud Setup Application prompts if you prefer to use an existing Cloud KMS key:

      • If you select Yes, you are prompted for the Cloud KMS key ring location, the Cloud KMS key ring, and the Cloud KMS key used for this session.
      • If you select No, you are prompted for the Cloud KMS key ring location. The Transfer Appliance Cloud Setup Application creates the Cloud KMS key in the location you select.

    The following is a sample session:

    Please enter the session ID (The session ID is shared separately by the Transfer
    Appliance support team)
    CUS-1120-TEST
    
    Would you like to use an existing storage bucket where your data will arrive?
    Choose Yes or No:
        Yes
       ▸No
    
    Name your bucket. Pick a globally unique, permanent name. Naming guidelines:
    https://cloud.google.com/storage/docs/naming-buckets
    
    2021-05-transfer-set
    
    Please wait. Fetching the list of available locations, this can take a few
    seconds..
    
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Cloud Storage Location:
      ▸ asia-east1
        asia-east2
        asia-northeast1
        asia-northeast2
    ↓   asia-northeast3
    
    [Optional] Enter an object prefix or press enter to skip:
    
    test
    
    Please select the encryption key type
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Key Type:
        Google Managed Key
      ▸ Customer Managed Key
    
    Would you like to use an existing Cloud KMS key?
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose Yes or No:
        Yes
       ▸No
    
    Please wait. Fetching the list of available KMS locations, this can take a few seconds..
    Use the arrow keys to navigate: ↓ ↑ → ←
    ? Choose key ring location:
      ▸ asia
        asia-east1
        asia-east2
        asia-northeast1
    ↓   asia-northeast2
    

    The Transfer Appliance Cloud Setup Application does the following when it completes:

    • Grants permissions to the Transfer Appliance service accounts used to transfer data to your Cloud Storage bucket.
    • Grants permission to the Transfer Appliance service accounts to access necessary Cloud KMS key data if you chose Customer Managed Key.
    • Displays the following information:

      • The Google Cloud cryptographic key resource name, if you chose to use a customer-managed Cloud KMS encryption key.
      • The Google Cloud Cloud Storage destination bucket name.
      • A Google Cloud Cloud Storage destination bucket prefix, if you supplied one.

      The information displayed is also stored within the home directory on Cloud Shell, named SESSION_ID-output.txt, where SESSION_ID is the session ID for this particular transfer.

      The names of the service accounts granted permission for this particular transfer are stored within the home directory on Cloud Shell, named cloudsetup.log.

  3. In the email sent by the Transfer Appliance Team, titled Google Transfer Appliance Access Credentials, enter the following information:

    • The Google Cloud Platform project ID.
    • Select your choice for Encryption:
      • Google-managed encryption key, if you chose to have Google to manage your encryption key.
      • Customer-managed encryption key, if you chose to manage your own encryption key. Select the desired encryption key from the Select a customer-managed encryption-key drop-down menu.
    • The Google Cloud Cloud Storage destination bucket name used for this transfer.
    • Optional: An Google Cloud Cloud Storage Destination Bucket Prefix

Troubleshooting

Error 400: Service account does not exist

Issue:

Transfer Appliance Cloud Setup Application displays the following message:

Service account SESSION_ID@transfer-appliance-zimbru.iam.gserviceaccount.com
does not exist.

Where SESSION_ID is the session ID provided to Transfer Appliance Cloud Setup Application.

Solution:

Verify the session ID for your transfer. The session ID is unique to each transfer session and shared by the Transfer Appliance Team. If you haven't received a session ID, contact data-support@google.com.

Error: Listing KMS locations

Issue:

Transfer Appliance Cloud Setup Application displays the following message:

Error: listing kms locations

Solution:

Do the following within Cloud Shell:

  1. Re-authenticate by running gcloud auth login.

  2. Retry Transfer Appliance Cloud Setup Application.

If the error persists, contact the Transfer Appliance Team at data-support@google.com.

Error: Creating Cloud KMS key constraint error

Issue:

Transfer Appliance Cloud Setup Application displays a message similar to the following:

Error: creating cloud kms key violates constraint error: code = FailedPrecondition
desc= europe-west6 violates constraint 'constraints/gcp.resourceLocations' on
the resource 'projects/test/locations/europe-west6'

Solution:

Your Google Cloud project may have organization policies that disallow creating Cloud Key Management Service keys in certain locations. The following are possible solutions:

  • Choose a different location to create the Cloud Key Management Service key.
  • Update the organization policy to allow Cloud Key Management Service key creation in the location you desire.

For more information see Restricting Resource Locations.