Storage Transfer API Connector Overview

The Workflows connector defines the built-in functions that can be used to access other Google Cloud products within a workflow.

This page provides an overview of the individual connector. There is no need to import or load connector libraries in a workflow—connectors work out of the box when used in a call step.

Storage Transfer API

Transfers data from external data sources to a Google Cloud Storage bucket or between Google Cloud Storage buckets. To learn more, see the Storage Transfer API documentation.

Storage Transfer connector sample

YAML

# This workflow demonstrates how to use the Cloud Storage Transfer Service connector:
# Create source and sink Cloud Storage buckets and then create a transfer job
# that transfers data from the source bucket to the sink bucket
# Run the transfer job and then set the status of the job to "DELETED"
# Expected output: "SUCCESS"
- init:
    assign:
      - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
      - transfer_job_name: "transferJobs/example-transfer-job"
      - src_bucket_name: "SOURCE_BUCKET_NAME"  # replace SOURCE_BUCKET_NAME placeholder
      - sink_bucket_name: "SINK_BUCKET_NAME"  # replace SINK_BUCKET_NAME placeholder
- create_source_bucket:
    call: googleapis.storage.v1.buckets.insert
    args:
      project: ${project_id}
      body:
        name: ${src_bucket_name}
- create_sink_bucket:
    call: googleapis.storage.v1.buckets.insert
    args:
      project: ${project_id}
      body:
        name: ${sink_bucket_name}
- create_transfer_job:
    call: googleapis.storagetransfer.v1.transferJobs.create
    args:
      body:
        name: ${transfer_job_name}
        description: "A job that transfers data from source bucket to sink bucket"
        projectId: ${project_id}
        transferSpec:
          gcsDataSink:
            bucketName: ${sink_bucket_name}
            path: ""
          gcsDataSource:
            bucketName: ${src_bucket_name}
            path: ""
        status: "DISABLED"
- enable_transfer_job:
    call: googleapis.storagetransfer.v1.transferJobs.patch
    args:
      jobName: ${transfer_job_name}
      body:
        projectId: ${project_id}
        transferJob:
          status: "ENABLED"
- run_transfer_job:
    call: googleapis.storagetransfer.v1.transferJobs.run
    args:
      jobName: ${transfer_job_name}
      body:
        projectId: ${project_id}
    result: r
- delete_transfer_job:
    # Transfer jobs become eligible for garbage collection 30 days
    # after their status is set to DELETED
    call: googleapis.storagetransfer.v1.transferJobs.patch
    args:
      jobName: ${transfer_job_name}
      body:
        projectId: ${project_id}
        transferJob:
          status: "DELETED"
- the_end:
    return: "SUCCESS"

JSON

[
  {
    "init": {
      "assign": [
        {
          "project_id": "${sys.get_env(\"GOOGLE_CLOUD_PROJECT_ID\")}"
        },
        {
          "transfer_job_name": "transferJobs/example-transfer-job"
        },
        {
          "src_bucket_name": "SOURCE_BUCKET_NAME"
        },
        {
          "sink_bucket_name": "SINK_BUCKET_NAME"
        }
      ]
    }
  },
  {
    "create_source_bucket": {
      "call": "googleapis.storage.v1.buckets.insert",
      "args": {
        "project": "${project_id}",
        "body": {
          "name": "${src_bucket_name}"
        }
      }
    }
  },
  {
    "create_sink_bucket": {
      "call": "googleapis.storage.v1.buckets.insert",
      "args": {
        "project": "${project_id}",
        "body": {
          "name": "${sink_bucket_name}"
        }
      }
    }
  },
  {
    "create_transfer_job": {
      "call": "googleapis.storagetransfer.v1.transferJobs.create",
      "args": {
        "body": {
          "name": "${transfer_job_name}",
          "description": "A job that transfers data from source bucket to sink bucket",
          "projectId": "${project_id}",
          "transferSpec": {
            "gcsDataSink": {
              "bucketName": "${sink_bucket_name}",
              "path": ""
            },
            "gcsDataSource": {
              "bucketName": "${src_bucket_name}",
              "path": ""
            }
          },
          "status": "DISABLED"
        }
      }
    }
  },
  {
    "enable_transfer_job": {
      "call": "googleapis.storagetransfer.v1.transferJobs.patch",
      "args": {
        "jobName": "${transfer_job_name}",
        "body": {
          "projectId": "${project_id}",
          "transferJob": {
            "status": "ENABLED"
          }
        }
      }
    }
  },
  {
    "run_transfer_job": {
      "call": "googleapis.storagetransfer.v1.transferJobs.run",
      "args": {
        "jobName": "${transfer_job_name}",
        "body": {
          "projectId": "${project_id}"
        }
      },
      "result": "r"
    }
  },
  {
    "delete_transfer_job": {
      "call": "googleapis.storagetransfer.v1.transferJobs.patch",
      "args": {
        "jobName": "${transfer_job_name}",
        "body": {
          "projectId": "${project_id}",
          "transferJob": {
            "status": "DELETED"
          }
        }
      }
    }
  },
  {
    "the_end": {
      "return": "SUCCESS"
    }
  }
]

Module: googleapis.storagetransfer.v1.googleServiceAccounts

Functions

get Returns the Google service account that is used by Storage Transfer Service to access buckets in the project where transfers run or in other projects. Each Google service account is associated with one Google Cloud Platform Console project. Users should add this service account to the Google Cloud Storage bucket ACLs to grant access to Storage Transfer Service. This service account is created and owned by Storage Transfer Service and can only be used by Storage Transfer Service.

Functions
`get`	Returns the Google service account that is used by Storage Transfer Service to access buckets in the project where transfers run or in other projects. Each Google service account is associated with one Google Cloud Platform Console project. Users should add this service account to the Google Cloud Storage bucket ACLs to grant access to Storage Transfer Service. This service account is created and owned by Storage Transfer Service and can only be used by Storage Transfer Service.

Module: googleapis.storagetransfer.v1.transferJobs

Functions
`create`	Creates a transfer job that runs periodically.
`get`	Gets a transfer job.
`list`	Lists transfer jobs.
`patch`	Updates a transfer job. Updating a job's transfer spec does not affect transfer operations that are running already. Note: The job's status field can be modified using this RPC (for example, to set a job's status to DELETED, DISABLED, or ENABLED).
`run`	Attempts to start a new TransferOperation for the current TransferJob. A TransferJob has a maximum of one active TransferOperation. If this method is called while a TransferOperation is active, an error wil be returned.

Module: googleapis.storagetransfer.v1.transferOperations

Functions
`cancel`	Cancels a transfer. Use the transferOperations.get method to check if the cancellation succeeded or if the operation completed despite the `cancel` request. When you cancel an operation, the currently running transfer is interrupted. For recurring transfer jobs, the next instance of the transfer job will still run. For example, if your job is configured to run every day at 1pm and you cancel Monday's operation at 1:05pm, Monday's transfer will stop. However, a transfer job will still be attempted on Tuesday. This applies only to currently running operations. If an operation is not currently running, `cancel` does nothing. Caution: Canceling a transfer job can leave your data in an unknown state. We recommend that you restore the state at both the destination and the source after the `cancel` request completes so that your data is in a consistent state. When you cancel a job, the next job computes a delta of files and may repair any inconsistent state. For instance, if you run a job every day, and today's job found 10 new files and transferred five files before you canceled the job, tomorrow's transfer operation will compute a new delta with the five files that were not copied today plus any new files discovered tomorrow.
`get`	Gets the latest state of a long-running operation. Clients can use this method to poll the operation result at intervals as recommended by the API service.
`list`	Lists transfer operations. Operations are ordered by their creation time in reverse chronological order.
`pause`	Pauses a transfer operation.
`resume`	Resumes a transfer operation that is paused.