StorageTransferJob


Property Value
Google Cloud Service Name Storage Transfer
Google Cloud Service Documentation /storage-transfer/docs/
Google Cloud REST Resource Name v1.transferJobs
Google Cloud REST Resource Documentation /storage-transfer/docs/reference/rest/v1/transferJobs
Config Connector Resource Short Names gcpstoragetransferjob
gcpstoragetransferjobs
storagetransferjob
Config Connector Service Name storagetransfer.googleapis.com
Config Connector Resource Fully Qualified Name storagetransferjobs.storagetransfer.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember No
Config Connector Default Average Reconcile Interval In Seconds 600

Custom Resource Definition Properties

Annotations

Fields
cnrm.cloud.google.com/project-id

Spec

Schema

description: string
notificationConfig:
  eventTypes:
  - string
  payloadFormat: string
  topicRef:
    external: string
    name: string
    namespace: string
resourceID: string
schedule:
  repeatInterval: string
  scheduleEndDate:
    day: integer
    month: integer
    year: integer
  scheduleStartDate:
    day: integer
    month: integer
    year: integer
  startTimeOfDay:
    hours: integer
    minutes: integer
    nanos: integer
    seconds: integer
status: string
transferSpec:
  awsS3DataSource:
    awsAccessKey:
      accessKeyId:
        value: string
        valueFrom:
          secretKeyRef:
            key: string
            name: string
      secretAccessKey:
        value: string
        valueFrom:
          secretKeyRef:
            key: string
            name: string
    bucketName: string
    path: string
    roleArn: string
  azureBlobStorageDataSource:
    azureCredentials:
      sasToken:
        value: string
        valueFrom:
          secretKeyRef:
            key: string
            name: string
    container: string
    path: string
    storageAccount: string
  gcsDataSink:
    bucketRef:
      external: string
      name: string
      namespace: string
    path: string
  gcsDataSource:
    bucketRef:
      external: string
      name: string
      namespace: string
    path: string
  httpDataSource:
    listUrl: string
  objectConditions:
    excludePrefixes:
    - string
    includePrefixes:
    - string
    lastModifiedBefore: string
    lastModifiedSince: string
    maxTimeElapsedSinceLastModification: string
    minTimeElapsedSinceLastModification: string
  posixDataSink:
    rootDirectory: string
  posixDataSource:
    rootDirectory: string
  sinkAgentPoolName: string
  sourceAgentPoolName: string
  transferOptions:
    deleteObjectsFromSourceAfterTransfer: boolean
    deleteObjectsUniqueInSink: boolean
    overwriteObjectsAlreadyExistingInSink: boolean
    overwriteWhen: string
Fields

description

Required

string

Unique description to identify the Transfer Job.

notificationConfig

Optional

object

Notification configuration.

notificationConfig.eventTypes

Optional

list (string)

Event types for which a notification is desired. If empty, send notifications for all event types. The valid types are "TRANSFER_OPERATION_SUCCESS", "TRANSFER_OPERATION_FAILED", "TRANSFER_OPERATION_ABORTED".

notificationConfig.eventTypes[]

Optional

string

notificationConfig.payloadFormat

Required*

string

The desired format of the notification message payloads. One of "NONE" or "JSON".

notificationConfig.topicRef

Required*

object

The PubSubTopic to which to publish notifications.

notificationConfig.topicRef.external

Optional

string

Allowed value: string of the format `projects/{{project}}/topics/{{value}}`, where {{value}} is the `name` field of a `PubSubTopic` resource.

notificationConfig.topicRef.name

Optional

string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

notificationConfig.topicRef.namespace

Optional

string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

resourceID

Optional

string

Immutable. Optional. The service-generated name of the resource. Used for acquisition only. Leave unset to create a new resource.

schedule

Optional

object

Schedule specification defining when the Transfer Job should be scheduled to start, end and what time to run.

schedule.repeatInterval

Optional

string

Interval between the start of each scheduled transfer. If unspecified, the default value is 24 hours. This value may not be less than 1 hour. A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".

schedule.scheduleEndDate

Optional

object

The last day the recurring transfer will be run. If schedule_end_date is the same as schedule_start_date, the transfer will be executed only once.

schedule.scheduleEndDate.day

Required*

integer

Day of month. Must be from 1 to 31 and valid for the year and month.

schedule.scheduleEndDate.month

Required*

integer

Month of year. Must be from 1 to 12.

schedule.scheduleEndDate.year

Required*

integer

Year of date. Must be from 1 to 9999.

schedule.scheduleStartDate

Required*

object

The first day the recurring transfer is scheduled to run. If schedule_start_date is in the past, the transfer will run for the first time on the following day.

schedule.scheduleStartDate.day

Required*

integer

Day of month. Must be from 1 to 31 and valid for the year and month.

schedule.scheduleStartDate.month

Required*

integer

Month of year. Must be from 1 to 12.

schedule.scheduleStartDate.year

Required*

integer

Year of date. Must be from 1 to 9999.

schedule.startTimeOfDay

Optional

object

The time in UTC at which the transfer will be scheduled to start in a day. Transfers may start later than this time. If not specified, recurring and one-time transfers that are scheduled to run today will run immediately; recurring transfers that are scheduled to run on a future date will start at approximately midnight UTC on that date. Note that when configuring a transfer with the Cloud Platform Console, the transfer's start time in a day is specified in your local timezone.

schedule.startTimeOfDay.hours

Required*

integer

Hours of day in 24 hour format. Should be from 0 to 23.

schedule.startTimeOfDay.minutes

Required*

integer

Minutes of hour of day. Must be from 0 to 59.

schedule.startTimeOfDay.nanos

Required*

integer

Fractions of seconds in nanoseconds. Must be from 0 to 999,999,999.

schedule.startTimeOfDay.seconds

Required*

integer

Seconds of minutes of the time. Must normally be from 0 to 59.

status

Optional

string

Status of the job. Default: ENABLED. NOTE: The effect of the new job status takes place during a subsequent job run. For example, if you change the job status from ENABLED to DISABLED, and an operation spawned by the transfer is running, the status change would not affect the current operation.

transferSpec

Required

object

Transfer specification.

transferSpec.awsS3DataSource

Optional

object

An AWS S3 data source.

transferSpec.awsS3DataSource.awsAccessKey

Optional

object

AWS credentials block.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId

Required*

object

AWS Key ID.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId.value

Optional

string

Value of the field. Cannot be used if 'valueFrom' is specified.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId.valueFrom

Optional

object

Source for the field's value. Cannot be used if 'value' is specified.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId.valueFrom.secretKeyRef

Optional

object

Reference to a value with the given key in the given Secret in the resource's namespace.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId.valueFrom.secretKeyRef.key

Required*

string

Key that identifies the value to be extracted.

transferSpec.awsS3DataSource.awsAccessKey.accessKeyId.valueFrom.secretKeyRef.name

Required*

string

Name of the Secret to extract a value from.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey

Required*

object

AWS Secret Access Key.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey.value

Optional

string

Value of the field. Cannot be used if 'valueFrom' is specified.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey.valueFrom

Optional

object

Source for the field's value. Cannot be used if 'value' is specified.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey.valueFrom.secretKeyRef

Optional

object

Reference to a value with the given key in the given Secret in the resource's namespace.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey.valueFrom.secretKeyRef.key

Required*

string

Key that identifies the value to be extracted.

transferSpec.awsS3DataSource.awsAccessKey.secretAccessKey.valueFrom.secretKeyRef.name

Required*

string

Name of the Secret to extract a value from.

transferSpec.awsS3DataSource.bucketName

Required*

string

S3 Bucket name.

transferSpec.awsS3DataSource.path

Optional

string

S3 Bucket path in bucket to transfer.

transferSpec.awsS3DataSource.roleArn

Optional

string

The Amazon Resource Name (ARN) of the role to support temporary credentials via 'AssumeRoleWithWebIdentity'. For more information about ARNs, see [IAM ARNs](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-arns). When a role ARN is provided, Transfer Service fetches temporary credentials for the session using a 'AssumeRoleWithWebIdentity' call for the provided role using the [GoogleServiceAccount][] for this project.

transferSpec.azureBlobStorageDataSource

Optional

object

An Azure Blob Storage data source.

transferSpec.azureBlobStorageDataSource.azureCredentials

Required*

object

Credentials used to authenticate API requests to Azure.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken

Required*

object

Azure shared access signature.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken.value

Optional

string

Value of the field. Cannot be used if 'valueFrom' is specified.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken.valueFrom

Optional

object

Source for the field's value. Cannot be used if 'value' is specified.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken.valueFrom.secretKeyRef

Optional

object

Reference to a value with the given key in the given Secret in the resource's namespace.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken.valueFrom.secretKeyRef.key

Required*

string

Key that identifies the value to be extracted.

transferSpec.azureBlobStorageDataSource.azureCredentials.sasToken.valueFrom.secretKeyRef.name

Required*

string

Name of the Secret to extract a value from.

transferSpec.azureBlobStorageDataSource.container

Required*

string

The container to transfer from the Azure Storage account.

transferSpec.azureBlobStorageDataSource.path

Optional

string

Root path to transfer objects. Must be an empty string or full path name that ends with a '/'. This field is treated as an object prefix. As such, it should generally not begin with a '/'.

transferSpec.azureBlobStorageDataSource.storageAccount

Required*

string

The name of the Azure Storage account.

transferSpec.gcsDataSink

Optional

object

A Google Cloud Storage data sink.

transferSpec.gcsDataSink.bucketRef

Required*

object

transferSpec.gcsDataSink.bucketRef.external

Optional

string

Allowed value: The `name` field of a `StorageBucket` resource.

transferSpec.gcsDataSink.bucketRef.name

Optional

string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

transferSpec.gcsDataSink.bucketRef.namespace

Optional

string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

transferSpec.gcsDataSink.path

Optional

string

Google Cloud Storage path in bucket to transfer.

transferSpec.gcsDataSource

Optional

object

A Google Cloud Storage data source.

transferSpec.gcsDataSource.bucketRef

Required*

object

transferSpec.gcsDataSource.bucketRef.external

Optional

string

Allowed value: The `name` field of a `StorageBucket` resource.

transferSpec.gcsDataSource.bucketRef.name

Optional

string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

transferSpec.gcsDataSource.bucketRef.namespace

Optional

string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

transferSpec.gcsDataSource.path

Optional

string

Google Cloud Storage path in bucket to transfer.

transferSpec.httpDataSource

Optional

object

A HTTP URL data source.

transferSpec.httpDataSource.listUrl

Required*

string

The URL that points to the file that stores the object list entries. This file must allow public access. Currently, only URLs with HTTP and HTTPS schemes are supported.

transferSpec.objectConditions

Optional

object

Only objects that satisfy these object conditions are included in the set of data source and data sink objects. Object conditions based on objects' last_modification_time do not exclude objects in a data sink.

transferSpec.objectConditions.excludePrefixes

Optional

list (string)

exclude_prefixes must follow the requirements described for include_prefixes.

transferSpec.objectConditions.excludePrefixes[]

Optional

string

transferSpec.objectConditions.includePrefixes

Optional

list (string)

If include_refixes is specified, objects that satisfy the object conditions must have names that start with one of the include_prefixes and that do not start with any of the exclude_prefixes. If include_prefixes is not specified, all objects except those that have names starting with one of the exclude_prefixes must satisfy the object conditions.

transferSpec.objectConditions.includePrefixes[]

Optional

string

transferSpec.objectConditions.lastModifiedBefore

Optional

string

If specified, only objects with a "last modification time" before this timestamp and objects that don't have a "last modification time" are transferred. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

transferSpec.objectConditions.lastModifiedSince

Optional

string

If specified, only objects with a "last modification time" on or after this timestamp and objects that don't have a "last modification time" are transferred. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

transferSpec.objectConditions.maxTimeElapsedSinceLastModification

Optional

string

A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".

transferSpec.objectConditions.minTimeElapsedSinceLastModification

Optional

string

A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s".

transferSpec.posixDataSink

Optional

object

A POSIX filesystem data sink.

transferSpec.posixDataSink.rootDirectory

Required*

string

Root directory path to the filesystem.

transferSpec.posixDataSource

Optional

object

A POSIX filesystem data source.

transferSpec.posixDataSource.rootDirectory

Required*

string

Root directory path to the filesystem.

transferSpec.sinkAgentPoolName

Optional

string

Immutable. Specifies the agent pool name associated with the posix data source. When unspecified, the default name is used.

transferSpec.sourceAgentPoolName

Optional

string

Immutable. Specifies the agent pool name associated with the posix data source. When unspecified, the default name is used.

transferSpec.transferOptions

Optional

object

Characteristics of how to treat files from datasource and sink during job. If the option delete_objects_unique_in_sink is true, object conditions based on objects' last_modification_time are ignored and do not exclude objects in a data source or a data sink.

transferSpec.transferOptions.deleteObjectsFromSourceAfterTransfer

Optional

boolean

Whether objects should be deleted from the source after they are transferred to the sink. Note that this option and delete_objects_unique_in_sink are mutually exclusive.

transferSpec.transferOptions.deleteObjectsUniqueInSink

Optional

boolean

Whether objects that exist only in the sink should be deleted. Note that this option and delete_objects_from_source_after_transfer are mutually exclusive.

transferSpec.transferOptions.overwriteObjectsAlreadyExistingInSink

Optional

boolean

Whether overwriting objects that already exist in the sink is allowed.

transferSpec.transferOptions.overwriteWhen

Optional

string

When to overwrite objects that already exist in the sink. If not set, overwrite behavior is determined by overwriteObjectsAlreadyExistingInSink.

* Field is required when parent field is specified

Status

Schema

conditions:
- lastTransitionTime: string
  message: string
  reason: string
  status: string
  type: string
creationTime: string
deletionTime: string
lastModificationTime: string
name: string
observedGeneration: integer
Fields
conditions

list (object)

Conditions represent the latest available observation of the resource's current state.

conditions[]

object

conditions[].lastTransitionTime

string

Last time the condition transitioned from one status to another.

conditions[].message

string

Human-readable message indicating details about last transition.

conditions[].reason

string

Unique, one-word, CamelCase reason for the condition's last transition.

conditions[].status

string

Status is the status of the condition. Can be True, False, Unknown.

conditions[].type

string

Type is the type of the condition.

creationTime

string

When the Transfer Job was created.

deletionTime

string

When the Transfer Job was deleted.

lastModificationTime

string

When the Transfer Job was last modified.

name

string

The name of the Transfer Job.

observedGeneration

integer

ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.

Sample YAML(s)

Typical Use Case

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: storagetransfer.cnrm.cloud.google.com/v1beta1
kind: StorageTransferJob
metadata:
  name: storagetransferjob-sample
spec:
  description: "Sample storage transfer job"
  schedule:
    startTimeOfDay:
      seconds: 0
      hours: 0
      minutes: 0
      nanos: 0
    scheduleEndDate:
      day: 31
      month: 12
      year: 9999
    scheduleStartDate:
      day: 28
      month: 1
      year: 2020
  status: ENABLED
  transferSpec:
    gcsDataSink:
      bucketRef:
        name: ${PROJECT_ID?}-storagetransferjob-dep1
    gcsDataSource:
      bucketRef:
        name: ${PROJECT_ID?}-storagetransferjob-dep2
    objectConditions:
      maxTimeElapsedSinceLastModification: 5s
      minTimeElapsedSinceLastModification: 2s
    transferOptions:
      deleteObjectsUniqueInSink: false
      overwriteObjectsAlreadyExistingInSink: true
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: storagetransferjob-dep1
spec:
  # replace ${PROJECT_NUMBER?} with your project number
  member: serviceAccount:project-${PROJECT_NUMBER?}@storage-transfer-service.iam.gserviceaccount.com
  role: roles/storage.admin
  resourceRef:
    kind: StorageBucket
    name: ${PROJECT_ID?}-storagetransferjob-dep1
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: storagetransferjob-dep2
spec:
  # replace ${PROJECT_NUMBER?} with your project number
  member: serviceAccount:project-${PROJECT_NUMBER?}@storage-transfer-service.iam.gserviceaccount.com
  role: roles/storage.admin
  resourceRef:
    kind: StorageBucket
    name: ${PROJECT_ID?}-storagetransferjob-dep2
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
  name: ${PROJECT_ID?}-storagetransferjob-dep1
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
  name: ${PROJECT_ID?}-storagetransferjob-dep2