Property | Value |
---|---|
Google Cloud Service Name | Storage Transfer |
Google Cloud Service Documentation | /storage-transfer/docs/ |
Google Cloud REST Resource Name | v1.transferJobs |
Google Cloud REST Resource Documentation | /storage-transfer/docs/reference/rest/v1/transferJobs |
Config Connector Resource Short Names | gcpstoragetransferjob gcpstoragetransferjobs storagetransferjob |
Config Connector Service Name | storagetransfer.googleapis.com |
Config Connector Resource Fully Qualified Name | storagetransferjobs.storagetransfer.cnrm.cloud.google.com |
Can Be Referenced by IAMPolicy/IAMPolicyMember | No |
Custom Resource Definition Properties
Annotations
Fields | |
---|---|
cnrm.cloud.google.com/project-id |
Spec
Schema
description: string
resourceID: string
schedule:
scheduleEndDate:
day: integer
month: integer
year: integer
scheduleStartDate:
day: integer
month: integer
year: integer
startTimeOfDay:
hours: integer
minutes: integer
nanos: integer
seconds: integer
status: string
transferSpec:
awsS3DataSource:
awsAccessKey:
accessKeyId:
value: string
valueFrom:
secretKeyRef:
key: string
name: string
secretAccessKey:
value: string
valueFrom:
secretKeyRef:
key: string
name: string
bucketName: string
gcsDataSink:
bucketRef:
external: string
name: string
namespace: string
gcsDataSource:
bucketRef:
external: string
name: string
namespace: string
httpDataSource:
listUrl: string
objectConditions:
excludePrefixes:
- string
includePrefixes:
- string
maxTimeElapsedSinceLastModification: string
minTimeElapsedSinceLastModification: string
transferOptions:
deleteObjectsFromSourceAfterTransfer: boolean
deleteObjectsUniqueInSink: boolean
overwriteObjectsAlreadyExistingInSink: boolean
Fields | |
---|---|
Required |
Unique description to identify the Transfer Job. |
Optional |
Immutable. Optional. The service-generated name of the resource. Used for acquisition only. Leave unset to create a new resource. |
Required |
Schedule specification defining when the Transfer Job should be scheduled to start, end and what time to run. |
Optional |
Immutable. The last day the recurring transfer will be run. If schedule_end_date is the same as schedule_start_date, the transfer will be executed only once. |
Required* |
Immutable. Day of month. Must be from 1 to 31 and valid for the year and month. |
Required* |
Immutable. Month of year. Must be from 1 to 12. |
Required* |
Immutable. Year of date. Must be from 1 to 9999. |
Required |
Immutable. The first day the recurring transfer is scheduled to run. If schedule_start_date is in the past, the transfer will run for the first time on the following day. |
Required |
Immutable. Day of month. Must be from 1 to 31 and valid for the year and month. |
Required |
Immutable. Month of year. Must be from 1 to 12. |
Required |
Immutable. Year of date. Must be from 1 to 9999. |
Optional |
Immutable. The time in UTC at which the transfer will be scheduled to start in a day. Transfers may start later than this time. If not specified, recurring and one-time transfers that are scheduled to run today will run immediately; recurring transfers that are scheduled to run on a future date will start at approximately midnight UTC on that date. Note that when configuring a transfer with the Cloud Platform Console, the transfer's start time in a day is specified in your local timezone. |
Required* |
Immutable. Hours of day in 24 hour format. Should be from 0 to 23. |
Required* |
Immutable. Minutes of hour of day. Must be from 0 to 59. |
Required* |
Immutable. Fractions of seconds in nanoseconds. Must be from 0 to 999,999,999. |
Required* |
Immutable. Seconds of minutes of the time. Must normally be from 0 to 59. |
Optional |
Status of the job. Default: ENABLED. NOTE: The effect of the new job status takes place during a subsequent job run. For example, if you change the job status from ENABLED to DISABLED, and an operation spawned by the transfer is running, the status change would not affect the current operation. |
Required |
Transfer specification. |
Optional |
An AWS S3 data source. |
Required* |
AWS credentials block. |
Required* |
AWS Key ID. |
Optional |
Value of the field. Cannot be used if 'valueFrom' is specified. |
Optional |
Source for the field's value. Cannot be used if 'value' is specified. |
Optional |
Reference to a value with the given key in the given Secret in the resource's namespace. |
Required* |
Key that identifies the value to be extracted. |
Required* |
Name of the Secret to extract a value from. |
Required* |
AWS Secret Access Key. |
Optional |
Value of the field. Cannot be used if 'valueFrom' is specified. |
Optional |
Source for the field's value. Cannot be used if 'value' is specified. |
Optional |
Reference to a value with the given key in the given Secret in the resource's namespace. |
Required* |
Key that identifies the value to be extracted. |
Required* |
Name of the Secret to extract a value from. |
Required* |
S3 Bucket name. |
Optional |
A Google Cloud Storage data sink. |
Required* |
|
Optional |
The name of a StorageBucket. |
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
Optional |
A Google Cloud Storage data source. |
Required* |
|
Optional |
The name of a StorageBucket. |
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
Optional |
An HTTP URL data source. |
Required* |
The URL that points to the file that stores the object list entries. This file must allow public access. Currently, only URLs with HTTP and HTTPS schemes are supported. |
Optional |
Only objects that satisfy these object conditions are included in the set of data source and data sink objects. Object conditions based on objects' last_modification_time do not exclude objects in a data sink. |
Optional |
exclude_prefixes must follow the requirements described for include_prefixes. |
Optional |
|
Optional |
If include_refixes is specified, objects that satisfy the object conditions must have names that start with one of the include_prefixes and that do not start with any of the exclude_prefixes. If include_prefixes is not specified, all objects except those that have names starting with one of the exclude_prefixes must satisfy the object conditions. |
Optional |
|
Optional |
A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". |
Optional |
A duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". |
Optional |
Characteristics of how to treat files from datasource and sink during job. If the option delete_objects_unique_in_sink is true, object conditions based on objects' last_modification_time are ignored and do not exclude objects in a data source or a data sink. |
Optional |
Whether objects should be deleted from the source after they are transferred to the sink. Note that this option and delete_objects_unique_in_sink are mutually exclusive. |
Optional |
Whether objects that exist only in the sink should be deleted. Note that this option and delete_objects_from_source_after_transfer are mutually exclusive. |
Optional |
Whether overwriting objects that already exist in the sink is allowed. |
* Field is required when parent field is specified
Status
Schema
conditions:
- lastTransitionTime: string
message: string
reason: string
status: string
type: string
creationTime: string
deletionTime: string
lastModificationTime: string
name: string
observedGeneration: integer
Fields | |
---|---|
conditions |
Conditions represent the latest available observation of the resource's current state. |
conditions.[] |
|
conditions.[].lastTransitionTime |
Last time the condition transitioned from one status to another. |
conditions.[].message |
Human-readable message indicating details about last transition. |
conditions.[].reason |
Unique, one-word, CamelCase reason for the condition's last transition. |
conditions.[].status |
Status is the status of the condition. Can be True, False, Unknown. |
conditions.[].type |
Type is the type of the condition. |
creationTime |
When the Transfer Job was created. |
deletionTime |
When the Transfer Job was deleted. |
lastModificationTime |
When the Transfer Job was last modified. |
name |
The name of the Transfer Job. |
observedGeneration |
ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource. |
Sample YAML(s)
Typical Use Case
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: storagetransfer.cnrm.cloud.google.com/v1beta1
kind: StorageTransferJob
metadata:
name: storagetransferjob-sample
spec:
description: "Sample storage transfer job"
schedule:
startTimeOfDay:
seconds: 0
hours: 0
minutes: 0
nanos: 0
scheduleEndDate:
day: 31
month: 12
year: 9999
scheduleStartDate:
day: 28
month: 1
year: 2020
status: ENABLED
transferSpec:
gcsDataSink:
bucketRef:
name: ${PROJECT_ID?}-storagetransferjob-dep1
gcsDataSource:
bucketRef:
name: ${PROJECT_ID?}-storagetransferjob-dep2
objectConditions:
maxTimeElapsedSinceLastModification: 5s
minTimeElapsedSinceLastModification: 2s
transferOptions:
deleteObjectsUniqueInSink: false
overwriteObjectsAlreadyExistingInSink: true
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
name: storagetransferjob-dep1
spec:
# replace ${PROJECT_NUMBER?} with your project number
member: serviceAccount:project-${PROJECT_NUMBER?}@storage-transfer-service.iam.gserviceaccount.com
role: roles/storage.admin
resourceRef:
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
name: ${PROJECT_ID?}-storagetransferjob-dep1
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
name: storagetransferjob-dep2
spec:
# replace ${PROJECT_NUMBER?} with your project number
member: serviceAccount:project-${PROJECT_NUMBER?}@storage-transfer-service.iam.gserviceaccount.com
role: roles/storage.admin
resourceRef:
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
name: ${PROJECT_ID?}-storagetransferjob-dep2
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
# StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
name: ${PROJECT_ID?}-storagetransferjob-dep1
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
# StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
name: ${PROJECT_ID?}-storagetransferjob-dep2