Property | Value |
---|---|
Google Cloud Service Name | Cloud Dataflow |
Google Cloud Service Documentation | /dataflow/docs/ |
Google Cloud REST Resource Name | v1b3.projects.jobs |
Google Cloud REST Resource Documentation | /dataflow/docs/reference/rest/v1b3/projects.jobs |
Config Connector Resource Short Names | gcpdataflowflextemplatejob gcpdataflowflextemplatejobs dataflowflextemplatejob |
Config Connector Service Name | dataflow.googleapis.com |
Config Connector Resource Fully Qualified Name | dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com |
Can Be Referenced by IAMPolicy/IAMPolicyMember | No |
Custom Resource Definition Properties
Annotations
Fields | |
---|---|
cnrm.cloud.google.com/on-delete |
|
cnrm.cloud.google.com/project-id |
Spec
Schema
containerSpecGcsPath: string
parameters: {}
region: string
Fields | |
---|---|
Required |
Immutable. |
Optional |
Immutable. |
Optional |
Immutable. The region in which the created job should run. |
Status
Schema
conditions:
- lastTransitionTime: string
message: string
reason: string
status: string
type: string
jobId: string
state: string
Fields | |
---|---|
conditions |
Conditions represents the latest available observation of the resource's current state. |
conditions.[] |
|
conditions.[].lastTransitionTime |
Last time the condition transitioned from one status to another. |
conditions.[].message |
Human-readable message indicating details about last transition. |
conditions.[].reason |
Unique, one-word, CamelCase reason for the condition's last transition. |
conditions.[].status |
Status is the status of the condition. Can be True, False, Unknown. |
conditions.[].type |
Type is the type of the condition. |
jobId |
|
state |
|
Sample YAML(s)
Batch Dataflow Flex Template Job
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
annotations:
cnrm.cloud.google.com/on-delete: "cancel"
name: dataflowflextemplatejob-sample-batch
spec:
region: us-central1
# This is a public, Google-maintained Dataflow Job flex template of a batch job
containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/File_Format_Conversion
parameters:
inputFileFormat: csv
outputFileFormat: avro
# This is a public, Google-maintained csv file expressly for this sample.
inputFileSpec: gs://config-connector-samples/dataflowflextemplate/numbertest.csv
# Replace ${PROJECT_ID?} with your project ID.
outputBucket: gs://${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
# This is a public, Google-maintained Avro schema file expressly for this sample.
schema: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
# StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
name: ${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
Streaming Dataflow Flex Template Job
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
annotations:
cnrm.cloud.google.com/on-delete: "drain"
name: dataflowflextemplatejob-sample-streaming
spec:
region: us-central1
# This is a public, Google-maintained Dataflow Job flex template of a streaming job
containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/PubSub_Avro_to_BigQuery
parameters:
# This is a public, Google-maintained Avro schema file expressly for this sample.
schemaPath: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
# Replace ${PROJECT_ID?} with your project ID.
inputSubscription: projects/${PROJECT_ID?}/subscriptions/dataflowflextemplatejob-dep-streaming
outputTopic: projects/${PROJECT_ID?}/topics/dataflowflextemplatejob-dep1-streaming
outputTableSpec: ${PROJECT_ID?}:dataflowflextemplatejobdepstreaming.dataflowflextemplatejobdepstreaming
createDisposition: CREATE_NEVER
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
name: dataflowflextemplatejobdepstreaming
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryTable
metadata:
name: dataflowflextemplatejobdepstreaming
spec:
datasetRef:
name: dataflowflextemplatejobdepstreaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
name: dataflowflextemplatejob-dep-streaming
spec:
topicRef:
name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
name: dataflowflextemplatejob-dep1-streaming