DataflowFlexTemplateJob

Stay organized with collections Save and categorize content based on your preferences.
Property Value
Google Cloud Service Name Cloud Dataflow
Google Cloud Service Documentation /dataflow/docs/
Google Cloud REST Resource Name v1b3.projects.jobs
Google Cloud REST Resource Documentation /dataflow/docs/reference/rest/v1b3/projects.jobs
Config Connector Resource Short Names gcpdataflowflextemplatejob
gcpdataflowflextemplatejobs
dataflowflextemplatejob
Config Connector Service Name dataflow.googleapis.com
Config Connector Resource Fully Qualified Name dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember No

Custom Resource Definition Properties

Annotations

Fields
cnrm.cloud.google.com/on-delete
cnrm.cloud.google.com/project-id
cnrm.cloud.google.com/skip-wait-on-job-termination
cnrm.cloud.google.com/state-into-spec

Spec

Schema

containerSpecGcsPath: string
parameters: {}
region: string
Fields

containerSpecGcsPath

Required

string

parameters

Optional

object

region

Optional

string

Immutable. The region in which the created job should run.

Status

Schema

conditions:
- lastTransitionTime: string
  message: string
  reason: string
  status: string
  type: string
jobId: string
observedGeneration: integer
state: string
Fields
conditions

list (object)

Conditions represent the latest available observation of the resource's current state.

conditions[]

object

conditions[].lastTransitionTime

string

Last time the condition transitioned from one status to another.

conditions[].message

string

Human-readable message indicating details about last transition.

conditions[].reason

string

Unique, one-word, CamelCase reason for the condition's last transition.

conditions[].status

string

Status is the status of the condition. Can be True, False, Unknown.

conditions[].type

string

Type is the type of the condition.

jobId

string

observedGeneration

integer

ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.

state

string

Sample YAML(s)

Batch Dataflow Flex Template Job

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
  annotations:
    cnrm.cloud.google.com/on-delete: "cancel"
  name: dataflowflextemplatejob-sample-batch
spec:
  region: us-central1
  # This is a public, Google-maintained Dataflow Job flex template of a batch job
  containerSpecGcsPath: gs://dataflow-templates/2022-10-03-00_RC00/flex/File_Format_Conversion
  parameters:
    inputFileFormat: csv
    outputFileFormat: avro
    # This is a public, Google-maintained csv file expressly for this sample.
    inputFileSpec: gs://config-connector-samples/dataflowflextemplate/numbertest.csv
    # Replace ${PROJECT_ID?} with your project ID.
    outputBucket: gs://${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
    # This is a public, Google-maintained Avro schema file expressly for this sample.
    schema: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
  name: ${PROJECT_ID?}-dataflowflextemplatejob-dep-batch

Streaming Dataflow Flex Template Job

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
  annotations:
    cnrm.cloud.google.com/on-delete: "drain"
  name: dataflowflextemplatejob-sample-streaming
spec:
  region: us-central1
  # This is a public, Google-maintained Dataflow Job flex template of a streaming job
  containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/PubSub_Avro_to_BigQuery
  parameters:
    # This is a public, Google-maintained Avro schema file expressly for this sample.
    schemaPath: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
    # Replace ${PROJECT_ID?} with your project ID.
    inputSubscription: projects/${PROJECT_ID?}/subscriptions/dataflowflextemplatejob-dep-streaming
    outputTopic: projects/${PROJECT_ID?}/topics/dataflowflextemplatejob-dep1-streaming
    outputTableSpec: ${PROJECT_ID?}:dataflowflextemplatejobdepstreaming.dataflowflextemplatejobdepstreaming
    createDisposition: CREATE_NEVER
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
  name: dataflowflextemplatejobdepstreaming
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryTable
metadata:
  name: dataflowflextemplatejobdepstreaming
spec:
  datasetRef:
    name: dataflowflextemplatejobdepstreaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
  name: dataflowflextemplatejob-dep-streaming
spec:
  topicRef:
    name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
  name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
  name: dataflowflextemplatejob-dep1-streaming