DataflowFlexTemplateJob

Property	Value
Google Cloud Service Name	Cloud Dataflow
Google Cloud Service Documentation	/dataflow/docs/
Google Cloud REST Resource Name	v1b3.projects.jobs
Google Cloud REST Resource Documentation	/dataflow/docs/reference/rest/v1b3/projects.jobs
Config Connector Resource Short Names	gcpdataflowflextemplatejob gcpdataflowflextemplatejobs dataflowflextemplatejob
Config Connector Service Name	dataflow.googleapis.com
Config Connector Resource Fully Qualified Name	dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember	No

Custom Resource Definition Properties

Annotations

Fields
`cnrm.cloud.google.com/on-delete`
`cnrm.cloud.google.com/project-id`
`cnrm.cloud.google.com/skip-wait-on-job-termination`

Spec

Schema

  containerSpecGcsPath: string
  parameters: {}
  region: string

Fields

Fields
`containerSpecGcsPath` Required	`string`
`parameters` Optional	`object`
`region` Optional	`string` Immutable. The region in which the created job should run.

containerSpecGcsPath

Required

string

parameters

Optional

object

region

Optional

string

Immutable. The region in which the created job should run.

Status

Schema

  conditions:
  - lastTransitionTime: string
    message: string
    reason: string
    status: string
    type: string
  jobId: string
  observedGeneration: integer
  state: string

Fields
`conditions`	`list (object)` Conditions represent the latest available observation of the resource's current state.
`conditions[]`	`object`
`conditions[].lastTransitionTime`	`string` Last time the condition transitioned from one status to another.
`conditions[].message`	`string` Human-readable message indicating details about last transition.
`conditions[].reason`	`string` Unique, one-word, CamelCase reason for the condition's last transition.
`conditions[].status`	`string` Status is the status of the condition. Can be True, False, Unknown.
`conditions[].type`	`string` Type is the type of the condition.
`jobId`	`string`
`observedGeneration`	`integer` ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.
`state`	`string`

Sample YAML(s)

Batch Dataflow Flex Template Job

  # Copyright 2020 Google LLC
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  
  apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
  kind: DataflowFlexTemplateJob
  metadata:
    annotations:
      cnrm.cloud.google.com/on-delete: "cancel"
    name: dataflowflextemplatejob-sample-batch
  spec:
    region: us-central1
    # This is a public, Google-maintained Dataflow Job flex template of a batch job
    containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/File_Format_Conversion
    parameters:
      inputFileFormat: csv
      outputFileFormat: avro
      # This is a public, Google-maintained csv file expressly for this sample.
      inputFileSpec: gs://config-connector-samples/dataflowflextemplate/numbertest.csv
      # Replace ${PROJECT_ID?} with your project ID.
      outputBucket: gs://${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
      # This is a public, Google-maintained Avro schema file expressly for this sample.
      schema: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
  ---
  apiVersion: storage.cnrm.cloud.google.com/v1beta1
  kind: StorageBucket
  metadata:
    # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
    name: ${PROJECT_ID?}-dataflowflextemplatejob-dep-batch

Streaming Dataflow Flex Template Job

  # Copyright 2020 Google LLC
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  
  apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
  kind: DataflowFlexTemplateJob
  metadata:
    annotations:
      cnrm.cloud.google.com/on-delete: "drain"
    name: dataflowflextemplatejob-sample-streaming
  spec:
    region: us-central1
    # This is a public, Google-maintained Dataflow Job flex template of a streaming job
    containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/PubSub_Avro_to_BigQuery
    parameters:
      # This is a public, Google-maintained Avro schema file expressly for this sample.
      schemaPath: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
      # Replace ${PROJECT_ID?} with your project ID.
      inputSubscription: projects/${PROJECT_ID?}/subscriptions/dataflowflextemplatejob-dep-streaming
      outputTopic: projects/${PROJECT_ID?}/topics/dataflowflextemplatejob-dep1-streaming
      outputTableSpec: ${PROJECT_ID?}:dataflowflextemplatejobdepstreaming.dataflowflextemplatejobdepstreaming
      createDisposition: CREATE_NEVER
  ---
  apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
  kind: BigQueryDataset
  metadata:
    name: dataflowflextemplatejobdepstreaming
  ---
  apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
  kind: BigQueryTable
  metadata:
    name: dataflowflextemplatejobdepstreaming
  spec:
    datasetRef:
      name: dataflowflextemplatejobdepstreaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubSubscription
  metadata:
    name: dataflowflextemplatejob-dep-streaming
  spec:
    topicRef:
      name: dataflowflextemplatejob-dep0-streaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    name: dataflowflextemplatejob-dep0-streaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    name: dataflowflextemplatejob-dep1-streaming