DataflowFlexTemplateJob

Property Value
Google Cloud Service Name Cloud Dataflow
Google Cloud Service Documentation /dataflow/docs/
Google Cloud REST Resource Name v1b3.projects.jobs
Google Cloud REST Resource Documentation /dataflow/docs/reference/rest/v1b3/projects.jobs
Config Connector Resource Short Names gcpdataflowflextemplatejob
gcpdataflowflextemplatejobs
dataflowflextemplatejob
Config Connector Service Name dataflow.googleapis.com
Config Connector Resource Fully Qualified Name dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember No

Custom Resource Definition Properties

Annotations

Fields
cnrm.cloud.google.com/on-delete
cnrm.cloud.google.com/project-id

Spec

Schema

  containerSpecGcsPath: string
  parameters: {}
  region: string
Fields

containerSpecGcsPath

Required

string

Immutable.

parameters

Optional

object

Immutable.

region

Optional

string

Immutable. The region in which the created job should run.

Status

Schema

  conditions:
  - lastTransitionTime: string
    message: string
    reason: string
    status: string
    type: string
  jobId: string
  state: string
Fields
conditions

list (object)

Conditions represents the latest available observation of the resource's current state.
conditions.[]

object
conditions.[].lastTransitionTime

string

Last time the condition transitioned from one status to another.
conditions.[].message

string

Human-readable message indicating details about last transition.
conditions.[].reason

string

Unique, one-word, CamelCase reason for the condition's last transition.
conditions.[].status

string

Status is the status of the condition. Can be True, False, Unknown.
conditions.[].type

string

Type is the type of the condition.
jobId

string
state

string

Sample YAML(s)

Batch Dataflow Flex Template Job

  # Copyright 2020 Google LLC
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  
  apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
  kind: DataflowFlexTemplateJob
  metadata:
    annotations:
      cnrm.cloud.google.com/on-delete: "cancel"
    name: dataflowflextemplatejob-sample-batch
  spec:
    region: us-central1
    # This is a public, Google-maintained Dataflow Job flex template of a batch job
    containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/File_Format_Conversion
    parameters:
      inputFileFormat: csv
      outputFileFormat: avro
      # This is a public, Google-maintained csv file expressly for this sample.
      inputFileSpec: gs://config-connector-samples/dataflowflextemplate/numbertest.csv
      # Replace ${PROJECT_ID?} with your project ID.
      outputBucket: gs://${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
      # This is a public, Google-maintained Avro schema file expressly for this sample.
      schema: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
  ---
  apiVersion: storage.cnrm.cloud.google.com/v1beta1
  kind: StorageBucket
  metadata:
    # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
    name: ${PROJECT_ID?}-dataflowflextemplatejob-dep-batch

Streaming Dataflow Flex Template Job

  # Copyright 2020 Google LLC
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  
  apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
  kind: DataflowFlexTemplateJob
  metadata:
    annotations:
      cnrm.cloud.google.com/on-delete: "drain"
    name: dataflowflextemplatejob-sample-streaming
  spec:
    region: us-central1
    # This is a public, Google-maintained Dataflow Job flex template of a streaming job
    containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/PubSub_Avro_to_BigQuery
    parameters:
      # This is a public, Google-maintained Avro schema file expressly for this sample.
      schemaPath: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
      # Replace ${PROJECT_ID?} with your project ID.
      inputSubscription: projects/${PROJECT_ID?}/subscriptions/dataflowflextemplatejob-dep-streaming
      outputTopic: projects/${PROJECT_ID?}/topics/dataflowflextemplatejob-dep1-streaming
      outputTableSpec: ${PROJECT_ID?}:dataflowflextemplatejobdepstreaming.dataflowflextemplatejobdepstreaming
      createDisposition: CREATE_NEVER
  ---
  apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
  kind: BigQueryDataset
  metadata:
    name: dataflowflextemplatejobdepstreaming
  ---
  apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
  kind: BigQueryTable
  metadata:
    name: dataflowflextemplatejobdepstreaming
  spec:
    datasetRef:
      name: dataflowflextemplatejobdepstreaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubSubscription
  metadata:
    name: dataflowflextemplatejob-dep-streaming
  spec:
    topicRef:
      name: dataflowflextemplatejob-dep0-streaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    name: dataflowflextemplatejob-dep0-streaming
  ---
  apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
  kind: PubSubTopic
  metadata:
    name: dataflowflextemplatejob-dep1-streaming