| Property | Value |
|---|---|
| Google Cloud Service Name | Cloud Dataflow |
| Google Cloud Service Documentation | /dataflow/docs/ |
| Google Cloud REST Resource Name | v1b3.projects.jobs |
| Google Cloud REST Resource Documentation | /dataflow/docs/reference/rest/v1b3/projects.jobs |
| Config Connector Resource Short Names | gcpdataflowflextemplatejob gcpdataflowflextemplatejobs dataflowflextemplatejob |
| Config Connector Service Name | dataflow.googleapis.com |
| Config Connector Resource Fully Qualified Name | dataflowflextemplatejobs.dataflow.cnrm.cloud.google.com |
| Can Be Referenced by IAMPolicy/IAMPolicyMember | No |
| Config Connector Default Average Reconcile Interval In Seconds | 600 |
Custom Resource Definition Properties
Annotations
| Fields | |
|---|---|
cnrm.cloud.google.com/on-delete |
|
cnrm.cloud.google.com/project-id |
|
cnrm.cloud.google.com/skip-wait-on-job-termination |
|
cnrm.cloud.google.com/state-into-spec |
|
Spec
Schema
additionalExperiments:
- string
autoscalingAlgorithm: string
containerSpecGcsPath: string
enableStreamingEngine: boolean
ipConfiguration: string
kmsKeyNameRef:
external: string
name: string
namespace: string
launcherMachineType: string
machineType: string
maxWorkers: integer
networkRef:
external: string
name: string
namespace: string
numWorkers: integer
parameters: {}
region: string
sdkContainerImage: string
serviceAccountEmailRef:
external: string
name: string
namespace: string
stagingLocation: string
subnetworkRef:
external: string
name: string
namespace: string
tempLocation: string
transformNameMapping: {}
| Fields | |
|---|---|
|
Optional |
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"]. |
|
Optional |
|
|
Optional |
The algorithm to use for autoscaling. |
|
Required |
|
|
Optional |
Immutable. Indicates if the job should use the streaming engine feature. |
|
Optional |
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE". |
|
Optional |
The name for the Cloud KMS key for the job. |
|
Optional |
Allowed value: The `selfLink` field of a `KMSCryptoKey` resource. |
|
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
|
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
|
Optional |
The machine type to use for launching the job. The default is n1-standard-1. |
|
Optional |
The machine type to use for the job. |
|
Optional |
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000. |
|
Optional |
|
|
Optional |
Allowed value: The `selfLink` field of a `ComputeNetwork` resource. |
|
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
|
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
|
Optional |
Immutable. The initial number of Google Compute Engine instances for the job. |
|
Optional |
|
|
Optional |
Immutable. The region in which the created job should run. |
|
Optional |
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines. |
|
Optional |
|
|
Optional |
Allowed value: The `email` field of an `IAMServiceAccount` resource. |
|
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
|
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
|
Optional |
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://. |
|
Optional |
|
|
Optional |
Allowed value: The `selfLink` field of a `ComputeSubnetwork` resource. |
|
Optional |
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names |
|
Optional |
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ |
|
Optional |
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://. |
|
Optional |
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job. |
Status
Schema
conditions:
- lastTransitionTime: string
message: string
reason: string
status: string
type: string
jobId: string
observedGeneration: integer
state: string
type: string
| Fields | |
|---|---|
conditions |
Conditions represent the latest available observation of the resource's current state. |
conditions[] |
|
conditions[].lastTransitionTime |
Last time the condition transitioned from one status to another. |
conditions[].message |
Human-readable message indicating details about last transition. |
conditions[].reason |
Unique, one-word, CamelCase reason for the condition's last transition. |
conditions[].status |
Status is the status of the condition. Can be True, False, Unknown. |
conditions[].type |
Type is the type of the condition. |
jobId |
|
observedGeneration |
ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource. |
state |
|
type |
The type of this job, selected from the JobType enum. |
Sample YAML(s)
Batch Dataflow Flex Template Job
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
annotations:
cnrm.cloud.google.com/on-delete: "cancel"
name: dataflowflextemplatejob-sample-batch
spec:
region: us-central1
# This is a public, Google-maintained Dataflow Job flex template of a batch job
containerSpecGcsPath: gs://dataflow-templates/2022-10-03-00_RC00/flex/File_Format_Conversion
parameters:
inputFileFormat: csv
outputFileFormat: avro
# This is a public, Google-maintained csv file expressly for this sample.
inputFileSpec: gs://config-connector-samples/dataflowflextemplate/numbertest.csv
# Replace ${PROJECT_ID?} with your project ID.
outputBucket: gs://${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
# This is a public, Google-maintained Avro schema file expressly for this sample.
schema: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
# StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
name: ${PROJECT_ID?}-dataflowflextemplatejob-dep-batch
Streaming Dataflow Flex Template Job
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: dataflow.cnrm.cloud.google.com/v1beta1
kind: DataflowFlexTemplateJob
metadata:
annotations:
cnrm.cloud.google.com/on-delete: "drain"
name: dataflowflextemplatejob-sample-streaming
spec:
region: us-central1
# This is a public, Google-maintained Dataflow Job flex template of a streaming job
containerSpecGcsPath: gs://dataflow-templates/2020-08-31-00_RC00/flex/PubSub_Avro_to_BigQuery
parameters:
# This is a public, Google-maintained Avro schema file expressly for this sample.
schemaPath: gs://config-connector-samples/dataflowflextemplate/numbers.avsc
# Replace ${PROJECT_ID?} with your project ID.
inputSubscription: projects/${PROJECT_ID?}/subscriptions/dataflowflextemplatejob-dep-streaming
outputTopic: projects/${PROJECT_ID?}/topics/dataflowflextemplatejob-dep1-streaming
outputTableSpec: ${PROJECT_ID?}:dataflowflextemplatejobdepstreaming.dataflowflextemplatejobdepstreaming
createDisposition: CREATE_NEVER
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
name: dataflowflextemplatejobdepstreaming
---
apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryTable
metadata:
name: dataflowflextemplatejobdepstreaming
spec:
datasetRef:
name: dataflowflextemplatejobdepstreaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
name: dataflowflextemplatejob-dep-streaming
spec:
topicRef:
name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
name: dataflowflextemplatejob-dep0-streaming
---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
name: dataflowflextemplatejob-dep1-streaming