Package types

Version 4.0.1

• 5.15.1 (latest)
• 5.15.0
• 5.14.0
• 5.13.0
• 5.12.0
• 5.10.2
• 5.9.3
• 5.8.0
• 5.7.0
• 5.6.0
• 5.5.1
• 5.4.3
• 5.3.0
• 5.2.0
• 5.1.0
• 5.0.3
• 4.0.3
• 3.3.2
• 3.2.0
• 3.1.1
• 3.0.0
• 2.6.2
• 2.5.0
• 2.4.0
• 2.3.1
• 2.2.0
• 2.0.2
• 1.1.3
• 1.0.1
• 0.8.2
• 0.7.0
• 0.6.1
• 0.5.0

API documentation for dataproc_v1.types package.

Classes

AcceleratorConfig

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine <https://cloud.google.com/compute/docs/gpus/>__.

AutoscalingConfig

Autoscaling Policy config associated with the cluster.

AutoscalingPolicy

Describes an autoscaling policy for Dataproc cluster autoscaler.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

AuxiliaryServicesConfig

Auxiliary services configuration for a Cluster.

BasicAutoscalingAlgorithm

Basic algorithm for autoscaling.

BasicYarnAutoscalingConfig

Basic autoscaling configurations for YARN.

Batch

A representation of a batch workload in the service.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

BatchOperationMetadata

Metadata describing the Batch operation.

CancelJobRequest

A request to cancel a job.

Cluster

Describes the identifying information, config, and status of a Dataproc cluster

ClusterConfig

The cluster config.

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

ClusterOperation

The cluster operation triggered by a workflow.

ClusterOperationMetadata

Metadata describing the operation.

ClusterOperationStatus

The status of the operation.

ClusterSelector

A selector that chooses target cluster for jobs based on metadata.

ClusterStatus

The status of a cluster and its instances.

Component

Cluster components that can be activated.

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs <https://cloud.google.com/compute/confidential-vm/docs>__

CreateAutoscalingPolicyRequest

A request to create an autoscaling policy.

CreateBatchRequest

A request to create a batch workload.

CreateClusterRequest

A request to create a cluster.

CreateWorkflowTemplateRequest

A request to create a workflow template.

DeleteAutoscalingPolicyRequest

A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.

DeleteBatchRequest

A request to delete a batch workload.

DeleteClusterRequest

A request to delete a cluster.

DeleteJobRequest

A request to delete a job.

DeleteWorkflowTemplateRequest

A request to delete a workflow template. Currently started workflows will remain running.

DiagnoseClusterRequest

A request to collect cluster diagnostic information.

DiagnoseClusterResults

The location of diagnostic output.

DiskConfig

Specifies the config of disk options for a group of VM instances.

EncryptionConfig

Encryption settings for the cluster.

EndpointConfig

Endpoint config for this cluster

EnvironmentConfig

Environment configuration for a workload.

ExecutionConfig

Execution configuration for a workload.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

FailureAction

Actions in response to failure of a resource associated with a cluster.

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

GetAutoscalingPolicyRequest

A request to fetch an autoscaling policy.

GetBatchRequest

A request to get the resource representation for a batch workload.

GetClusterRequest

Request to get the resource representation for a cluster in a project.

GetJobRequest

A request to get the resource representation for a job in a project.

GetWorkflowTemplateRequest

A request to fetch a workflow template.

GkeClusterConfig

The cluster's GKE config.

GkeNodePoolConfig

The configuration of a GKE NodePool used by a Dataproc-on-GKE cluster <https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster>__.

GkeNodePoolTarget

GKE NodePools that Dataproc workloads run on.

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce <https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html> jobs on Apache Hadoop YARN <https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html>.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

HiveJob

A Dataproc job for running Apache Hive <https://hive.apache.org/>__ queries on YARN.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

InstanceGroupAutoscalingPolicyConfig

Configuration for the size bounds of an instance group, including its proportional size to other groups.

InstanceGroupConfig

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

InstantiateInlineWorkflowTemplateRequest

A request to instantiate an inline workflow template.

InstantiateWorkflowTemplateRequest

A request to instantiate a workflow template.

Job

A Dataproc job resource.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

JobMetadata

Job Operation metadata.

JobPlacement

Dataproc job config.

JobReference

Encapsulates the full scoping used to reference a job.

JobScheduling

Job scheduling options.

JobStatus

Dataproc job status.

KerberosConfig

Specifies Kerberos related configuration.

KubernetesClusterConfig

The configuration for running the Dataproc cluster on Kubernetes.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

KubernetesSoftwareConfig

The software configuration for this Dataproc cluster running on Kubernetes.

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

ListAutoscalingPoliciesRequest

A request to list autoscaling policies in a project.

ListAutoscalingPoliciesResponse

A response to a request to list autoscaling policies in a project.

ListBatchesRequest

A request to list batch workloads in a project.

ListBatchesResponse

A list of batch workloads.

ListClustersRequest

A request to list the clusters in a project.

ListClustersResponse

The list of all clusters in a project.

ListJobsRequest

A request to list jobs in a project.

ListJobsResponse

A list of jobs in a project.

ListWorkflowTemplatesRequest

A request to list workflow templates in a project.

ListWorkflowTemplatesResponse

A response to a request to list workflow templates in a project.

LoggingConfig

The runtime logging config of the job.

ManagedCluster

Cluster that is managed by the workflow.

ManagedGroupConfig

Specifies the resources used to actively manage an instance group.

MetastoreConfig

Specifies a Metastore configuration.

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups.

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

OrderedJob

A job executed by the workflow.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

ParameterValidation

Configuration for parameter validation.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

PeripheralsConfig

Auxiliary services configuration for a workload.

PigJob

A Dataproc job for running Apache Pig <https://pig.apache.org/>__ queries on YARN.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

PrestoJob

A Dataproc job for running Presto <https://prestosql.io/> queries. IMPORTANT: The Dataproc Presto Optional Component <https://cloud.google.com/dataproc/docs/concepts/components/presto> must be enabled when the cluster is created to submit a Presto job to the cluster.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

PySparkBatch

A configuration for running an Apache PySpark <https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html>__ batch workload.

PySparkJob

A Dataproc job for running Apache PySpark <https://spark.apache.org/docs/0.9.0/python-programming-guide.html>__ applications on YARN.

QueryList

A list of queries to run on a cluster.

RegexValidation

Validation based on regular expressions.

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

RuntimeConfig

Runtime configuration for a workload.

RuntimeInfo

Runtime information about workload execution.

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs <https://cloud.google.com/security/shielded-cloud/shielded-vm>__.

SoftwareConfig

Specifies the selection and config of software inside the cluster.

SparkBatch

A configuration for running an Apache Spark <http://spark.apache.org/>__ batch workload.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

SparkHistoryServerConfig

Spark History Server configuration for the workload.

SparkJob

A Dataproc job for running Apache Spark <http://spark.apache.org/>__ applications on YARN.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

SparkRBatch

A configuration for running an Apache SparkR <https://spark.apache.org/docs/latest/sparkr.html>__ batch workload.

SparkRJob

A Dataproc job for running Apache SparkR <https://spark.apache.org/docs/latest/sparkr.html>__ applications on YARN.

SparkSqlBatch

A configuration for running Apache Spark SQL <http://spark.apache.org/sql/>__ queries as a batch workload.

SparkSqlJob

A Dataproc job for running Apache Spark SQL <http://spark.apache.org/sql/>__ queries.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

StartClusterRequest

A request to start a cluster.

StopClusterRequest

A request to stop a cluster.

SubmitJobRequest

A request to submit a job.

TemplateParameter

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

• Labels
• File uris
• Job properties
• Job arguments
• Script variables
• Main class (in HadoopJob and SparkJob)
• Zone (in ClusterSelector)

UpdateAutoscalingPolicyRequest

A request to update an autoscaling policy.

UpdateClusterRequest

A request to update a cluster.

UpdateJobRequest

A request to update a job.

UpdateWorkflowTemplateRequest

A request to update a workflow template.

ValueValidation

Validation based on a list of allowed values.

VirtualClusterConfig

Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster <https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster>__.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

WorkflowGraph

The workflow graph.

WorkflowMetadata

A Dataproc workflow template resource.

WorkflowNode

The workflow node.

WorkflowTemplate

A Dataproc workflow template resource.

WorkflowTemplatePlacement

Specifies workflow execution target.

Either managed_cluster or cluster_selector is required.

This message has oneof_ fields (mutually exclusive fields). For each oneof, at most one member field can be set at the same time. Setting any member of the oneof automatically clears all other members.

.. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields

YarnApplication

A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.