Classes
AcceleratorConfig
Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.
AutoscalingConfig
Autoscaling Policy config associated with the cluster.
AutoscalingPolicy
Describes an autoscaling policy for Dataproc cluster autoscaler.
AutoscalingPolicyName
Resource name for the AutoscalingPolicy resource.
AutoscalingPolicyService
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyService.AutoscalingPolicyServiceBase
Base class for server-side implementations of AutoscalingPolicyService
AutoscalingPolicyService.AutoscalingPolicyServiceClient
Client for AutoscalingPolicyService
AutoscalingPolicyServiceClient
AutoscalingPolicyService client wrapper, for convenient use.
AutoscalingPolicyServiceClientBuilder
Builder class for AutoscalingPolicyServiceClient to provide simple configuration of credentials, endpoint etc.
AutoscalingPolicyServiceClientImpl
AutoscalingPolicyService client wrapper implementation, for convenient use.
AutoscalingPolicyServiceSettings
Settings for AutoscalingPolicyServiceClient instances.
AuxiliaryNodeGroup
Node group identification and configuration information.
AuxiliaryServicesConfig
Auxiliary services configuration for a Cluster.
BasicAutoscalingAlgorithm
Basic algorithm for autoscaling.
BasicYarnAutoscalingConfig
Basic autoscaling configurations for YARN.
Batch
A representation of a batch workload in the service.
Batch.Types
Container for nested types declared in the Batch message type.
Batch.Types.StateHistory
Historical state information.
BatchController
The BatchController provides methods to manage batch workloads.
BatchController.BatchControllerBase
Base class for server-side implementations of BatchController
BatchController.BatchControllerClient
Client for BatchController
BatchControllerClient
BatchController client wrapper, for convenient use.
BatchControllerClientBuilder
Builder class for BatchControllerClient to provide simple configuration of credentials, endpoint etc.
BatchControllerClientImpl
BatchController client wrapper implementation, for convenient use.
BatchControllerSettings
Settings for BatchControllerClient instances.
BatchName
Resource name for the Batch resource.
BatchOperationMetadata
Metadata describing the Batch operation.
BatchOperationMetadata.Types
Container for nested types declared in the BatchOperationMetadata message type.
CancelJobRequest
A request to cancel a job.
Cluster
Describes the identifying information, config, and status of a Dataproc cluster
ClusterConfig
The cluster config.
ClusterController
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterController.ClusterControllerBase
Base class for server-side implementations of ClusterController
ClusterController.ClusterControllerClient
Client for ClusterController
ClusterControllerClient
ClusterController client wrapper, for convenient use.
ClusterControllerClientBuilder
Builder class for ClusterControllerClient to provide simple configuration of credentials, endpoint etc.
ClusterControllerClientImpl
ClusterController client wrapper implementation, for convenient use.
ClusterControllerSettings
Settings for ClusterControllerClient instances.
ClusterMetrics
Contains cluster daemon metrics, such as HDFS and YARN stats.
Beta Feature: This report is available for testing purposes only. It may be changed before final release.
ClusterName
Resource name for the Cluster resource.
ClusterOperation
The cluster operation triggered by a workflow.
ClusterOperationMetadata
Metadata describing the operation.
ClusterOperationStatus
The status of the operation.
ClusterOperationStatus.Types
Container for nested types declared in the ClusterOperationStatus message type.
ClusterRegionName
Resource name for the ClusterRegion resource.
ClusterSelector
A selector that chooses target cluster for jobs based on metadata.
ClusterStatus
The status of a cluster and its instances.
ClusterStatus.Types
Container for nested types declared in the ClusterStatus message type.
ConfidentialInstanceConfig
Confidential Instance Config for clusters using Confidential VMs
CreateAutoscalingPolicyRequest
A request to create an autoscaling policy.
CreateBatchRequest
A request to create a batch workload.
CreateClusterRequest
A request to create a cluster.
CreateNodeGroupRequest
A request to create a node group.
CreateSessionRequest
A request to create a session.
CreateSessionTemplateRequest
A request to create a session template.
CreateWorkflowTemplateRequest
A request to create a workflow template.
DataprocMetricConfig
Dataproc metric config.
DataprocMetricConfig.Types
Container for nested types declared in the DataprocMetricConfig message type.
DataprocMetricConfig.Types.Metric
A Dataproc custom metric.
DeleteAutoscalingPolicyRequest
A request to delete an autoscaling policy.
Autoscaling policies in use by one or more clusters will not be deleted.
DeleteBatchRequest
A request to delete a batch workload.
DeleteClusterRequest
A request to delete a cluster.
DeleteJobRequest
A request to delete a job.
DeleteSessionRequest
A request to delete a session.
DeleteSessionTemplateRequest
A request to delete a session template.
DeleteWorkflowTemplateRequest
A request to delete a workflow template.
Currently started workflows will remain running.
DiagnoseClusterRequest
A request to collect cluster diagnostic information.
DiagnoseClusterResults
The location of diagnostic output.
DiskConfig
Specifies the config of disk options for a group of VM instances.
DriverSchedulingConfig
Driver scheduling configuration.
EncryptionConfig
Encryption settings for the cluster.
EndpointConfig
Endpoint config for this cluster
EnvironmentConfig
Environment configuration for a workload.
ExecutionConfig
Execution configuration for a workload.
GceClusterConfig
Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.
GceClusterConfig.Types
Container for nested types declared in the GceClusterConfig message type.
GetAutoscalingPolicyRequest
A request to fetch an autoscaling policy.
GetBatchRequest
A request to get the resource representation for a batch workload.
GetClusterRequest
Request to get the resource representation for a cluster in a project.
GetJobRequest
A request to get the resource representation for a job in a project.
GetNodeGroupRequest
A request to get a node group .
GetSessionRequest
A request to get the resource representation for a session.
GetSessionTemplateRequest
A request to get the resource representation for a session template.
GetWorkflowTemplateRequest
A request to fetch a workflow template.
GkeClusterConfig
The cluster's GKE config.
GkeNodePoolConfig
The configuration of a GKE node pool used by a Dataproc-on-GKE cluster.
GkeNodePoolConfig.Types
Container for nested types declared in the GkeNodePoolConfig message type.
GkeNodePoolConfig.Types.GkeNodeConfig
Parameters that describe cluster nodes.
GkeNodePoolConfig.Types.GkeNodePoolAcceleratorConfig
A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.
GkeNodePoolConfig.Types.GkeNodePoolAutoscalingConfig
GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.
GkeNodePoolTarget
GKE node pools that Dataproc workloads run on.
GkeNodePoolTarget.Types
Container for nested types declared in the GkeNodePoolTarget message type.
HadoopJob
A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.
HiveJob
A Dataproc job for running Apache Hive queries on YARN.
IdentityConfig
Identity related configuration, including service account based secure multi-tenancy user mappings.
InstanceFlexibilityPolicy
Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.
InstanceFlexibilityPolicy.Types
Container for nested types declared in the InstanceFlexibilityPolicy message type.
InstanceFlexibilityPolicy.Types.InstanceSelection
Defines machines types and a rank to which the machines types belong.
InstanceFlexibilityPolicy.Types.InstanceSelectionResult
Defines a mapping from machine types to the number of VMs that are created with each machine type.
InstanceGroupAutoscalingPolicyConfig
Configuration for the size bounds of an instance group, including its proportional size to other groups.
InstanceGroupConfig
The config settings for Compute Engine resources in an instance group, such as a master or worker group.
InstanceGroupConfig.Types
Container for nested types declared in the InstanceGroupConfig message type.
InstanceReference
A reference to a Compute Engine instance.
InstantiateInlineWorkflowTemplateRequest
A request to instantiate an inline workflow template.
InstantiateWorkflowTemplateRequest
A request to instantiate a workflow template.
Job
A Dataproc job resource.
JobController
The JobController provides methods to manage jobs.
JobController.JobControllerBase
Base class for server-side implementations of JobController
JobController.JobControllerClient
Client for JobController
JobControllerClient
JobController client wrapper, for convenient use.
JobControllerClientBuilder
Builder class for JobControllerClient to provide simple configuration of credentials, endpoint etc.
JobControllerClientImpl
JobController client wrapper implementation, for convenient use.
JobControllerSettings
Settings for JobControllerClient instances.
JobMetadata
Job Operation metadata.
JobPlacement
Dataproc job config.
JobReference
Encapsulates the full scoping used to reference a job.
JobScheduling
Job scheduling options.
JobStatus
Dataproc job status.
JobStatus.Types
Container for nested types declared in the JobStatus message type.
JupyterConfig
Jupyter configuration for an interactive session.
JupyterConfig.Types
Container for nested types declared in the JupyterConfig message type.
KerberosConfig
Specifies Kerberos related configuration.
KubernetesClusterConfig
The configuration for running the Dataproc cluster on Kubernetes.
KubernetesSoftwareConfig
The software configuration for this Dataproc cluster running on Kubernetes.
LifecycleConfig
Specifies the cluster auto-delete schedule configuration.
ListAutoscalingPoliciesRequest
A request to list autoscaling policies in a project.
ListAutoscalingPoliciesResponse
A response to a request to list autoscaling policies in a project.
ListBatchesRequest
A request to list batch workloads in a project.
ListBatchesResponse
A list of batch workloads.
ListClustersRequest
A request to list the clusters in a project.
ListClustersResponse
The list of all clusters in a project.
ListJobsRequest
A request to list jobs in a project.
ListJobsRequest.Types
Container for nested types declared in the ListJobsRequest message type.
ListJobsResponse
A list of jobs in a project.
ListSessionTemplatesRequest
A request to list session templates in a project.
ListSessionTemplatesResponse
A list of session templates.
ListSessionsRequest
A request to list sessions in a project.
ListSessionsResponse
A list of interactive sessions.
ListWorkflowTemplatesRequest
A request to list workflow templates in a project.
ListWorkflowTemplatesResponse
A response to a request to list workflow templates in a project.
LoggingConfig
The runtime logging config of the job.
LoggingConfig.Types
Container for nested types declared in the LoggingConfig message type.
ManagedCluster
Cluster that is managed by the workflow.
ManagedGroupConfig
Specifies the resources used to actively manage an instance group.
MetastoreConfig
Specifies a Metastore configuration.
NodeGroup
Dataproc Node Group.
The Dataproc NodeGroup resource is not related to the
Dataproc [NodeGroupAffinity][google.cloud.dataproc.v1.NodeGroupAffinity]
resource.
NodeGroup.Types
Container for nested types declared in the NodeGroup message type.
NodeGroupAffinity
Node Group Affinity for clusters using sole-tenant node groups.
The Dataproc NodeGroupAffinity resource is not related to the
Dataproc [NodeGroup][google.cloud.dataproc.v1.NodeGroup] resource.
NodeGroupController
The NodeGroupControllerService provides methods to manage node groups
of Compute Engine managed instances.
NodeGroupController.NodeGroupControllerBase
Base class for server-side implementations of NodeGroupController
NodeGroupController.NodeGroupControllerClient
Client for NodeGroupController
NodeGroupControllerClient
NodeGroupController client wrapper, for convenient use.
NodeGroupControllerClientBuilder
Builder class for NodeGroupControllerClient to provide simple configuration of credentials, endpoint etc.
NodeGroupControllerClientImpl
NodeGroupController client wrapper implementation, for convenient use.
NodeGroupControllerSettings
Settings for NodeGroupControllerClient instances.
NodeGroupName
Resource name for the NodeGroup resource.
NodeGroupOperationMetadata
Metadata describing the node group operation.
NodeGroupOperationMetadata.Types
Container for nested types declared in the NodeGroupOperationMetadata message type.
NodeInitializationAction
Specifies an executable to run on a fully configured node and a timeout period for executable completion.
OrderedJob
A job executed by the workflow.
ParameterValidation
Configuration for parameter validation.
PeripheralsConfig
Auxiliary services configuration for a workload.
PigJob
A Dataproc job for running Apache Pig queries on YARN.
PrestoJob
A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.
PyPiRepositoryConfig
Configuration for PyPi repository
PySparkBatch
A configuration for running an Apache PySpark batch workload.
PySparkJob
A Dataproc job for running Apache PySpark applications on YARN.
QueryList
A list of queries to run on a cluster.
RegexValidation
Validation based on regular expressions.
RegionName
Resource name for the Region resource.
RepositoryConfig
Configuration for dependency repositories
ReservationAffinity
Reservation Affinity for consuming Zonal reservation.
ReservationAffinity.Types
Container for nested types declared in the ReservationAffinity message type.
ResizeNodeGroupRequest
A request to resize a node group.
RuntimeConfig
Runtime configuration for a workload.
RuntimeInfo
Runtime information about workload execution.
SecurityConfig
Security related configuration, including encryption, Kerberos, etc.
ServiceName
Resource name for the Service resource.
Session
A representation of a session.
Session.Types
Container for nested types declared in the Session message type.
Session.Types.SessionStateHistory
Historical state information.
SessionController
The SessionController provides methods to manage interactive sessions.
SessionController.SessionControllerBase
Base class for server-side implementations of SessionController
SessionController.SessionControllerClient
Client for SessionController
SessionControllerClient
SessionController client wrapper, for convenient use.
SessionControllerClientBuilder
Builder class for SessionControllerClient to provide simple configuration of credentials, endpoint etc.
SessionControllerClientImpl
SessionController client wrapper implementation, for convenient use.
SessionControllerSettings
Settings for SessionControllerClient instances.
SessionName
Resource name for the Session resource.
SessionOperationMetadata
Metadata describing the Session operation.
SessionOperationMetadata.Types
Container for nested types declared in the SessionOperationMetadata message type.
SessionTemplate
A representation of a session template.
SessionTemplateController
The SessionTemplateController provides methods to manage session templates.
SessionTemplateController.SessionTemplateControllerBase
Base class for server-side implementations of SessionTemplateController
SessionTemplateController.SessionTemplateControllerClient
Client for SessionTemplateController
SessionTemplateControllerClient
SessionTemplateController client wrapper, for convenient use.
SessionTemplateControllerClientBuilder
Builder class for SessionTemplateControllerClient to provide simple configuration of credentials, endpoint etc.
SessionTemplateControllerClientImpl
SessionTemplateController client wrapper implementation, for convenient use.
SessionTemplateControllerSettings
Settings for SessionTemplateControllerClient instances.
SessionTemplateName
Resource name for the SessionTemplate resource.
ShieldedInstanceConfig
Shielded Instance Config for clusters using Compute Engine Shielded VMs.
SoftwareConfig
Specifies the selection and config of software inside the cluster.
SparkBatch
A configuration for running an Apache Spark batch workload.
SparkHistoryServerConfig
Spark History Server configuration for the workload.
SparkJob
A Dataproc job for running Apache Spark applications on YARN.
SparkRBatch
A configuration for running an Apache SparkR batch workload.
SparkRJob
A Dataproc job for running Apache SparkR applications on YARN.
SparkSqlBatch
A configuration for running Apache Spark SQL queries as a batch workload.
SparkSqlJob
A Dataproc job for running Apache Spark SQL queries.
StartClusterRequest
A request to start a cluster.
StartupConfig
Configuration to handle the startup of instances during cluster create and update process.
StopClusterRequest
A request to stop a cluster.
SubmitJobRequest
A request to submit a job.
TemplateParameter
A configurable parameter that replaces one or more fields in the template. Parameterizable fields:
- Labels
- File uris
- Job properties
- Job arguments
- Script variables
- Main class (in HadoopJob and SparkJob)
- Zone (in ClusterSelector)
TerminateSessionRequest
A request to terminate an interactive session.
TrinoJob
A Dataproc job for running Trino queries. IMPORTANT: The Dataproc Trino Optional Component must be enabled when the cluster is created to submit a Trino job to the cluster.
UpdateAutoscalingPolicyRequest
A request to update an autoscaling policy.
UpdateClusterRequest
A request to update a cluster.
UpdateJobRequest
A request to update a job.
UpdateSessionTemplateRequest
A request to update a session template.
UpdateWorkflowTemplateRequest
A request to update a workflow template.
UsageMetrics
Usage metrics represent approximate total resources consumed by a workload.
UsageSnapshot
The usage snapshot represents the resources consumed by a workload at a specified time.
ValueValidation
Validation based on a list of allowed values.
VirtualClusterConfig
The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.
WorkflowGraph
The workflow graph.
WorkflowMetadata
A Dataproc workflow template resource.
WorkflowMetadata.Types
Container for nested types declared in the WorkflowMetadata message type.
WorkflowNode
The workflow node.
WorkflowNode.Types
Container for nested types declared in the WorkflowNode message type.
WorkflowTemplate
A Dataproc workflow template resource.
WorkflowTemplateName
Resource name for the WorkflowTemplate resource.
WorkflowTemplatePlacement
Specifies workflow execution target.
Either managed_cluster or cluster_selector is required.
WorkflowTemplateService
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateService.WorkflowTemplateServiceBase
Base class for server-side implementations of WorkflowTemplateService
WorkflowTemplateService.WorkflowTemplateServiceClient
Client for WorkflowTemplateService
WorkflowTemplateServiceClient
WorkflowTemplateService client wrapper, for convenient use.
WorkflowTemplateServiceClientBuilder
Builder class for WorkflowTemplateServiceClient to provide simple configuration of credentials, endpoint etc.
WorkflowTemplateServiceClientImpl
WorkflowTemplateService client wrapper implementation, for convenient use.
WorkflowTemplateServiceSettings
Settings for WorkflowTemplateServiceClient instances.
YarnApplication
A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>.
Beta Feature: This report is available for testing purposes only. It may be changed before final release.
YarnApplication.Types
Container for nested types declared in the YarnApplication message type.
Enums
AutoscalingPolicy.AlgorithmOneofCase
Enum of possible cases for the "algorithm" oneof.
AutoscalingPolicyName.ResourceNameType
The possible contents of AutoscalingPolicyName.
BasicAutoscalingAlgorithm.ConfigOneofCase
Enum of possible cases for the "config" oneof.
Batch.BatchConfigOneofCase
Enum of possible cases for the "batch_config" oneof.
Batch.Types.State
The batch state.
BatchName.ResourceNameType
The possible contents of BatchName.
BatchOperationMetadata.Types.BatchOperationType
Operation type for Batch resources
ClusterName.ResourceNameType
The possible contents of ClusterName.
ClusterOperationStatus.Types.State
The operation state.
ClusterRegionName.ResourceNameType
The possible contents of ClusterRegionName.
ClusterStatus.Types.State
The cluster state.
ClusterStatus.Types.Substate
The cluster substate.
Component
Cluster components that can be activated.
DataprocMetricConfig.Types.MetricSource
A source for the collection of Dataproc custom metrics (see Custom metrics).
ExecutionConfig.NetworkOneofCase
Enum of possible cases for the "network" oneof.
FailureAction
Actions in response to failure of a resource associated with a cluster.
GceClusterConfig.Types.PrivateIpv6GoogleAccess
PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes
can communicate with Google Services through gRPC over IPv6.
These values are directly mapped to corresponding values in the
Compute Engine Instance
fields.
GkeNodePoolTarget.Types.Role
Role specifies the tasks that will run on the node pool. Roles can be
specific to workloads. Exactly one
[GkeNodePoolTarget][google.cloud.dataproc.v1.GkeNodePoolTarget] within the
virtual cluster must have the DEFAULT role, which is used to run all
workloads that are not associated with a node pool.
HadoopJob.DriverOneofCase
Enum of possible cases for the "driver" oneof.
HiveJob.QueriesOneofCase
Enum of possible cases for the "queries" oneof.
InstanceGroupConfig.Types.Preemptibility
Controls the use of preemptible instances within the group.
Job.TypeJobOneofCase
Enum of possible cases for the "type_job" oneof.
JobStatus.Types.State
The job state.
JobStatus.Types.Substate
The job substate.
JupyterConfig.Types.Kernel
Jupyter kernel types.
KubernetesClusterConfig.ConfigOneofCase
Enum of possible cases for the "config" oneof.
LifecycleConfig.TtlOneofCase
Enum of possible cases for the "ttl" oneof.
ListJobsRequest.Types.JobStateMatcher
A matcher that specifies categories of job states.
LoggingConfig.Types.Level
The Log4j level for job execution. When running an Apache Hive job, Cloud Dataproc configures the Hive client to an equivalent verbosity level.
NodeGroup.Types.Role
Node pool roles.
NodeGroupName.ResourceNameType
The possible contents of NodeGroupName.
NodeGroupOperationMetadata.Types.NodeGroupOperationType
Operation type for node group resources.
OrderedJob.JobTypeOneofCase
Enum of possible cases for the "job_type" oneof.
ParameterValidation.ValidationTypeOneofCase
Enum of possible cases for the "validation_type" oneof.
PigJob.QueriesOneofCase
Enum of possible cases for the "queries" oneof.
PrestoJob.QueriesOneofCase
Enum of possible cases for the "queries" oneof.
RegionName.ResourceNameType
The possible contents of RegionName.
ReservationAffinity.Types.Type
Indicates whether to consume capacity from an reservation or not.
ServiceName.ResourceNameType
The possible contents of ServiceName.
Session.SessionConfigOneofCase
Enum of possible cases for the "session_config" oneof.
Session.Types.State
The session state.
SessionName.ResourceNameType
The possible contents of SessionName.
SessionOperationMetadata.Types.SessionOperationType
Operation type for Session resources
SessionTemplate.SessionConfigOneofCase
Enum of possible cases for the "session_config" oneof.
SessionTemplateName.ResourceNameType
The possible contents of SessionTemplateName.
SparkBatch.DriverOneofCase
Enum of possible cases for the "driver" oneof.
SparkJob.DriverOneofCase
Enum of possible cases for the "driver" oneof.
SparkSqlJob.QueriesOneofCase
Enum of possible cases for the "queries" oneof.
TrinoJob.QueriesOneofCase
Enum of possible cases for the "queries" oneof.
VirtualClusterConfig.InfrastructureConfigOneofCase
Enum of possible cases for the "infrastructure_config" oneof.
WorkflowMetadata.Types.State
The operation state.
WorkflowNode.Types.NodeState
The workflow node state.
WorkflowTemplateName.ResourceNameType
The possible contents of WorkflowTemplateName.
WorkflowTemplatePlacement.PlacementOneofCase
Enum of possible cases for the "placement" oneof.
YarnApplication.Types.State
The application state, corresponding to <code>YarnProtos.YarnApplicationStateProto</code>.