Package com.google.cloud.dataproc.v1 (3.1.2)

The interfaces provided are listed below, along with usage samples.

AutoscalingPolicyServiceClient

Service Description: The API interface for managing autoscaling policies in the Dataproc API.

Sample for AutoscalingPolicyServiceClient:


 try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
   AutoscalingPolicy response =
       autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
 }

BatchControllerClient

Service Description: The BatchController provides methods to manage batch workloads.

Sample for BatchControllerClient:


 try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
   BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
   Batch response = batchControllerClient.getBatch(name);
 }

ClusterControllerClient

Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

Sample for ClusterControllerClient:


 try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   String clusterName = "clusterName-1141738587";
   Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
 }

JobControllerClient

Service Description: The JobController provides methods to manage jobs.

Sample for JobControllerClient:


 try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   Job job = Job.newBuilder().build();
   Job response = jobControllerClient.submitJob(projectId, region, job);
 }

WorkflowTemplateServiceClient

Service Description: The API interface for managing Workflow Templates in the Dataproc API.

Sample for WorkflowTemplateServiceClient:


 try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
   WorkflowTemplate response =
       workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
 }

Classes

AcceleratorConfig

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

Protobuf type google.cloud.dataproc.v1.AcceleratorConfig

AcceleratorConfig.Builder

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

Protobuf type google.cloud.dataproc.v1.AcceleratorConfig

AutoscalingConfig

Autoscaling Policy config associated with the cluster.

Protobuf type google.cloud.dataproc.v1.AutoscalingConfig

AutoscalingConfig.Builder

Autoscaling Policy config associated with the cluster.

Protobuf type google.cloud.dataproc.v1.AutoscalingConfig

AutoscalingPolicy

Describes an autoscaling policy for Dataproc cluster autoscaler.

Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy

AutoscalingPolicy.Builder

Describes an autoscaling policy for Dataproc cluster autoscaler.

Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy

AutoscalingPolicyName.Builder

Builder for projects/{project}/locations/{location}/autoscalingPolicies/{autoscaling_policy}.

AutoscalingPolicyName.ProjectRegionAutoscalingPolicyBuilder

Builder for projects/{project}/regions/{region}/autoscalingPolicies/{autoscaling_policy}.

AutoscalingPolicyServiceClient

Service Description: The API interface for managing autoscaling policies in the Dataproc API.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
   AutoscalingPolicy response =
       autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
 }

Note: close() needs to be called on the AutoscalingPolicyServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.

This class can be customized by passing in a custom instance of AutoscalingPolicyServiceSettings to create(). For example:

To customize credentials:


 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     AutoscalingPolicyServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);

To customize the endpoint:


 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     AutoscalingPolicyServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesFixedSizeCollection

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPage

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPagedResponse

AutoscalingPolicyServiceGrpc

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceBlockingStub

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceFutureStub

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceImplBase

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceStub

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceSettings

Settings class to configure an instance of AutoscalingPolicyServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createAutoscalingPolicy to 30 seconds:


 AutoscalingPolicyServiceSettings.Builder autoscalingPolicyServiceSettingsBuilder =
     AutoscalingPolicyServiceSettings.newBuilder();
 autoscalingPolicyServiceSettingsBuilder
     .createAutoscalingPolicySettings()
     .setRetrySettings(
         autoscalingPolicyServiceSettingsBuilder
             .createAutoscalingPolicySettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     autoscalingPolicyServiceSettingsBuilder.build();

AutoscalingPolicyServiceSettings.Builder

Builder for AutoscalingPolicyServiceSettings.

AuxiliaryServicesConfig

Auxiliary services configuration for a Cluster.

Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig

AuxiliaryServicesConfig.Builder

Auxiliary services configuration for a Cluster.

Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig

BasicAutoscalingAlgorithm

Basic algorithm for autoscaling.

Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm

BasicAutoscalingAlgorithm.Builder

Basic algorithm for autoscaling.

Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm

BasicYarnAutoscalingConfig

Basic autoscaling configurations for YARN.

Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig

BasicYarnAutoscalingConfig.Builder

Basic autoscaling configurations for YARN.

Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig

Batch

A representation of a batch workload in the service.

Protobuf type google.cloud.dataproc.v1.Batch

Batch.Builder

A representation of a batch workload in the service.

Protobuf type google.cloud.dataproc.v1.Batch

Batch.StateHistory

Historical state information.

Protobuf type google.cloud.dataproc.v1.Batch.StateHistory

Batch.StateHistory.Builder

Historical state information.

Protobuf type google.cloud.dataproc.v1.Batch.StateHistory

BatchControllerClient

Service Description: The BatchController provides methods to manage batch workloads.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
   BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
   Batch response = batchControllerClient.getBatch(name);
 }

Note: close() needs to be called on the BatchControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of BatchControllerSettings to create(). For example:

To customize credentials:


 BatchControllerSettings batchControllerSettings =
     BatchControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 BatchControllerClient batchControllerClient =
     BatchControllerClient.create(batchControllerSettings);

To customize the endpoint:


 BatchControllerSettings batchControllerSettings =
     BatchControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 BatchControllerClient batchControllerClient =
     BatchControllerClient.create(batchControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

BatchControllerClient.ListBatchesFixedSizeCollection

BatchControllerClient.ListBatchesPage

BatchControllerClient.ListBatchesPagedResponse

BatchControllerGrpc

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerBlockingStub

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerFutureStub

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerImplBase

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerStub

The BatchController provides methods to manage batch workloads.

BatchControllerSettings

Settings class to configure an instance of BatchControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getBatch to 30 seconds:


 BatchControllerSettings.Builder batchControllerSettingsBuilder =
     BatchControllerSettings.newBuilder();
 batchControllerSettingsBuilder
     .getBatchSettings()
     .setRetrySettings(
         batchControllerSettingsBuilder
             .getBatchSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 BatchControllerSettings batchControllerSettings = batchControllerSettingsBuilder.build();

BatchControllerSettings.Builder

Builder for BatchControllerSettings.

BatchName

BatchName.Builder

Builder for projects/{project}/locations/{location}/batches/{batch}.

BatchOperationMetadata

Metadata describing the Batch operation.

Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata

BatchOperationMetadata.Builder

Metadata describing the Batch operation.

Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata

BatchesProto

CancelJobRequest

A request to cancel a job.

Protobuf type google.cloud.dataproc.v1.CancelJobRequest

CancelJobRequest.Builder

A request to cancel a job.

Protobuf type google.cloud.dataproc.v1.CancelJobRequest

Cluster

Describes the identifying information, config, and status of a Dataproc cluster

Protobuf type google.cloud.dataproc.v1.Cluster

Cluster.Builder

Describes the identifying information, config, and status of a Dataproc cluster

Protobuf type google.cloud.dataproc.v1.Cluster

ClusterConfig

The cluster config.

Protobuf type google.cloud.dataproc.v1.ClusterConfig

ClusterConfig.Builder

The cluster config.

Protobuf type google.cloud.dataproc.v1.ClusterConfig

ClusterControllerClient

Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   String clusterName = "clusterName-1141738587";
   Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
 }

Note: close() needs to be called on the ClusterControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of ClusterControllerSettings to create(). For example:

To customize credentials:


 ClusterControllerSettings clusterControllerSettings =
     ClusterControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 ClusterControllerClient clusterControllerClient =
     ClusterControllerClient.create(clusterControllerSettings);

To customize the endpoint:


 ClusterControllerSettings clusterControllerSettings =
     ClusterControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 ClusterControllerClient clusterControllerClient =
     ClusterControllerClient.create(clusterControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

ClusterControllerClient.ListClustersFixedSizeCollection

ClusterControllerClient.ListClustersPage

ClusterControllerClient.ListClustersPagedResponse

ClusterControllerGrpc

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerBlockingStub

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerFutureStub

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerImplBase

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerStub

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerSettings

Settings class to configure an instance of ClusterControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getCluster to 30 seconds:


 ClusterControllerSettings.Builder clusterControllerSettingsBuilder =
     ClusterControllerSettings.newBuilder();
 clusterControllerSettingsBuilder
     .getClusterSettings()
     .setRetrySettings(
         clusterControllerSettingsBuilder
             .getClusterSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 ClusterControllerSettings clusterControllerSettings = clusterControllerSettingsBuilder.build();

ClusterControllerSettings.Builder

Builder for ClusterControllerSettings.

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats. Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.ClusterMetrics

ClusterMetrics.Builder

Contains cluster daemon metrics, such as HDFS and YARN stats. Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.ClusterMetrics

ClusterOperation

The cluster operation triggered by a workflow.

Protobuf type google.cloud.dataproc.v1.ClusterOperation

ClusterOperation.Builder

The cluster operation triggered by a workflow.

Protobuf type google.cloud.dataproc.v1.ClusterOperation

ClusterOperationMetadata

Metadata describing the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata

ClusterOperationMetadata.Builder

Metadata describing the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata

ClusterOperationStatus

The status of the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus

ClusterOperationStatus.Builder

The status of the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus

ClusterSelector

A selector that chooses target cluster for jobs based on metadata.

Protobuf type google.cloud.dataproc.v1.ClusterSelector

ClusterSelector.Builder

A selector that chooses target cluster for jobs based on metadata.

Protobuf type google.cloud.dataproc.v1.ClusterSelector

ClusterStatus

The status of a cluster and its instances.

Protobuf type google.cloud.dataproc.v1.ClusterStatus

ClusterStatus.Builder

The status of a cluster and its instances.

Protobuf type google.cloud.dataproc.v1.ClusterStatus

ClustersProto

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs

Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig

ConfidentialInstanceConfig.Builder

Confidential Instance Config for clusters using Confidential VMs

Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig

CreateAutoscalingPolicyRequest

A request to create an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest

CreateAutoscalingPolicyRequest.Builder

A request to create an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest

CreateBatchRequest

A request to create a batch workload.

Protobuf type google.cloud.dataproc.v1.CreateBatchRequest

CreateBatchRequest.Builder

A request to create a batch workload.

Protobuf type google.cloud.dataproc.v1.CreateBatchRequest

CreateClusterRequest

A request to create a cluster.

Protobuf type google.cloud.dataproc.v1.CreateClusterRequest

CreateClusterRequest.Builder

A request to create a cluster.

Protobuf type google.cloud.dataproc.v1.CreateClusterRequest

CreateWorkflowTemplateRequest

A request to create a workflow template.

Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest

CreateWorkflowTemplateRequest.Builder

A request to create a workflow template.

Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest

DeleteAutoscalingPolicyRequest

A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.

Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest

DeleteAutoscalingPolicyRequest.Builder

A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.

Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest

DeleteBatchRequest

A request to delete a batch workload.

Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest

DeleteBatchRequest.Builder

A request to delete a batch workload.

Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest

DeleteClusterRequest

A request to delete a cluster.

Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest

DeleteClusterRequest.Builder

A request to delete a cluster.

Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest

DeleteJobRequest

A request to delete a job.

Protobuf type google.cloud.dataproc.v1.DeleteJobRequest

DeleteJobRequest.Builder

A request to delete a job.

Protobuf type google.cloud.dataproc.v1.DeleteJobRequest

DeleteWorkflowTemplateRequest

A request to delete a workflow template. Currently started workflows will remain running.

Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest

DeleteWorkflowTemplateRequest.Builder

A request to delete a workflow template. Currently started workflows will remain running.

Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest

DiagnoseClusterRequest

A request to collect cluster diagnostic information.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest

DiagnoseClusterRequest.Builder

A request to collect cluster diagnostic information.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest

DiagnoseClusterResults

The location of diagnostic output.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults

DiagnoseClusterResults.Builder

The location of diagnostic output.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults

DiskConfig

Specifies the config of disk options for a group of VM instances.

Protobuf type google.cloud.dataproc.v1.DiskConfig

DiskConfig.Builder

Specifies the config of disk options for a group of VM instances.

Protobuf type google.cloud.dataproc.v1.DiskConfig

EncryptionConfig

Encryption settings for the cluster.

Protobuf type google.cloud.dataproc.v1.EncryptionConfig

EncryptionConfig.Builder

Encryption settings for the cluster.

Protobuf type google.cloud.dataproc.v1.EncryptionConfig

EndpointConfig

Endpoint config for this cluster

Protobuf type google.cloud.dataproc.v1.EndpointConfig

EndpointConfig.Builder

Endpoint config for this cluster

Protobuf type google.cloud.dataproc.v1.EndpointConfig

EnvironmentConfig

Environment configuration for a workload.

Protobuf type google.cloud.dataproc.v1.EnvironmentConfig

EnvironmentConfig.Builder

Environment configuration for a workload.

Protobuf type google.cloud.dataproc.v1.EnvironmentConfig

ExecutionConfig

Execution configuration for a workload.

Protobuf type google.cloud.dataproc.v1.ExecutionConfig

ExecutionConfig.Builder

Execution configuration for a workload.

Protobuf type google.cloud.dataproc.v1.ExecutionConfig

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Protobuf type google.cloud.dataproc.v1.GceClusterConfig

GceClusterConfig.Builder

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Protobuf type google.cloud.dataproc.v1.GceClusterConfig

GetAutoscalingPolicyRequest

A request to fetch an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest

GetAutoscalingPolicyRequest.Builder

A request to fetch an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest

GetBatchRequest

A request to get the resource representation for a batch workload.

Protobuf type google.cloud.dataproc.v1.GetBatchRequest

GetBatchRequest.Builder

A request to get the resource representation for a batch workload.

Protobuf type google.cloud.dataproc.v1.GetBatchRequest

GetClusterRequest

Request to get the resource representation for a cluster in a project.

Protobuf type google.cloud.dataproc.v1.GetClusterRequest

GetClusterRequest.Builder

Request to get the resource representation for a cluster in a project.

Protobuf type google.cloud.dataproc.v1.GetClusterRequest

GetJobRequest

A request to get the resource representation for a job in a project.

Protobuf type google.cloud.dataproc.v1.GetJobRequest

GetJobRequest.Builder

A request to get the resource representation for a job in a project.

Protobuf type google.cloud.dataproc.v1.GetJobRequest

GetWorkflowTemplateRequest

A request to fetch a workflow template.

Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest

GetWorkflowTemplateRequest.Builder

A request to fetch a workflow template.

Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest

GkeClusterConfig

The cluster's GKE config.

Protobuf type google.cloud.dataproc.v1.GkeClusterConfig

GkeClusterConfig.Builder

The cluster's GKE config.

Protobuf type google.cloud.dataproc.v1.GkeClusterConfig

GkeNodePoolConfig

The configuration of a GKE NodePool used by a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig

GkeNodePoolConfig.Builder

The configuration of a GKE NodePool used by a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig

GkeNodePoolConfig.GkeNodeConfig

Parameters that describe cluster nodes.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig

GkeNodePoolConfig.GkeNodeConfig.Builder

Parameters that describe cluster nodes.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig

GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a NodePool.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

GkeNodePoolConfig.GkeNodePoolAcceleratorConfig.Builder

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a NodePool.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolConfig.GkeNodePoolAutoscalingConfig.Builder

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolTarget

GKE NodePools that Dataproc workloads run on.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget

GkeNodePoolTarget.Builder

GKE NodePools that Dataproc workloads run on.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

Protobuf type google.cloud.dataproc.v1.HadoopJob

HadoopJob.Builder

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

Protobuf type google.cloud.dataproc.v1.HadoopJob

HiveJob

A Dataproc job for running Apache Hive queries on YARN.

Protobuf type google.cloud.dataproc.v1.HiveJob

HiveJob.Builder

A Dataproc job for running Apache Hive queries on YARN.

Protobuf type google.cloud.dataproc.v1.HiveJob

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

Protobuf type google.cloud.dataproc.v1.IdentityConfig

IdentityConfig.Builder

Identity related configuration, including service account based secure multi-tenancy user mappings.

Protobuf type google.cloud.dataproc.v1.IdentityConfig

InstanceGroupAutoscalingPolicyConfig

Configuration for the size bounds of an instance group, including its proportional size to other groups.

Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig

InstanceGroupAutoscalingPolicyConfig.Builder

Configuration for the size bounds of an instance group, including its proportional size to other groups.

Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig

InstanceGroupConfig

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig

InstanceGroupConfig.Builder

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig

InstantiateInlineWorkflowTemplateRequest

A request to instantiate an inline workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest

InstantiateInlineWorkflowTemplateRequest.Builder

A request to instantiate an inline workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest

InstantiateWorkflowTemplateRequest

A request to instantiate a workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest

InstantiateWorkflowTemplateRequest.Builder

A request to instantiate a workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest

Job

A Dataproc job resource.

Protobuf type google.cloud.dataproc.v1.Job

Job.Builder

A Dataproc job resource.

Protobuf type google.cloud.dataproc.v1.Job

JobControllerClient

Service Description: The JobController provides methods to manage jobs.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   Job job = Job.newBuilder().build();
   Job response = jobControllerClient.submitJob(projectId, region, job);
 }

Note: close() needs to be called on the JobControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of JobControllerSettings to create(). For example:

To customize credentials:


 JobControllerSettings jobControllerSettings =
     JobControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

To customize the endpoint:


 JobControllerSettings jobControllerSettings =
     JobControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

JobControllerClient.ListJobsFixedSizeCollection

JobControllerClient.ListJobsPage

JobControllerClient.ListJobsPagedResponse

JobControllerGrpc

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerBlockingStub

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerFutureStub

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerImplBase

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerStub

The JobController provides methods to manage jobs.

JobControllerSettings

Settings class to configure an instance of JobControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of submitJob to 30 seconds:


 JobControllerSettings.Builder jobControllerSettingsBuilder = JobControllerSettings.newBuilder();
 jobControllerSettingsBuilder
     .submitJobSettings()
     .setRetrySettings(
         jobControllerSettingsBuilder
             .submitJobSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 JobControllerSettings jobControllerSettings = jobControllerSettingsBuilder.build();

JobControllerSettings.Builder

Builder for JobControllerSettings.

JobMetadata

Job Operation metadata.

Protobuf type google.cloud.dataproc.v1.JobMetadata

JobMetadata.Builder

Job Operation metadata.

Protobuf type google.cloud.dataproc.v1.JobMetadata

JobPlacement

Dataproc job config.

Protobuf type google.cloud.dataproc.v1.JobPlacement

JobPlacement.Builder

Dataproc job config.

Protobuf type google.cloud.dataproc.v1.JobPlacement

JobReference

Encapsulates the full scoping used to reference a job.

Protobuf type google.cloud.dataproc.v1.JobReference

JobReference.Builder

Encapsulates the full scoping used to reference a job.

Protobuf type google.cloud.dataproc.v1.JobReference

JobScheduling

Job scheduling options.

Protobuf type google.cloud.dataproc.v1.JobScheduling

JobScheduling.Builder

Job scheduling options.

Protobuf type google.cloud.dataproc.v1.JobScheduling

JobStatus

Dataproc job status.

Protobuf type google.cloud.dataproc.v1.JobStatus

JobStatus.Builder

Dataproc job status.

Protobuf type google.cloud.dataproc.v1.JobStatus

JobsProto

KerberosConfig

Specifies Kerberos related configuration.

Protobuf type google.cloud.dataproc.v1.KerberosConfig

KerberosConfig.Builder

Specifies Kerberos related configuration.

Protobuf type google.cloud.dataproc.v1.KerberosConfig

KubernetesClusterConfig

The configuration for running the Dataproc cluster on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig

KubernetesClusterConfig.Builder

The configuration for running the Dataproc cluster on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig

KubernetesSoftwareConfig

The software configuration for this Dataproc cluster running on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig

KubernetesSoftwareConfig.Builder

The software configuration for this Dataproc cluster running on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

Protobuf type google.cloud.dataproc.v1.LifecycleConfig

LifecycleConfig.Builder

Specifies the cluster auto-delete schedule configuration.

Protobuf type google.cloud.dataproc.v1.LifecycleConfig

ListAutoscalingPoliciesRequest

A request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest

ListAutoscalingPoliciesRequest.Builder

A request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest

ListAutoscalingPoliciesResponse

A response to a request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse

ListAutoscalingPoliciesResponse.Builder

A response to a request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse

ListBatchesRequest

A request to list batch workloads in a project.

Protobuf type google.cloud.dataproc.v1.ListBatchesRequest

ListBatchesRequest.Builder

A request to list batch workloads in a project.

Protobuf type google.cloud.dataproc.v1.ListBatchesRequest

ListBatchesResponse

A list of batch workloads.

Protobuf type google.cloud.dataproc.v1.ListBatchesResponse

ListBatchesResponse.Builder

A list of batch workloads.

Protobuf type google.cloud.dataproc.v1.ListBatchesResponse

ListClustersRequest

A request to list the clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersRequest

ListClustersRequest.Builder

A request to list the clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersRequest

ListClustersResponse

The list of all clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersResponse

ListClustersResponse.Builder

The list of all clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersResponse

ListJobsRequest

A request to list jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsRequest

ListJobsRequest.Builder

A request to list jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsRequest

ListJobsResponse

A list of jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsResponse

ListJobsResponse.Builder

A list of jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsResponse

ListWorkflowTemplatesRequest

A request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest

ListWorkflowTemplatesRequest.Builder

A request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest

ListWorkflowTemplatesResponse

A response to a request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse

ListWorkflowTemplatesResponse.Builder

A response to a request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse

LocationName

LocationName.Builder

Builder for projects/{project}/locations/{location}.

LoggingConfig

The runtime logging config of the job.

Protobuf type google.cloud.dataproc.v1.LoggingConfig

LoggingConfig.Builder

The runtime logging config of the job.

Protobuf type google.cloud.dataproc.v1.LoggingConfig

ManagedCluster

Cluster that is managed by the workflow.

Protobuf type google.cloud.dataproc.v1.ManagedCluster

ManagedCluster.Builder

Cluster that is managed by the workflow.

Protobuf type google.cloud.dataproc.v1.ManagedCluster

ManagedGroupConfig

Specifies the resources used to actively manage an instance group.

Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig

ManagedGroupConfig.Builder

Specifies the resources used to actively manage an instance group.

Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig

MetastoreConfig

Specifies a Metastore configuration.

Protobuf type google.cloud.dataproc.v1.MetastoreConfig

MetastoreConfig.Builder

Specifies a Metastore configuration.

Protobuf type google.cloud.dataproc.v1.MetastoreConfig

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups.

Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity

NodeGroupAffinity.Builder

Node Group Affinity for clusters using sole-tenant node groups.

Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

Protobuf type google.cloud.dataproc.v1.NodeInitializationAction

NodeInitializationAction.Builder

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

Protobuf type google.cloud.dataproc.v1.NodeInitializationAction

OperationsProto

OrderedJob

A job executed by the workflow.

Protobuf type google.cloud.dataproc.v1.OrderedJob

OrderedJob.Builder

A job executed by the workflow.

Protobuf type google.cloud.dataproc.v1.OrderedJob

ParameterValidation

Configuration for parameter validation.

Protobuf type google.cloud.dataproc.v1.ParameterValidation

ParameterValidation.Builder

Configuration for parameter validation.

Protobuf type google.cloud.dataproc.v1.ParameterValidation

PeripheralsConfig

Auxiliary services configuration for a workload.

Protobuf type google.cloud.dataproc.v1.PeripheralsConfig

PeripheralsConfig.Builder

Auxiliary services configuration for a workload.

Protobuf type google.cloud.dataproc.v1.PeripheralsConfig

PigJob

A Dataproc job for running Apache Pig queries on YARN.

Protobuf type google.cloud.dataproc.v1.PigJob

PigJob.Builder

A Dataproc job for running Apache Pig queries on YARN.

Protobuf type google.cloud.dataproc.v1.PigJob

PrestoJob

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

Protobuf type google.cloud.dataproc.v1.PrestoJob

PrestoJob.Builder

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

Protobuf type google.cloud.dataproc.v1.PrestoJob

PySparkBatch

A configuration for running an Apache PySpark batch workload.

Protobuf type google.cloud.dataproc.v1.PySparkBatch

PySparkBatch.Builder

A configuration for running an Apache PySpark batch workload.

Protobuf type google.cloud.dataproc.v1.PySparkBatch

PySparkJob

A Dataproc job for running Apache PySpark applications on YARN.

Protobuf type google.cloud.dataproc.v1.PySparkJob

PySparkJob.Builder

A Dataproc job for running Apache PySpark applications on YARN.

Protobuf type google.cloud.dataproc.v1.PySparkJob

QueryList

A list of queries to run on a cluster.

Protobuf type google.cloud.dataproc.v1.QueryList

QueryList.Builder

A list of queries to run on a cluster.

Protobuf type google.cloud.dataproc.v1.QueryList

RegexValidation

Validation based on regular expressions.

Protobuf type google.cloud.dataproc.v1.RegexValidation

RegexValidation.Builder

Validation based on regular expressions.

Protobuf type google.cloud.dataproc.v1.RegexValidation

RegionName

RegionName.Builder

Builder for projects/{project}/regions/{region}.

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

Protobuf type google.cloud.dataproc.v1.ReservationAffinity

ReservationAffinity.Builder

Reservation Affinity for consuming Zonal reservation.

Protobuf type google.cloud.dataproc.v1.ReservationAffinity

RuntimeConfig

Runtime configuration for a workload.

Protobuf type google.cloud.dataproc.v1.RuntimeConfig

RuntimeConfig.Builder

Runtime configuration for a workload.

Protobuf type google.cloud.dataproc.v1.RuntimeConfig

RuntimeInfo

Runtime information about workload execution.

Protobuf type google.cloud.dataproc.v1.RuntimeInfo

RuntimeInfo.Builder

Runtime information about workload execution.

Protobuf type google.cloud.dataproc.v1.RuntimeInfo

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

Protobuf type google.cloud.dataproc.v1.SecurityConfig

SecurityConfig.Builder

Security related configuration, including encryption, Kerberos, etc.

Protobuf type google.cloud.dataproc.v1.SecurityConfig

SharedProto

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig

ShieldedInstanceConfig.Builder

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig

SoftwareConfig

Specifies the selection and config of software inside the cluster.

Protobuf type google.cloud.dataproc.v1.SoftwareConfig

SoftwareConfig.Builder

Specifies the selection and config of software inside the cluster.

Protobuf type google.cloud.dataproc.v1.SoftwareConfig

SparkBatch

A configuration for running an Apache Spark batch workload.

Protobuf type google.cloud.dataproc.v1.SparkBatch

SparkBatch.Builder

A configuration for running an Apache Spark batch workload.

Protobuf type google.cloud.dataproc.v1.SparkBatch

SparkHistoryServerConfig

Spark History Server configuration for the workload.

Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig

SparkHistoryServerConfig.Builder

Spark History Server configuration for the workload.

Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig

SparkJob

A Dataproc job for running Apache Spark applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkJob

SparkJob.Builder

A Dataproc job for running Apache Spark applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkJob

SparkRBatch

A configuration for running an Apache SparkR batch workload.

Protobuf type google.cloud.dataproc.v1.SparkRBatch

SparkRBatch.Builder

A configuration for running an Apache SparkR batch workload.

Protobuf type google.cloud.dataproc.v1.SparkRBatch

SparkRJob

A Dataproc job for running Apache SparkR applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkRJob

SparkRJob.Builder

A Dataproc job for running Apache SparkR applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkRJob

SparkSqlBatch

A configuration for running Apache Spark SQL queries as a batch workload.

Protobuf type google.cloud.dataproc.v1.SparkSqlBatch

SparkSqlBatch.Builder

A configuration for running Apache Spark SQL queries as a batch workload.

Protobuf type google.cloud.dataproc.v1.SparkSqlBatch

SparkSqlJob

A Dataproc job for running Apache Spark SQL queries.

Protobuf type google.cloud.dataproc.v1.SparkSqlJob

SparkSqlJob.Builder

A Dataproc job for running Apache Spark SQL queries.

Protobuf type google.cloud.dataproc.v1.SparkSqlJob

StartClusterRequest

A request to start a cluster.

Protobuf type google.cloud.dataproc.v1.StartClusterRequest

StartClusterRequest.Builder

A request to start a cluster.

Protobuf type google.cloud.dataproc.v1.StartClusterRequest

StopClusterRequest

A request to stop a cluster.

Protobuf type google.cloud.dataproc.v1.StopClusterRequest

StopClusterRequest.Builder

A request to stop a cluster.

Protobuf type google.cloud.dataproc.v1.StopClusterRequest

SubmitJobRequest

A request to submit a job.

Protobuf type google.cloud.dataproc.v1.SubmitJobRequest

SubmitJobRequest.Builder

A request to submit a job.

Protobuf type google.cloud.dataproc.v1.SubmitJobRequest

TemplateParameter

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

Labels
File uris
Job properties
Job arguments
Script variables
Main class (in HadoopJob and SparkJob)
Zone (in ClusterSelector)

Protobuf type google.cloud.dataproc.v1.TemplateParameter

TemplateParameter.Builder

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

Labels
File uris
Job properties
Job arguments
Script variables
Main class (in HadoopJob and SparkJob)
Zone (in ClusterSelector)

Protobuf type google.cloud.dataproc.v1.TemplateParameter

UpdateAutoscalingPolicyRequest

A request to update an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest

UpdateAutoscalingPolicyRequest.Builder

A request to update an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest

UpdateClusterRequest

A request to update a cluster.

Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest

UpdateClusterRequest.Builder

A request to update a cluster.

Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest

UpdateJobRequest

A request to update a job.

Protobuf type google.cloud.dataproc.v1.UpdateJobRequest

UpdateJobRequest.Builder

A request to update a job.

Protobuf type google.cloud.dataproc.v1.UpdateJobRequest

UpdateWorkflowTemplateRequest

A request to update a workflow template.

Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest

UpdateWorkflowTemplateRequest.Builder

A request to update a workflow template.

Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest

ValueValidation

Validation based on a list of allowed values.

Protobuf type google.cloud.dataproc.v1.ValueValidation

ValueValidation.Builder

Validation based on a list of allowed values.

Protobuf type google.cloud.dataproc.v1.ValueValidation

VirtualClusterConfig

Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig

VirtualClusterConfig.Builder

Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig

WorkflowGraph

The workflow graph.

Protobuf type google.cloud.dataproc.v1.WorkflowGraph

WorkflowGraph.Builder

The workflow graph.

Protobuf type google.cloud.dataproc.v1.WorkflowGraph

WorkflowMetadata

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowMetadata

WorkflowMetadata.Builder

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowMetadata

WorkflowNode

The workflow node.

Protobuf type google.cloud.dataproc.v1.WorkflowNode

WorkflowNode.Builder

The workflow node.

Protobuf type google.cloud.dataproc.v1.WorkflowNode

WorkflowTemplate

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplate

WorkflowTemplate.Builder

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplate

WorkflowTemplateName

WorkflowTemplateName.Builder

Builder for projects/{project}/regions/{region}/workflowTemplates/{workflow_template}.

WorkflowTemplateName.ProjectLocationWorkflowTemplateBuilder

Builder for projects/{project}/locations/{location}/workflowTemplates/{workflow_template}.

WorkflowTemplatePlacement

Specifies workflow execution target. Either managed_cluster or cluster_selector is required.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement

WorkflowTemplatePlacement.Builder

Specifies workflow execution target. Either managed_cluster or cluster_selector is required.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement

WorkflowTemplateServiceClient

Service Description: The API interface for managing Workflow Templates in the Dataproc API.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
   WorkflowTemplate response =
       workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
 }

Note: close() needs to be called on the WorkflowTemplateServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of WorkflowTemplateServiceSettings to create(). For example:

To customize credentials:


 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     WorkflowTemplateServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);

To customize the endpoint:


 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     WorkflowTemplateServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

WorkflowTemplateServiceClient.ListWorkflowTemplatesFixedSizeCollection

WorkflowTemplateServiceClient.ListWorkflowTemplatesPage

WorkflowTemplateServiceClient.ListWorkflowTemplatesPagedResponse

WorkflowTemplateServiceGrpc

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceBlockingStub

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceFutureStub

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceImplBase

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceStub

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceSettings

Settings class to configure an instance of WorkflowTemplateServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createWorkflowTemplate to 30 seconds:


 WorkflowTemplateServiceSettings.Builder workflowTemplateServiceSettingsBuilder =
     WorkflowTemplateServiceSettings.newBuilder();
 workflowTemplateServiceSettingsBuilder
     .createWorkflowTemplateSettings()
     .setRetrySettings(
         workflowTemplateServiceSettingsBuilder
             .createWorkflowTemplateSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     workflowTemplateServiceSettingsBuilder.build();

WorkflowTemplateServiceSettings.Builder

Builder for WorkflowTemplateServiceSettings.

WorkflowTemplatesProto

YarnApplication

A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>. Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.YarnApplication

YarnApplication.Builder

Protobuf type google.cloud.dataproc.v1.YarnApplication

Interfaces

AcceleratorConfigOrBuilder

AutoscalingConfigOrBuilder

AutoscalingPolicyOrBuilder

AuxiliaryServicesConfigOrBuilder

BasicAutoscalingAlgorithmOrBuilder

BasicYarnAutoscalingConfigOrBuilder

Batch.StateHistoryOrBuilder

BatchOperationMetadataOrBuilder

BatchOrBuilder

CancelJobRequestOrBuilder

ClusterConfigOrBuilder

ClusterMetricsOrBuilder

ClusterOperationMetadataOrBuilder

ClusterOperationOrBuilder

ClusterOperationStatusOrBuilder

ClusterOrBuilder

ClusterSelectorOrBuilder

ClusterStatusOrBuilder

ConfidentialInstanceConfigOrBuilder

CreateAutoscalingPolicyRequestOrBuilder

CreateBatchRequestOrBuilder

CreateClusterRequestOrBuilder

CreateWorkflowTemplateRequestOrBuilder

DeleteAutoscalingPolicyRequestOrBuilder

DeleteBatchRequestOrBuilder

DeleteClusterRequestOrBuilder

DeleteJobRequestOrBuilder

DeleteWorkflowTemplateRequestOrBuilder

DiagnoseClusterRequestOrBuilder

DiagnoseClusterResultsOrBuilder

DiskConfigOrBuilder

EncryptionConfigOrBuilder

EndpointConfigOrBuilder

EnvironmentConfigOrBuilder

ExecutionConfigOrBuilder

GceClusterConfigOrBuilder

GetAutoscalingPolicyRequestOrBuilder

GetBatchRequestOrBuilder

GetClusterRequestOrBuilder

GetJobRequestOrBuilder

GetWorkflowTemplateRequestOrBuilder

GkeClusterConfigOrBuilder

GkeNodePoolConfig.GkeNodeConfigOrBuilder

GkeNodePoolConfig.GkeNodePoolAcceleratorConfigOrBuilder

GkeNodePoolConfig.GkeNodePoolAutoscalingConfigOrBuilder

GkeNodePoolConfigOrBuilder

GkeNodePoolTargetOrBuilder

HadoopJobOrBuilder

HiveJobOrBuilder

IdentityConfigOrBuilder

InstanceGroupAutoscalingPolicyConfigOrBuilder

InstanceGroupConfigOrBuilder

InstantiateInlineWorkflowTemplateRequestOrBuilder

InstantiateWorkflowTemplateRequestOrBuilder

JobMetadataOrBuilder

JobOrBuilder

JobPlacementOrBuilder

JobReferenceOrBuilder

JobSchedulingOrBuilder

JobStatusOrBuilder

KerberosConfigOrBuilder

KubernetesClusterConfigOrBuilder

KubernetesSoftwareConfigOrBuilder

LifecycleConfigOrBuilder

ListAutoscalingPoliciesRequestOrBuilder

ListAutoscalingPoliciesResponseOrBuilder

ListBatchesRequestOrBuilder

ListBatchesResponseOrBuilder

ListClustersRequestOrBuilder

ListClustersResponseOrBuilder

ListJobsRequestOrBuilder

ListJobsResponseOrBuilder

ListWorkflowTemplatesRequestOrBuilder

ListWorkflowTemplatesResponseOrBuilder

LoggingConfigOrBuilder

ManagedClusterOrBuilder

ManagedGroupConfigOrBuilder

MetastoreConfigOrBuilder

NodeGroupAffinityOrBuilder

NodeInitializationActionOrBuilder

OrderedJobOrBuilder

ParameterValidationOrBuilder

PeripheralsConfigOrBuilder

PigJobOrBuilder

PrestoJobOrBuilder

PySparkBatchOrBuilder

PySparkJobOrBuilder

QueryListOrBuilder

RegexValidationOrBuilder

ReservationAffinityOrBuilder

RuntimeConfigOrBuilder

RuntimeInfoOrBuilder

SecurityConfigOrBuilder

ShieldedInstanceConfigOrBuilder

SoftwareConfigOrBuilder

SparkBatchOrBuilder

SparkHistoryServerConfigOrBuilder

SparkJobOrBuilder

SparkRBatchOrBuilder

SparkRJobOrBuilder

SparkSqlBatchOrBuilder

SparkSqlJobOrBuilder

StartClusterRequestOrBuilder

StopClusterRequestOrBuilder

SubmitJobRequestOrBuilder

TemplateParameterOrBuilder

UpdateAutoscalingPolicyRequestOrBuilder

UpdateClusterRequestOrBuilder

UpdateJobRequestOrBuilder

UpdateWorkflowTemplateRequestOrBuilder

ValueValidationOrBuilder

VirtualClusterConfigOrBuilder

WorkflowGraphOrBuilder

WorkflowMetadataOrBuilder

WorkflowNodeOrBuilder

WorkflowTemplateOrBuilder

WorkflowTemplatePlacementOrBuilder

YarnApplicationOrBuilder

Enums

AutoscalingPolicy.AlgorithmCase

Batch.BatchConfigCase

Batch.State

The batch state.

Protobuf enum google.cloud.dataproc.v1.Batch.State

BatchOperationMetadata.BatchOperationType

Operation type for Batch resources

Protobuf enum google.cloud.dataproc.v1.BatchOperationMetadata.BatchOperationType

ClusterOperationStatus.State

The operation state.

Protobuf enum google.cloud.dataproc.v1.ClusterOperationStatus.State

ClusterStatus.State

The cluster state.

Protobuf enum google.cloud.dataproc.v1.ClusterStatus.State

ClusterStatus.Substate

The cluster substate.

Protobuf enum google.cloud.dataproc.v1.ClusterStatus.Substate

Component

Cluster components that can be activated.

Protobuf enum google.cloud.dataproc.v1.Component

ExecutionConfig.NetworkCase

FailureAction

Actions in response to failure of a resource associated with a cluster.

Protobuf enum google.cloud.dataproc.v1.FailureAction

GceClusterConfig.PrivateIpv6GoogleAccess

PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes can communicate with Google Services through gRPC over IPv6. These values are directly mapped to corresponding values in the Compute Engine Instance fields.

Protobuf enum google.cloud.dataproc.v1.GceClusterConfig.PrivateIpv6GoogleAccess

GkeNodePoolTarget.Role

Role specifies whose tasks will run on the NodePool. The roles can be specific to workloads. Exactly one GkeNodePoolTarget within the VirtualCluster must have 'default' role, which is used to run all workloads that are not associated with a NodePool.

Protobuf enum google.cloud.dataproc.v1.GkeNodePoolTarget.Role