Package com.google.cloud.dataproc.v1 (4.21.0)

A client to Cloud Dataproc API

The interfaces provided are listed below, along with usage samples.

AutoscalingPolicyServiceClient

Service Description: The API interface for managing autoscaling policies in the Dataproc API.

Sample for AutoscalingPolicyServiceClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
   AutoscalingPolicy response =
       autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
 }

BatchControllerClient

Service Description: The BatchController provides methods to manage batch workloads.

Sample for BatchControllerClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
   BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
   Batch response = batchControllerClient.getBatch(name);
 }

ClusterControllerClient

Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

Sample for ClusterControllerClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   String clusterName = "clusterName-1141738587";
   Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
 }

JobControllerClient

Service Description: The JobController provides methods to manage jobs.

Sample for JobControllerClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   Job job = Job.newBuilder().build();
   Job response = jobControllerClient.submitJob(projectId, region, job);
 }

NodeGroupControllerClient

Service Description: The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

Sample for NodeGroupControllerClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (NodeGroupControllerClient nodeGroupControllerClient = NodeGroupControllerClient.create()) {
   NodeGroupName name = NodeGroupName.of("[PROJECT]", "[REGION]", "[CLUSTER]", "[NODE_GROUP]");
   NodeGroup response = nodeGroupControllerClient.getNodeGroup(name);
 }

WorkflowTemplateServiceClient

Service Description: The API interface for managing Workflow Templates in the Dataproc API.

Sample for WorkflowTemplateServiceClient:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
   WorkflowTemplate response =
       workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
 }

Classes

AcceleratorConfig

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

Protobuf type google.cloud.dataproc.v1.AcceleratorConfig

AcceleratorConfig.Builder

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.

Protobuf type google.cloud.dataproc.v1.AcceleratorConfig

AutoscalingConfig

Autoscaling Policy config associated with the cluster.

Protobuf type google.cloud.dataproc.v1.AutoscalingConfig

AutoscalingConfig.Builder

Autoscaling Policy config associated with the cluster.

Protobuf type google.cloud.dataproc.v1.AutoscalingConfig

AutoscalingPolicy

Describes an autoscaling policy for Dataproc cluster autoscaler.

Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy

AutoscalingPolicy.Builder

Describes an autoscaling policy for Dataproc cluster autoscaler.

Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy

AutoscalingPolicyName.Builder

Builder for projects/{project}/locations/{location}/autoscalingPolicies/{autoscaling_policy}.

AutoscalingPolicyName.ProjectRegionAutoscalingPolicyBuilder

Builder for projects/{project}/regions/{region}/autoscalingPolicies/{autoscaling_policy}.

AutoscalingPolicyServiceClient

Service Description: The API interface for managing autoscaling policies in the Dataproc API.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
   AutoscalingPolicy response =
       autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
 }

Note: close() needs to be called on the AutoscalingPolicyServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.

This class can be customized by passing in a custom instance of AutoscalingPolicyServiceSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     AutoscalingPolicyServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     AutoscalingPolicyServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     AutoscalingPolicyServiceSettings.newHttpJsonBuilder().build();
 AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
     AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesFixedSizeCollection

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPage

AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPagedResponse

AutoscalingPolicyServiceGrpc

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceBlockingStub

A stub to allow clients to do synchronous rpc calls to service AutoscalingPolicyService.

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service AutoscalingPolicyService.

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceImplBase

Base class for the server implementation of the service AutoscalingPolicyService.

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceStub

A stub to allow clients to do asynchronous rpc calls to service AutoscalingPolicyService.

The API interface for managing autoscaling policies in the Dataproc API.

AutoscalingPolicyServiceSettings

Settings class to configure an instance of AutoscalingPolicyServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createAutoscalingPolicy to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 AutoscalingPolicyServiceSettings.Builder autoscalingPolicyServiceSettingsBuilder =
     AutoscalingPolicyServiceSettings.newBuilder();
 autoscalingPolicyServiceSettingsBuilder
     .createAutoscalingPolicySettings()
     .setRetrySettings(
         autoscalingPolicyServiceSettingsBuilder
             .createAutoscalingPolicySettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
     autoscalingPolicyServiceSettingsBuilder.build();

AutoscalingPolicyServiceSettings.Builder

Builder for AutoscalingPolicyServiceSettings.

AuxiliaryNodeGroup

Node group identification and configuration information.

Protobuf type google.cloud.dataproc.v1.AuxiliaryNodeGroup

AuxiliaryNodeGroup.Builder

Node group identification and configuration information.

Protobuf type google.cloud.dataproc.v1.AuxiliaryNodeGroup

AuxiliaryServicesConfig

Auxiliary services configuration for a Cluster.

Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig

AuxiliaryServicesConfig.Builder

Auxiliary services configuration for a Cluster.

Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig

BasicAutoscalingAlgorithm

Basic algorithm for autoscaling.

Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm

BasicAutoscalingAlgorithm.Builder

Basic algorithm for autoscaling.

Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm

BasicYarnAutoscalingConfig

Basic autoscaling configurations for YARN.

Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig

BasicYarnAutoscalingConfig.Builder

Basic autoscaling configurations for YARN.

Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig

Batch

A representation of a batch workload in the service.

Protobuf type google.cloud.dataproc.v1.Batch

Batch.Builder

A representation of a batch workload in the service.

Protobuf type google.cloud.dataproc.v1.Batch

Batch.StateHistory

Historical state information.

Protobuf type google.cloud.dataproc.v1.Batch.StateHistory

Batch.StateHistory.Builder

Historical state information.

Protobuf type google.cloud.dataproc.v1.Batch.StateHistory

BatchControllerClient

Service Description: The BatchController provides methods to manage batch workloads.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
   BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
   Batch response = batchControllerClient.getBatch(name);
 }

Note: close() needs to be called on the BatchControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of BatchControllerSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 BatchControllerSettings batchControllerSettings =
     BatchControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 BatchControllerClient batchControllerClient =
     BatchControllerClient.create(batchControllerSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 BatchControllerSettings batchControllerSettings =
     BatchControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 BatchControllerClient batchControllerClient =
     BatchControllerClient.create(batchControllerSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 BatchControllerSettings batchControllerSettings =
     BatchControllerSettings.newHttpJsonBuilder().build();
 BatchControllerClient batchControllerClient =
     BatchControllerClient.create(batchControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

BatchControllerClient.ListBatchesFixedSizeCollection

BatchControllerClient.ListBatchesPage

BatchControllerClient.ListBatchesPagedResponse

BatchControllerGrpc

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerBlockingStub

A stub to allow clients to do synchronous rpc calls to service BatchController.

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service BatchController.

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerImplBase

Base class for the server implementation of the service BatchController.

The BatchController provides methods to manage batch workloads.

BatchControllerGrpc.BatchControllerStub

A stub to allow clients to do asynchronous rpc calls to service BatchController.

The BatchController provides methods to manage batch workloads.

BatchControllerSettings

Settings class to configure an instance of BatchControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getBatch to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 BatchControllerSettings.Builder batchControllerSettingsBuilder =
     BatchControllerSettings.newBuilder();
 batchControllerSettingsBuilder
     .getBatchSettings()
     .setRetrySettings(
         batchControllerSettingsBuilder
             .getBatchSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 BatchControllerSettings batchControllerSettings = batchControllerSettingsBuilder.build();

BatchControllerSettings.Builder

Builder for BatchControllerSettings.

BatchName

BatchName.Builder

Builder for projects/{project}/locations/{location}/batches/{batch}.

BatchOperationMetadata

Metadata describing the Batch operation.

Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata

BatchOperationMetadata.Builder

Metadata describing the Batch operation.

Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata

BatchesProto

CancelJobRequest

A request to cancel a job.

Protobuf type google.cloud.dataproc.v1.CancelJobRequest

CancelJobRequest.Builder

A request to cancel a job.

Protobuf type google.cloud.dataproc.v1.CancelJobRequest

Cluster

Describes the identifying information, config, and status of a Dataproc cluster

Protobuf type google.cloud.dataproc.v1.Cluster

Cluster.Builder

Describes the identifying information, config, and status of a Dataproc cluster

Protobuf type google.cloud.dataproc.v1.Cluster

ClusterConfig

The cluster config.

Protobuf type google.cloud.dataproc.v1.ClusterConfig

ClusterConfig.Builder

The cluster config.

Protobuf type google.cloud.dataproc.v1.ClusterConfig

ClusterControllerClient

Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   String clusterName = "clusterName-1141738587";
   Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
 }

Note: close() needs to be called on the ClusterControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of ClusterControllerSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 ClusterControllerSettings clusterControllerSettings =
     ClusterControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 ClusterControllerClient clusterControllerClient =
     ClusterControllerClient.create(clusterControllerSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 ClusterControllerSettings clusterControllerSettings =
     ClusterControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 ClusterControllerClient clusterControllerClient =
     ClusterControllerClient.create(clusterControllerSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 ClusterControllerSettings clusterControllerSettings =
     ClusterControllerSettings.newHttpJsonBuilder().build();
 ClusterControllerClient clusterControllerClient =
     ClusterControllerClient.create(clusterControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

ClusterControllerClient.ListClustersFixedSizeCollection

ClusterControllerClient.ListClustersPage

ClusterControllerClient.ListClustersPagedResponse

ClusterControllerGrpc

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerBlockingStub

A stub to allow clients to do synchronous rpc calls to service ClusterController.

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service ClusterController.

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerImplBase

Base class for the server implementation of the service ClusterController.

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerGrpc.ClusterControllerStub

A stub to allow clients to do asynchronous rpc calls to service ClusterController.

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterControllerSettings

Settings class to configure an instance of ClusterControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getCluster to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 ClusterControllerSettings.Builder clusterControllerSettingsBuilder =
     ClusterControllerSettings.newBuilder();
 clusterControllerSettingsBuilder
     .getClusterSettings()
     .setRetrySettings(
         clusterControllerSettingsBuilder
             .getClusterSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 ClusterControllerSettings clusterControllerSettings = clusterControllerSettingsBuilder.build();

ClusterControllerSettings.Builder

Builder for ClusterControllerSettings.

ClusterMetrics

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.ClusterMetrics

ClusterMetrics.Builder

Contains cluster daemon metrics, such as HDFS and YARN stats.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.ClusterMetrics

ClusterOperation

The cluster operation triggered by a workflow.

Protobuf type google.cloud.dataproc.v1.ClusterOperation

ClusterOperation.Builder

The cluster operation triggered by a workflow.

Protobuf type google.cloud.dataproc.v1.ClusterOperation

ClusterOperationMetadata

Metadata describing the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata

ClusterOperationMetadata.Builder

Metadata describing the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata

ClusterOperationStatus

The status of the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus

ClusterOperationStatus.Builder

The status of the operation.

Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus

ClusterRegionName

ClusterRegionName.Builder

Builder for projects/{project}/regions/{region}/clusters/{cluster}.

ClusterSelector

A selector that chooses target cluster for jobs based on metadata.

Protobuf type google.cloud.dataproc.v1.ClusterSelector

ClusterSelector.Builder

A selector that chooses target cluster for jobs based on metadata.

Protobuf type google.cloud.dataproc.v1.ClusterSelector

ClusterStatus

The status of a cluster and its instances.

Protobuf type google.cloud.dataproc.v1.ClusterStatus

ClusterStatus.Builder

The status of a cluster and its instances.

Protobuf type google.cloud.dataproc.v1.ClusterStatus

ClustersProto

ConfidentialInstanceConfig

Confidential Instance Config for clusters using Confidential VMs

Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig

ConfidentialInstanceConfig.Builder

Confidential Instance Config for clusters using Confidential VMs

Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig

CreateAutoscalingPolicyRequest

A request to create an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest

CreateAutoscalingPolicyRequest.Builder

A request to create an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest

CreateBatchRequest

A request to create a batch workload.

Protobuf type google.cloud.dataproc.v1.CreateBatchRequest

CreateBatchRequest.Builder

A request to create a batch workload.

Protobuf type google.cloud.dataproc.v1.CreateBatchRequest

CreateClusterRequest

A request to create a cluster.

Protobuf type google.cloud.dataproc.v1.CreateClusterRequest

CreateClusterRequest.Builder

A request to create a cluster.

Protobuf type google.cloud.dataproc.v1.CreateClusterRequest

CreateNodeGroupRequest

A request to create a node group.

Protobuf type google.cloud.dataproc.v1.CreateNodeGroupRequest

CreateNodeGroupRequest.Builder

A request to create a node group.

Protobuf type google.cloud.dataproc.v1.CreateNodeGroupRequest

CreateWorkflowTemplateRequest

A request to create a workflow template.

Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest

CreateWorkflowTemplateRequest.Builder

A request to create a workflow template.

Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest

DataprocMetricConfig

Dataproc metric config.

Protobuf type google.cloud.dataproc.v1.DataprocMetricConfig

DataprocMetricConfig.Builder

Dataproc metric config.

Protobuf type google.cloud.dataproc.v1.DataprocMetricConfig

DataprocMetricConfig.Metric

A Dataproc custom metric.

Protobuf type google.cloud.dataproc.v1.DataprocMetricConfig.Metric

DataprocMetricConfig.Metric.Builder

A Dataproc custom metric.

Protobuf type google.cloud.dataproc.v1.DataprocMetricConfig.Metric

DeleteAutoscalingPolicyRequest

A request to delete an autoscaling policy.

Autoscaling policies in use by one or more clusters will not be deleted.

Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest

DeleteAutoscalingPolicyRequest.Builder

A request to delete an autoscaling policy.

Autoscaling policies in use by one or more clusters will not be deleted.

Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest

DeleteBatchRequest

A request to delete a batch workload.

Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest

DeleteBatchRequest.Builder

A request to delete a batch workload.

Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest

DeleteClusterRequest

A request to delete a cluster.

Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest

DeleteClusterRequest.Builder

A request to delete a cluster.

Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest

DeleteJobRequest

A request to delete a job.

Protobuf type google.cloud.dataproc.v1.DeleteJobRequest

DeleteJobRequest.Builder

A request to delete a job.

Protobuf type google.cloud.dataproc.v1.DeleteJobRequest

DeleteWorkflowTemplateRequest

A request to delete a workflow template.

Currently started workflows will remain running.

Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest

DeleteWorkflowTemplateRequest.Builder

A request to delete a workflow template.

Currently started workflows will remain running.

Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest

DiagnoseClusterRequest

A request to collect cluster diagnostic information.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest

DiagnoseClusterRequest.Builder

A request to collect cluster diagnostic information.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest

DiagnoseClusterResults

The location of diagnostic output.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults

DiagnoseClusterResults.Builder

The location of diagnostic output.

Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults

DiskConfig

Specifies the config of disk options for a group of VM instances.

Protobuf type google.cloud.dataproc.v1.DiskConfig

DiskConfig.Builder

Specifies the config of disk options for a group of VM instances.

Protobuf type google.cloud.dataproc.v1.DiskConfig

DriverSchedulingConfig

Driver scheduling configuration.

Protobuf type google.cloud.dataproc.v1.DriverSchedulingConfig

DriverSchedulingConfig.Builder

Driver scheduling configuration.

Protobuf type google.cloud.dataproc.v1.DriverSchedulingConfig

EncryptionConfig

Encryption settings for the cluster.

Protobuf type google.cloud.dataproc.v1.EncryptionConfig

EncryptionConfig.Builder

Encryption settings for the cluster.

Protobuf type google.cloud.dataproc.v1.EncryptionConfig

EndpointConfig

Endpoint config for this cluster

Protobuf type google.cloud.dataproc.v1.EndpointConfig

EndpointConfig.Builder

Endpoint config for this cluster

Protobuf type google.cloud.dataproc.v1.EndpointConfig

EnvironmentConfig

Environment configuration for a workload.

Protobuf type google.cloud.dataproc.v1.EnvironmentConfig

EnvironmentConfig.Builder

Environment configuration for a workload.

Protobuf type google.cloud.dataproc.v1.EnvironmentConfig

ExecutionConfig

Execution configuration for a workload.

Protobuf type google.cloud.dataproc.v1.ExecutionConfig

ExecutionConfig.Builder

Execution configuration for a workload.

Protobuf type google.cloud.dataproc.v1.ExecutionConfig

GceClusterConfig

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Protobuf type google.cloud.dataproc.v1.GceClusterConfig

GceClusterConfig.Builder

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Protobuf type google.cloud.dataproc.v1.GceClusterConfig

GetAutoscalingPolicyRequest

A request to fetch an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest

GetAutoscalingPolicyRequest.Builder

A request to fetch an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest

GetBatchRequest

A request to get the resource representation for a batch workload.

Protobuf type google.cloud.dataproc.v1.GetBatchRequest

GetBatchRequest.Builder

A request to get the resource representation for a batch workload.

Protobuf type google.cloud.dataproc.v1.GetBatchRequest

GetClusterRequest

Request to get the resource representation for a cluster in a project.

Protobuf type google.cloud.dataproc.v1.GetClusterRequest

GetClusterRequest.Builder

Request to get the resource representation for a cluster in a project.

Protobuf type google.cloud.dataproc.v1.GetClusterRequest

GetJobRequest

A request to get the resource representation for a job in a project.

Protobuf type google.cloud.dataproc.v1.GetJobRequest

GetJobRequest.Builder

A request to get the resource representation for a job in a project.

Protobuf type google.cloud.dataproc.v1.GetJobRequest

GetNodeGroupRequest

A request to get a node group .

Protobuf type google.cloud.dataproc.v1.GetNodeGroupRequest

GetNodeGroupRequest.Builder

A request to get a node group .

Protobuf type google.cloud.dataproc.v1.GetNodeGroupRequest

GetWorkflowTemplateRequest

A request to fetch a workflow template.

Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest

GetWorkflowTemplateRequest.Builder

A request to fetch a workflow template.

Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest

GkeClusterConfig

The cluster's GKE config.

Protobuf type google.cloud.dataproc.v1.GkeClusterConfig

GkeClusterConfig.Builder

The cluster's GKE config.

Protobuf type google.cloud.dataproc.v1.GkeClusterConfig

GkeNodePoolConfig

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig

GkeNodePoolConfig.Builder

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig

GkeNodePoolConfig.GkeNodeConfig

Parameters that describe cluster nodes.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig

GkeNodePoolConfig.GkeNodeConfig.Builder

Parameters that describe cluster nodes.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig

GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

GkeNodePoolConfig.GkeNodePoolAcceleratorConfig.Builder

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig

GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolConfig.GkeNodePoolAutoscalingConfig.Builder

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig

GkeNodePoolTarget

GKE node pools that Dataproc workloads run on.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget

GkeNodePoolTarget.Builder

GKE node pools that Dataproc workloads run on.

Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget

HadoopJob

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

Protobuf type google.cloud.dataproc.v1.HadoopJob

HadoopJob.Builder

A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.

Protobuf type google.cloud.dataproc.v1.HadoopJob

HiveJob

A Dataproc job for running Apache Hive queries on YARN.

Protobuf type google.cloud.dataproc.v1.HiveJob

HiveJob.Builder

A Dataproc job for running Apache Hive queries on YARN.

Protobuf type google.cloud.dataproc.v1.HiveJob

IdentityConfig

Identity related configuration, including service account based secure multi-tenancy user mappings.

Protobuf type google.cloud.dataproc.v1.IdentityConfig

IdentityConfig.Builder

Identity related configuration, including service account based secure multi-tenancy user mappings.

Protobuf type google.cloud.dataproc.v1.IdentityConfig

InstanceFlexibilityPolicy

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy

InstanceFlexibilityPolicy.Builder

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy

InstanceFlexibilityPolicy.InstanceSelection

Defines machines types and a rank to which the machines types belong.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy.InstanceSelection

InstanceFlexibilityPolicy.InstanceSelection.Builder

Defines machines types and a rank to which the machines types belong.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy.InstanceSelection

InstanceFlexibilityPolicy.InstanceSelectionResult

Defines a mapping from machine types to the number of VMs that are created with each machine type.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy.InstanceSelectionResult

InstanceFlexibilityPolicy.InstanceSelectionResult.Builder

Defines a mapping from machine types to the number of VMs that are created with each machine type.

Protobuf type google.cloud.dataproc.v1.InstanceFlexibilityPolicy.InstanceSelectionResult

InstanceGroupAutoscalingPolicyConfig

Configuration for the size bounds of an instance group, including its proportional size to other groups.

Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig

InstanceGroupAutoscalingPolicyConfig.Builder

Configuration for the size bounds of an instance group, including its proportional size to other groups.

Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig

InstanceGroupConfig

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig

InstanceGroupConfig.Builder

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig

InstanceReference

A reference to a Compute Engine instance.

Protobuf type google.cloud.dataproc.v1.InstanceReference

InstanceReference.Builder

A reference to a Compute Engine instance.

Protobuf type google.cloud.dataproc.v1.InstanceReference

InstantiateInlineWorkflowTemplateRequest

A request to instantiate an inline workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest

InstantiateInlineWorkflowTemplateRequest.Builder

A request to instantiate an inline workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest

InstantiateWorkflowTemplateRequest

A request to instantiate a workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest

InstantiateWorkflowTemplateRequest.Builder

A request to instantiate a workflow template.

Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest

Job

A Dataproc job resource.

Protobuf type google.cloud.dataproc.v1.Job

Job.Builder

A Dataproc job resource.

Protobuf type google.cloud.dataproc.v1.Job

JobControllerClient

Service Description: The JobController provides methods to manage jobs.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
   String projectId = "projectId-894832108";
   String region = "region-934795532";
   Job job = Job.newBuilder().build();
   Job response = jobControllerClient.submitJob(projectId, region, job);
 }

Note: close() needs to be called on the JobControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of JobControllerSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobControllerSettings jobControllerSettings =
     JobControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobControllerSettings jobControllerSettings =
     JobControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobControllerSettings jobControllerSettings =
     JobControllerSettings.newHttpJsonBuilder().build();
 JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

JobControllerClient.ListJobsFixedSizeCollection

JobControllerClient.ListJobsPage

JobControllerClient.ListJobsPagedResponse

JobControllerGrpc

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerBlockingStub

A stub to allow clients to do synchronous rpc calls to service JobController.

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service JobController.

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerImplBase

Base class for the server implementation of the service JobController.

The JobController provides methods to manage jobs.

JobControllerGrpc.JobControllerStub

A stub to allow clients to do asynchronous rpc calls to service JobController.

The JobController provides methods to manage jobs.

JobControllerSettings

Settings class to configure an instance of JobControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of submitJob to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 JobControllerSettings.Builder jobControllerSettingsBuilder = JobControllerSettings.newBuilder();
 jobControllerSettingsBuilder
     .submitJobSettings()
     .setRetrySettings(
         jobControllerSettingsBuilder
             .submitJobSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 JobControllerSettings jobControllerSettings = jobControllerSettingsBuilder.build();

JobControllerSettings.Builder

Builder for JobControllerSettings.

JobMetadata

Job Operation metadata.

Protobuf type google.cloud.dataproc.v1.JobMetadata

JobMetadata.Builder

Job Operation metadata.

Protobuf type google.cloud.dataproc.v1.JobMetadata

JobPlacement

Dataproc job config.

Protobuf type google.cloud.dataproc.v1.JobPlacement

JobPlacement.Builder

Dataproc job config.

Protobuf type google.cloud.dataproc.v1.JobPlacement

JobReference

Encapsulates the full scoping used to reference a job.

Protobuf type google.cloud.dataproc.v1.JobReference

JobReference.Builder

Encapsulates the full scoping used to reference a job.

Protobuf type google.cloud.dataproc.v1.JobReference

JobScheduling

Job scheduling options.

Protobuf type google.cloud.dataproc.v1.JobScheduling

JobScheduling.Builder

Job scheduling options.

Protobuf type google.cloud.dataproc.v1.JobScheduling

JobStatus

Dataproc job status.

Protobuf type google.cloud.dataproc.v1.JobStatus

JobStatus.Builder

Dataproc job status.

Protobuf type google.cloud.dataproc.v1.JobStatus

JobsProto

KerberosConfig

Specifies Kerberos related configuration.

Protobuf type google.cloud.dataproc.v1.KerberosConfig

KerberosConfig.Builder

Specifies Kerberos related configuration.

Protobuf type google.cloud.dataproc.v1.KerberosConfig

KubernetesClusterConfig

The configuration for running the Dataproc cluster on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig

KubernetesClusterConfig.Builder

The configuration for running the Dataproc cluster on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig

KubernetesSoftwareConfig

The software configuration for this Dataproc cluster running on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig

KubernetesSoftwareConfig.Builder

The software configuration for this Dataproc cluster running on Kubernetes.

Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig

LifecycleConfig

Specifies the cluster auto-delete schedule configuration.

Protobuf type google.cloud.dataproc.v1.LifecycleConfig

LifecycleConfig.Builder

Specifies the cluster auto-delete schedule configuration.

Protobuf type google.cloud.dataproc.v1.LifecycleConfig

ListAutoscalingPoliciesRequest

A request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest

ListAutoscalingPoliciesRequest.Builder

A request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest

ListAutoscalingPoliciesResponse

A response to a request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse

ListAutoscalingPoliciesResponse.Builder

A response to a request to list autoscaling policies in a project.

Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse

ListBatchesRequest

A request to list batch workloads in a project.

Protobuf type google.cloud.dataproc.v1.ListBatchesRequest

ListBatchesRequest.Builder

A request to list batch workloads in a project.

Protobuf type google.cloud.dataproc.v1.ListBatchesRequest

ListBatchesResponse

A list of batch workloads.

Protobuf type google.cloud.dataproc.v1.ListBatchesResponse

ListBatchesResponse.Builder

A list of batch workloads.

Protobuf type google.cloud.dataproc.v1.ListBatchesResponse

ListClustersRequest

A request to list the clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersRequest

ListClustersRequest.Builder

A request to list the clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersRequest

ListClustersResponse

The list of all clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersResponse

ListClustersResponse.Builder

The list of all clusters in a project.

Protobuf type google.cloud.dataproc.v1.ListClustersResponse

ListJobsRequest

A request to list jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsRequest

ListJobsRequest.Builder

A request to list jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsRequest

ListJobsResponse

A list of jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsResponse

ListJobsResponse.Builder

A list of jobs in a project.

Protobuf type google.cloud.dataproc.v1.ListJobsResponse

ListWorkflowTemplatesRequest

A request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest

ListWorkflowTemplatesRequest.Builder

A request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest

ListWorkflowTemplatesResponse

A response to a request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse

ListWorkflowTemplatesResponse.Builder

A response to a request to list workflow templates in a project.

Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse

LocationName

LocationName.Builder

Builder for projects/{project}/locations/{location}.

LoggingConfig

The runtime logging config of the job.

Protobuf type google.cloud.dataproc.v1.LoggingConfig

LoggingConfig.Builder

The runtime logging config of the job.

Protobuf type google.cloud.dataproc.v1.LoggingConfig

ManagedCluster

Cluster that is managed by the workflow.

Protobuf type google.cloud.dataproc.v1.ManagedCluster

ManagedCluster.Builder

Cluster that is managed by the workflow.

Protobuf type google.cloud.dataproc.v1.ManagedCluster

ManagedGroupConfig

Specifies the resources used to actively manage an instance group.

Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig

ManagedGroupConfig.Builder

Specifies the resources used to actively manage an instance group.

Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig

MetastoreConfig

Specifies a Metastore configuration.

Protobuf type google.cloud.dataproc.v1.MetastoreConfig

MetastoreConfig.Builder

Specifies a Metastore configuration.

Protobuf type google.cloud.dataproc.v1.MetastoreConfig

NodeGroup

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.

Protobuf type google.cloud.dataproc.v1.NodeGroup

NodeGroup.Builder

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.

Protobuf type google.cloud.dataproc.v1.NodeGroup

NodeGroupAffinity

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.

Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity

NodeGroupAffinity.Builder

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.

Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity

NodeGroupControllerClient

Service Description: The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (NodeGroupControllerClient nodeGroupControllerClient = NodeGroupControllerClient.create()) {
   NodeGroupName name = NodeGroupName.of("[PROJECT]", "[REGION]", "[CLUSTER]", "[NODE_GROUP]");
   NodeGroup response = nodeGroupControllerClient.getNodeGroup(name);
 }

Note: close() needs to be called on the NodeGroupControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of NodeGroupControllerSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 NodeGroupControllerSettings nodeGroupControllerSettings =
     NodeGroupControllerSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 NodeGroupControllerClient nodeGroupControllerClient =
     NodeGroupControllerClient.create(nodeGroupControllerSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 NodeGroupControllerSettings nodeGroupControllerSettings =
     NodeGroupControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
 NodeGroupControllerClient nodeGroupControllerClient =
     NodeGroupControllerClient.create(nodeGroupControllerSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 NodeGroupControllerSettings nodeGroupControllerSettings =
     NodeGroupControllerSettings.newHttpJsonBuilder().build();
 NodeGroupControllerClient nodeGroupControllerClient =
     NodeGroupControllerClient.create(nodeGroupControllerSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

NodeGroupControllerGrpc

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupControllerGrpc.NodeGroupControllerBlockingStub

A stub to allow clients to do synchronous rpc calls to service NodeGroupController.

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupControllerGrpc.NodeGroupControllerFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service NodeGroupController.

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupControllerGrpc.NodeGroupControllerImplBase

Base class for the server implementation of the service NodeGroupController.

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupControllerGrpc.NodeGroupControllerStub

A stub to allow clients to do asynchronous rpc calls to service NodeGroupController.

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupControllerSettings

Settings class to configure an instance of NodeGroupControllerClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of getNodeGroup to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 NodeGroupControllerSettings.Builder nodeGroupControllerSettingsBuilder =
     NodeGroupControllerSettings.newBuilder();
 nodeGroupControllerSettingsBuilder
     .getNodeGroupSettings()
     .setRetrySettings(
         nodeGroupControllerSettingsBuilder
             .getNodeGroupSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 NodeGroupControllerSettings nodeGroupControllerSettings =
     nodeGroupControllerSettingsBuilder.build();

NodeGroupControllerSettings.Builder

Builder for NodeGroupControllerSettings.

NodeGroupName

NodeGroupName.Builder

Builder for projects/{project}/regions/{region}/clusters/{cluster}/nodeGroups/{node_group}.

NodeGroupOperationMetadata

Metadata describing the node group operation.

Protobuf type google.cloud.dataproc.v1.NodeGroupOperationMetadata

NodeGroupOperationMetadata.Builder

Metadata describing the node group operation.

Protobuf type google.cloud.dataproc.v1.NodeGroupOperationMetadata

NodeGroupsProto

NodeInitializationAction

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

Protobuf type google.cloud.dataproc.v1.NodeInitializationAction

NodeInitializationAction.Builder

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

Protobuf type google.cloud.dataproc.v1.NodeInitializationAction

OperationsProto

OrderedJob

A job executed by the workflow.

Protobuf type google.cloud.dataproc.v1.OrderedJob

OrderedJob.Builder

A job executed by the workflow.

Protobuf type google.cloud.dataproc.v1.OrderedJob

ParameterValidation

Configuration for parameter validation.

Protobuf type google.cloud.dataproc.v1.ParameterValidation

ParameterValidation.Builder

Configuration for parameter validation.

Protobuf type google.cloud.dataproc.v1.ParameterValidation

PeripheralsConfig

Auxiliary services configuration for a workload.

Protobuf type google.cloud.dataproc.v1.PeripheralsConfig

PeripheralsConfig.Builder

Auxiliary services configuration for a workload.

Protobuf type google.cloud.dataproc.v1.PeripheralsConfig

PigJob

A Dataproc job for running Apache Pig queries on YARN.

Protobuf type google.cloud.dataproc.v1.PigJob

PigJob.Builder

A Dataproc job for running Apache Pig queries on YARN.

Protobuf type google.cloud.dataproc.v1.PigJob

PrestoJob

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

Protobuf type google.cloud.dataproc.v1.PrestoJob

PrestoJob.Builder

A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.

Protobuf type google.cloud.dataproc.v1.PrestoJob

PySparkBatch

A configuration for running an Apache PySpark batch workload.

Protobuf type google.cloud.dataproc.v1.PySparkBatch

PySparkBatch.Builder

A configuration for running an Apache PySpark batch workload.

Protobuf type google.cloud.dataproc.v1.PySparkBatch

PySparkJob

A Dataproc job for running Apache PySpark applications on YARN.

Protobuf type google.cloud.dataproc.v1.PySparkJob

PySparkJob.Builder

A Dataproc job for running Apache PySpark applications on YARN.

Protobuf type google.cloud.dataproc.v1.PySparkJob

QueryList

A list of queries to run on a cluster.

Protobuf type google.cloud.dataproc.v1.QueryList

QueryList.Builder

A list of queries to run on a cluster.

Protobuf type google.cloud.dataproc.v1.QueryList

RegexValidation

Validation based on regular expressions.

Protobuf type google.cloud.dataproc.v1.RegexValidation

RegexValidation.Builder

Validation based on regular expressions.

Protobuf type google.cloud.dataproc.v1.RegexValidation

RegionName

RegionName.Builder

Builder for projects/{project}/regions/{region}.

ReservationAffinity

Reservation Affinity for consuming Zonal reservation.

Protobuf type google.cloud.dataproc.v1.ReservationAffinity

ReservationAffinity.Builder

Reservation Affinity for consuming Zonal reservation.

Protobuf type google.cloud.dataproc.v1.ReservationAffinity

ResizeNodeGroupRequest

A request to resize a node group.

Protobuf type google.cloud.dataproc.v1.ResizeNodeGroupRequest

ResizeNodeGroupRequest.Builder

A request to resize a node group.

Protobuf type google.cloud.dataproc.v1.ResizeNodeGroupRequest

RuntimeConfig

Runtime configuration for a workload.

Protobuf type google.cloud.dataproc.v1.RuntimeConfig

RuntimeConfig.Builder

Runtime configuration for a workload.

Protobuf type google.cloud.dataproc.v1.RuntimeConfig

RuntimeInfo

Runtime information about workload execution.

Protobuf type google.cloud.dataproc.v1.RuntimeInfo

RuntimeInfo.Builder

Runtime information about workload execution.

Protobuf type google.cloud.dataproc.v1.RuntimeInfo

SecurityConfig

Security related configuration, including encryption, Kerberos, etc.

Protobuf type google.cloud.dataproc.v1.SecurityConfig

SecurityConfig.Builder

Security related configuration, including encryption, Kerberos, etc.

Protobuf type google.cloud.dataproc.v1.SecurityConfig

SharedProto

ShieldedInstanceConfig

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig

ShieldedInstanceConfig.Builder

Shielded Instance Config for clusters using Compute Engine Shielded VMs.

Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig

SoftwareConfig

Specifies the selection and config of software inside the cluster.

Protobuf type google.cloud.dataproc.v1.SoftwareConfig

SoftwareConfig.Builder

Specifies the selection and config of software inside the cluster.

Protobuf type google.cloud.dataproc.v1.SoftwareConfig

SparkBatch

A configuration for running an Apache Spark batch workload.

Protobuf type google.cloud.dataproc.v1.SparkBatch

SparkBatch.Builder

A configuration for running an Apache Spark batch workload.

Protobuf type google.cloud.dataproc.v1.SparkBatch

SparkHistoryServerConfig

Spark History Server configuration for the workload.

Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig

SparkHistoryServerConfig.Builder

Spark History Server configuration for the workload.

Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig

SparkJob

A Dataproc job for running Apache Spark applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkJob

SparkJob.Builder

A Dataproc job for running Apache Spark applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkJob

SparkRBatch

A configuration for running an Apache SparkR batch workload.

Protobuf type google.cloud.dataproc.v1.SparkRBatch

SparkRBatch.Builder

A configuration for running an Apache SparkR batch workload.

Protobuf type google.cloud.dataproc.v1.SparkRBatch

SparkRJob

A Dataproc job for running Apache SparkR applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkRJob

SparkRJob.Builder

A Dataproc job for running Apache SparkR applications on YARN.

Protobuf type google.cloud.dataproc.v1.SparkRJob

SparkSqlBatch

A configuration for running Apache Spark SQL queries as a batch workload.

Protobuf type google.cloud.dataproc.v1.SparkSqlBatch

SparkSqlBatch.Builder

A configuration for running Apache Spark SQL queries as a batch workload.

Protobuf type google.cloud.dataproc.v1.SparkSqlBatch

SparkSqlJob

A Dataproc job for running Apache Spark SQL queries.

Protobuf type google.cloud.dataproc.v1.SparkSqlJob

SparkSqlJob.Builder

A Dataproc job for running Apache Spark SQL queries.

Protobuf type google.cloud.dataproc.v1.SparkSqlJob

StartClusterRequest

A request to start a cluster.

Protobuf type google.cloud.dataproc.v1.StartClusterRequest

StartClusterRequest.Builder

A request to start a cluster.

Protobuf type google.cloud.dataproc.v1.StartClusterRequest

StopClusterRequest

A request to stop a cluster.

Protobuf type google.cloud.dataproc.v1.StopClusterRequest

StopClusterRequest.Builder

A request to stop a cluster.

Protobuf type google.cloud.dataproc.v1.StopClusterRequest

SubmitJobRequest

A request to submit a job.

Protobuf type google.cloud.dataproc.v1.SubmitJobRequest

SubmitJobRequest.Builder

A request to submit a job.

Protobuf type google.cloud.dataproc.v1.SubmitJobRequest

TemplateParameter

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

Labels
File uris
Job properties
Job arguments
Script variables
Main class (in HadoopJob and SparkJob)
Zone (in ClusterSelector)

Protobuf type google.cloud.dataproc.v1.TemplateParameter

TemplateParameter.Builder

A configurable parameter that replaces one or more fields in the template. Parameterizable fields:

Labels
File uris
Job properties
Job arguments
Script variables
Main class (in HadoopJob and SparkJob)
Zone (in ClusterSelector)

Protobuf type google.cloud.dataproc.v1.TemplateParameter

TrinoJob

A Dataproc job for running Trino queries. IMPORTANT: The Dataproc Trino Optional Component must be enabled when the cluster is created to submit a Trino job to the cluster.

Protobuf type google.cloud.dataproc.v1.TrinoJob

TrinoJob.Builder

A Dataproc job for running Trino queries. IMPORTANT: The Dataproc Trino Optional Component must be enabled when the cluster is created to submit a Trino job to the cluster.

Protobuf type google.cloud.dataproc.v1.TrinoJob

UpdateAutoscalingPolicyRequest

A request to update an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest

UpdateAutoscalingPolicyRequest.Builder

A request to update an autoscaling policy.

Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest

UpdateClusterRequest

A request to update a cluster.

Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest

UpdateClusterRequest.Builder

A request to update a cluster.

Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest

UpdateJobRequest

A request to update a job.

Protobuf type google.cloud.dataproc.v1.UpdateJobRequest

UpdateJobRequest.Builder

A request to update a job.

Protobuf type google.cloud.dataproc.v1.UpdateJobRequest

UpdateWorkflowTemplateRequest

A request to update a workflow template.

Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest

UpdateWorkflowTemplateRequest.Builder

A request to update a workflow template.

Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest

UsageMetrics

Usage metrics represent approximate total resources consumed by a workload.

Protobuf type google.cloud.dataproc.v1.UsageMetrics

UsageMetrics.Builder

Usage metrics represent approximate total resources consumed by a workload.

Protobuf type google.cloud.dataproc.v1.UsageMetrics

UsageSnapshot

The usage snaphot represents the resources consumed by a workload at a specified time.

Protobuf type google.cloud.dataproc.v1.UsageSnapshot

UsageSnapshot.Builder

The usage snaphot represents the resources consumed by a workload at a specified time.

Protobuf type google.cloud.dataproc.v1.UsageSnapshot

ValueValidation

Validation based on a list of allowed values.

Protobuf type google.cloud.dataproc.v1.ValueValidation

ValueValidation.Builder

Validation based on a list of allowed values.

Protobuf type google.cloud.dataproc.v1.ValueValidation

VirtualClusterConfig

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig

VirtualClusterConfig.Builder

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.

Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig

WorkflowGraph

The workflow graph.

Protobuf type google.cloud.dataproc.v1.WorkflowGraph

WorkflowGraph.Builder

The workflow graph.

Protobuf type google.cloud.dataproc.v1.WorkflowGraph

WorkflowMetadata

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowMetadata

WorkflowMetadata.Builder

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowMetadata

WorkflowNode

The workflow node.

Protobuf type google.cloud.dataproc.v1.WorkflowNode

WorkflowNode.Builder

The workflow node.

Protobuf type google.cloud.dataproc.v1.WorkflowNode

WorkflowTemplate

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplate

WorkflowTemplate.Builder

A Dataproc workflow template resource.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplate

WorkflowTemplateName

WorkflowTemplateName.Builder

Builder for projects/{project}/regions/{region}/workflowTemplates/{workflow_template}.

WorkflowTemplateName.ProjectLocationWorkflowTemplateBuilder

Builder for projects/{project}/locations/{location}/workflowTemplates/{workflow_template}.

WorkflowTemplatePlacement

Specifies workflow execution target.

Either managed_cluster or cluster_selector is required.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement

WorkflowTemplatePlacement.Builder

Specifies workflow execution target.

Either managed_cluster or cluster_selector is required.

Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement

WorkflowTemplateServiceClient

Service Description: The API interface for managing Workflow Templates in the Dataproc API.

This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create()) {
   LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
   WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
   WorkflowTemplate response =
       workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
 }

Note: close() needs to be called on the WorkflowTemplateServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().

The surface of this class includes several types of Java methods for each of the API's methods:

A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.

See the individual methods for example code.

This class can be customized by passing in a custom instance of WorkflowTemplateServiceSettings to create(). For example:

To customize credentials:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     WorkflowTemplateServiceSettings.newBuilder()
         .setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
         .build();
 WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);

To customize the endpoint:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     WorkflowTemplateServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
 WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);

To use REST (HTTP1.1/JSON) transport (instead of gRPC) for sending and receiving requests over the wire:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     WorkflowTemplateServiceSettings.newHttpJsonBuilder().build();
 WorkflowTemplateServiceClient workflowTemplateServiceClient =
     WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);

Please refer to the GitHub repository's samples for more quickstart code snippets.

WorkflowTemplateServiceClient.ListWorkflowTemplatesFixedSizeCollection

WorkflowTemplateServiceClient.ListWorkflowTemplatesPage

WorkflowTemplateServiceClient.ListWorkflowTemplatesPagedResponse

WorkflowTemplateServiceGrpc

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceBlockingStub

A stub to allow clients to do synchronous rpc calls to service WorkflowTemplateService.

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceFutureStub

A stub to allow clients to do ListenableFuture-style rpc calls to service WorkflowTemplateService.

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceImplBase

Base class for the server implementation of the service WorkflowTemplateService.

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceGrpc.WorkflowTemplateServiceStub

A stub to allow clients to do asynchronous rpc calls to service WorkflowTemplateService.

The API interface for managing Workflow Templates in the Dataproc API.

WorkflowTemplateServiceSettings

Settings class to configure an instance of WorkflowTemplateServiceClient.

The default instance has everything set to sensible defaults:

The default service address (dataproc.googleapis.com) and default port (443) are used.
Credentials are acquired automatically through Application Default Credentials.
Retries are configured for idempotent methods but not for non-idempotent methods.

The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.

For example, to set the total timeout of createWorkflowTemplate to 30 seconds:


 // This snippet has been automatically generated and should be regarded as a code template only.
 // It will require modifications to work:
 // - It may require correct/in-range values for request initialization.
 // - It may require specifying regional endpoints when creating the service client as shown in
 // https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
 WorkflowTemplateServiceSettings.Builder workflowTemplateServiceSettingsBuilder =
     WorkflowTemplateServiceSettings.newBuilder();
 workflowTemplateServiceSettingsBuilder
     .createWorkflowTemplateSettings()
     .setRetrySettings(
         workflowTemplateServiceSettingsBuilder
             .createWorkflowTemplateSettings()
             .getRetrySettings()
             .toBuilder()
             .setTotalTimeout(Duration.ofSeconds(30))
             .build());
 WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
     workflowTemplateServiceSettingsBuilder.build();

WorkflowTemplateServiceSettings.Builder

Builder for WorkflowTemplateServiceSettings.

WorkflowTemplatesProto

YarnApplication

A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.YarnApplication

YarnApplication.Builder

A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>.

Beta Feature: This report is available for testing purposes only. It may be changed before final release.

Protobuf type google.cloud.dataproc.v1.YarnApplication

Interfaces

AcceleratorConfigOrBuilder

AutoscalingConfigOrBuilder

AutoscalingPolicyOrBuilder

AutoscalingPolicyServiceGrpc.AsyncService

The API interface for managing autoscaling policies in the Dataproc API.

AuxiliaryNodeGroupOrBuilder

AuxiliaryServicesConfigOrBuilder

BasicAutoscalingAlgorithmOrBuilder

BasicYarnAutoscalingConfigOrBuilder

Batch.StateHistoryOrBuilder

BatchControllerGrpc.AsyncService

The BatchController provides methods to manage batch workloads.

BatchOperationMetadataOrBuilder

BatchOrBuilder

CancelJobRequestOrBuilder

ClusterConfigOrBuilder

ClusterControllerGrpc.AsyncService

The ClusterControllerService provides methods to manage clusters of Compute Engine instances.

ClusterMetricsOrBuilder

ClusterOperationMetadataOrBuilder

ClusterOperationOrBuilder

ClusterOperationStatusOrBuilder

ClusterOrBuilder

ClusterSelectorOrBuilder

ClusterStatusOrBuilder

ConfidentialInstanceConfigOrBuilder

CreateAutoscalingPolicyRequestOrBuilder

CreateBatchRequestOrBuilder

CreateClusterRequestOrBuilder

CreateNodeGroupRequestOrBuilder

CreateWorkflowTemplateRequestOrBuilder

DataprocMetricConfig.MetricOrBuilder

DataprocMetricConfigOrBuilder

DeleteAutoscalingPolicyRequestOrBuilder

DeleteBatchRequestOrBuilder

DeleteClusterRequestOrBuilder

DeleteJobRequestOrBuilder

DeleteWorkflowTemplateRequestOrBuilder

DiagnoseClusterRequestOrBuilder

DiagnoseClusterResultsOrBuilder

DiskConfigOrBuilder

DriverSchedulingConfigOrBuilder

EncryptionConfigOrBuilder

EndpointConfigOrBuilder

EnvironmentConfigOrBuilder

ExecutionConfigOrBuilder

GceClusterConfigOrBuilder

GetAutoscalingPolicyRequestOrBuilder

GetBatchRequestOrBuilder

GetClusterRequestOrBuilder

GetJobRequestOrBuilder

GetNodeGroupRequestOrBuilder

GetWorkflowTemplateRequestOrBuilder

GkeClusterConfigOrBuilder

GkeNodePoolConfig.GkeNodeConfigOrBuilder

GkeNodePoolConfig.GkeNodePoolAcceleratorConfigOrBuilder

GkeNodePoolConfig.GkeNodePoolAutoscalingConfigOrBuilder

GkeNodePoolConfigOrBuilder

GkeNodePoolTargetOrBuilder

HadoopJobOrBuilder

HiveJobOrBuilder

IdentityConfigOrBuilder

InstanceFlexibilityPolicy.InstanceSelectionOrBuilder

InstanceFlexibilityPolicy.InstanceSelectionResultOrBuilder

InstanceFlexibilityPolicyOrBuilder

InstanceGroupAutoscalingPolicyConfigOrBuilder

InstanceGroupConfigOrBuilder

InstanceReferenceOrBuilder

InstantiateInlineWorkflowTemplateRequestOrBuilder

InstantiateWorkflowTemplateRequestOrBuilder

JobControllerGrpc.AsyncService

The JobController provides methods to manage jobs.

JobMetadataOrBuilder

JobOrBuilder

JobPlacementOrBuilder

JobReferenceOrBuilder

JobSchedulingOrBuilder

JobStatusOrBuilder

KerberosConfigOrBuilder

KubernetesClusterConfigOrBuilder

KubernetesSoftwareConfigOrBuilder

LifecycleConfigOrBuilder

ListAutoscalingPoliciesRequestOrBuilder

ListAutoscalingPoliciesResponseOrBuilder

ListBatchesRequestOrBuilder

ListBatchesResponseOrBuilder

ListClustersRequestOrBuilder

ListClustersResponseOrBuilder

ListJobsRequestOrBuilder

ListJobsResponseOrBuilder

ListWorkflowTemplatesRequestOrBuilder

ListWorkflowTemplatesResponseOrBuilder

LoggingConfigOrBuilder

ManagedClusterOrBuilder

ManagedGroupConfigOrBuilder

MetastoreConfigOrBuilder

NodeGroupAffinityOrBuilder

NodeGroupControllerGrpc.AsyncService

The NodeGroupControllerService provides methods to manage node groups of Compute Engine managed instances.

NodeGroupOperationMetadataOrBuilder

NodeGroupOrBuilder

NodeInitializationActionOrBuilder

OrderedJobOrBuilder

ParameterValidationOrBuilder

PeripheralsConfigOrBuilder

PigJobOrBuilder

PrestoJobOrBuilder

PySparkBatchOrBuilder

PySparkJobOrBuilder

QueryListOrBuilder

RegexValidationOrBuilder

ReservationAffinityOrBuilder

ResizeNodeGroupRequestOrBuilder

RuntimeConfigOrBuilder

RuntimeInfoOrBuilder

SecurityConfigOrBuilder

ShieldedInstanceConfigOrBuilder

SoftwareConfigOrBuilder

SparkBatchOrBuilder

SparkHistoryServerConfigOrBuilder

SparkJobOrBuilder

SparkRBatchOrBuilder

SparkRJobOrBuilder

SparkSqlBatchOrBuilder

SparkSqlJobOrBuilder

StartClusterRequestOrBuilder

StopClusterRequestOrBuilder

SubmitJobRequestOrBuilder

TemplateParameterOrBuilder

TrinoJobOrBuilder

UpdateAutoscalingPolicyRequestOrBuilder

UpdateClusterRequestOrBuilder

UpdateJobRequestOrBuilder

UpdateWorkflowTemplateRequestOrBuilder

UsageMetricsOrBuilder

UsageSnapshotOrBuilder

ValueValidationOrBuilder

VirtualClusterConfigOrBuilder

WorkflowGraphOrBuilder

WorkflowMetadataOrBuilder

WorkflowNodeOrBuilder

WorkflowTemplateOrBuilder

WorkflowTemplatePlacementOrBuilder

WorkflowTemplateServiceGrpc.AsyncService

The API interface for managing Workflow Templates in the Dataproc API.

YarnApplicationOrBuilder

Enums

AutoscalingPolicy.AlgorithmCase

BasicAutoscalingAlgorithm.ConfigCase

Batch.BatchConfigCase

Batch.State

The batch state.

Protobuf enum google.cloud.dataproc.v1.Batch.State

BatchOperationMetadata.BatchOperationType

Operation type for Batch resources

Protobuf enum google.cloud.dataproc.v1.BatchOperationMetadata.BatchOperationType

ClusterOperationStatus.State

The operation state.

Protobuf enum google.cloud.dataproc.v1.ClusterOperationStatus.State

ClusterStatus.State

The cluster state.

Protobuf enum google.cloud.dataproc.v1.ClusterStatus.State

ClusterStatus.Substate

The cluster substate.

Protobuf enum google.cloud.dataproc.v1.ClusterStatus.Substate

Component

Cluster components that can be activated.

Protobuf enum google.cloud.dataproc.v1.Component

DataprocMetricConfig.MetricSource

A source for the collection of Dataproc custom metrics (see Custom metrics).

Protobuf enum google.cloud.dataproc.v1.DataprocMetricConfig.MetricSource

ExecutionConfig.NetworkCase

FailureAction

Actions in response to failure of a resource associated with a cluster.

Protobuf enum google.cloud.dataproc.v1.FailureAction

GceClusterConfig.PrivateIpv6GoogleAccess

PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes can communicate with Google Services through gRPC over IPv6. These values are directly mapped to corresponding values in the Compute Engine Instance fields.

Protobuf enum google.cloud.dataproc.v1.GceClusterConfig.PrivateIpv6GoogleAccess

GkeNodePoolTarget.Role

Role specifies the tasks that will run on the node pool. Roles can be specific to workloads. Exactly one GkeNodePoolTarget within the virtual cluster must have the DEFAULT role, which is used to run all workloads that are not associated with a node pool.

Protobuf enum google.cloud.dataproc.v1.GkeNodePoolTarget.Role