The interfaces provided are listed below, along with usage samples.
AutoscalingPolicyServiceClient
Service Description: The API interface for managing autoscaling policies in the Dataproc API.
Sample for AutoscalingPolicyServiceClient:
try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
AutoscalingPolicyServiceClient.create()) {
LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
AutoscalingPolicy response =
autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
}
BatchControllerClient
Service Description: The BatchController provides methods to manage batch workloads.
Sample for BatchControllerClient:
try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
Batch response = batchControllerClient.getBatch(name);
}
ClusterControllerClient
Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
Sample for ClusterControllerClient:
try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
String projectId = "projectId-894832108";
String region = "region-934795532";
String clusterName = "clusterName-1141738587";
Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
}
JobControllerClient
Service Description: The JobController provides methods to manage jobs.
Sample for JobControllerClient:
try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
String projectId = "projectId-894832108";
String region = "region-934795532";
Job job = Job.newBuilder().build();
Job response = jobControllerClient.submitJob(projectId, region, job);
}
WorkflowTemplateServiceClient
Service Description: The API interface for managing Workflow Templates in the Dataproc API.
Sample for WorkflowTemplateServiceClient:
try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
WorkflowTemplateServiceClient.create()) {
LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
WorkflowTemplate response =
workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
}
Classes
AcceleratorConfig
Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.
Protobuf type google.cloud.dataproc.v1.AcceleratorConfig
AcceleratorConfig.Builder
Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine.
Protobuf type google.cloud.dataproc.v1.AcceleratorConfig
AutoscalingConfig
Autoscaling Policy config associated with the cluster.
Protobuf type google.cloud.dataproc.v1.AutoscalingConfig
AutoscalingConfig.Builder
Autoscaling Policy config associated with the cluster.
Protobuf type google.cloud.dataproc.v1.AutoscalingConfig
AutoscalingPoliciesProto
AutoscalingPolicy
Describes an autoscaling policy for Dataproc cluster autoscaler.
Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy
AutoscalingPolicy.Builder
Describes an autoscaling policy for Dataproc cluster autoscaler.
Protobuf type google.cloud.dataproc.v1.AutoscalingPolicy
AutoscalingPolicyName
AutoscalingPolicyName.Builder
Builder for projects/{project}/locations/{location}/autoscalingPolicies/{autoscaling_policy}.
AutoscalingPolicyName.ProjectRegionAutoscalingPolicyBuilder
Builder for projects/{project}/regions/{region}/autoscalingPolicies/{autoscaling_policy}.
AutoscalingPolicyServiceClient
Service Description: The API interface for managing autoscaling policies in the Dataproc API.
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
AutoscalingPolicyServiceClient.create()) {
LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
AutoscalingPolicy response =
autoscalingPolicyServiceClient.createAutoscalingPolicy(parent, policy);
}
Note: close() needs to be called on the AutoscalingPolicyServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of AutoscalingPolicyServiceSettings to create(). For example:
To customize credentials:
AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
AutoscalingPolicyServiceSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);
To customize the endpoint:
AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
AutoscalingPolicyServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
AutoscalingPolicyServiceClient autoscalingPolicyServiceClient =
AutoscalingPolicyServiceClient.create(autoscalingPolicyServiceSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
AutoscalingPolicyServiceClient.ListAutoscalingPoliciesFixedSizeCollection
AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPage
AutoscalingPolicyServiceClient.ListAutoscalingPoliciesPagedResponse
AutoscalingPolicyServiceGrpc
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceBlockingStub
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceFutureStub
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceImplBase
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyServiceGrpc.AutoscalingPolicyServiceStub
The API interface for managing autoscaling policies in the Dataproc API.
AutoscalingPolicyServiceSettings
Settings class to configure an instance of AutoscalingPolicyServiceClient.
The default instance has everything set to sensible defaults:
- The default service address (dataproc.googleapis.com) and default port (443) are used.
- Credentials are acquired automatically through Application Default Credentials.
- Retries are configured for idempotent methods but not for non-idempotent methods.
The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.
For example, to set the total timeout of createAutoscalingPolicy to 30 seconds:
AutoscalingPolicyServiceSettings.Builder autoscalingPolicyServiceSettingsBuilder =
AutoscalingPolicyServiceSettings.newBuilder();
autoscalingPolicyServiceSettingsBuilder
.createAutoscalingPolicySettings()
.setRetrySettings(
autoscalingPolicyServiceSettingsBuilder
.createAutoscalingPolicySettings()
.getRetrySettings()
.toBuilder()
.setTotalTimeout(Duration.ofSeconds(30))
.build());
AutoscalingPolicyServiceSettings autoscalingPolicyServiceSettings =
autoscalingPolicyServiceSettingsBuilder.build();
AutoscalingPolicyServiceSettings.Builder
Builder for AutoscalingPolicyServiceSettings.
AuxiliaryServicesConfig
Auxiliary services configuration for a Cluster.
Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig
AuxiliaryServicesConfig.Builder
Auxiliary services configuration for a Cluster.
Protobuf type google.cloud.dataproc.v1.AuxiliaryServicesConfig
BasicAutoscalingAlgorithm
Basic algorithm for autoscaling.
Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm
BasicAutoscalingAlgorithm.Builder
Basic algorithm for autoscaling.
Protobuf type google.cloud.dataproc.v1.BasicAutoscalingAlgorithm
BasicYarnAutoscalingConfig
Basic autoscaling configurations for YARN.
Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig
BasicYarnAutoscalingConfig.Builder
Basic autoscaling configurations for YARN.
Protobuf type google.cloud.dataproc.v1.BasicYarnAutoscalingConfig
Batch
A representation of a batch workload in the service.
Protobuf type google.cloud.dataproc.v1.Batch
Batch.Builder
A representation of a batch workload in the service.
Protobuf type google.cloud.dataproc.v1.Batch
Batch.StateHistory
Historical state information.
Protobuf type google.cloud.dataproc.v1.Batch.StateHistory
Batch.StateHistory.Builder
Historical state information.
Protobuf type google.cloud.dataproc.v1.Batch.StateHistory
BatchControllerClient
Service Description: The BatchController provides methods to manage batch workloads.
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (BatchControllerClient batchControllerClient = BatchControllerClient.create()) {
BatchName name = BatchName.of("[PROJECT]", "[LOCATION]", "[BATCH]");
Batch response = batchControllerClient.getBatch(name);
}
Note: close() needs to be called on the BatchControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of BatchControllerSettings to create(). For example:
To customize credentials:
BatchControllerSettings batchControllerSettings =
BatchControllerSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
BatchControllerClient batchControllerClient =
BatchControllerClient.create(batchControllerSettings);
To customize the endpoint:
BatchControllerSettings batchControllerSettings =
BatchControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
BatchControllerClient batchControllerClient =
BatchControllerClient.create(batchControllerSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
BatchControllerClient.ListBatchesFixedSizeCollection
BatchControllerClient.ListBatchesPage
BatchControllerClient.ListBatchesPagedResponse
BatchControllerGrpc
The BatchController provides methods to manage batch workloads.
BatchControllerGrpc.BatchControllerBlockingStub
The BatchController provides methods to manage batch workloads.
BatchControllerGrpc.BatchControllerFutureStub
The BatchController provides methods to manage batch workloads.
BatchControllerGrpc.BatchControllerImplBase
The BatchController provides methods to manage batch workloads.
BatchControllerGrpc.BatchControllerStub
The BatchController provides methods to manage batch workloads.
BatchControllerSettings
Settings class to configure an instance of BatchControllerClient.
The default instance has everything set to sensible defaults:
- The default service address (dataproc.googleapis.com) and default port (443) are used.
- Credentials are acquired automatically through Application Default Credentials.
- Retries are configured for idempotent methods but not for non-idempotent methods.
The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.
For example, to set the total timeout of getBatch to 30 seconds:
BatchControllerSettings.Builder batchControllerSettingsBuilder =
BatchControllerSettings.newBuilder();
batchControllerSettingsBuilder
.getBatchSettings()
.setRetrySettings(
batchControllerSettingsBuilder
.getBatchSettings()
.getRetrySettings()
.toBuilder()
.setTotalTimeout(Duration.ofSeconds(30))
.build());
BatchControllerSettings batchControllerSettings = batchControllerSettingsBuilder.build();
BatchControllerSettings.Builder
Builder for BatchControllerSettings.
BatchName
BatchName.Builder
Builder for projects/{project}/locations/{location}/batches/{batch}.
BatchOperationMetadata
Metadata describing the Batch operation.
Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata
BatchOperationMetadata.Builder
Metadata describing the Batch operation.
Protobuf type google.cloud.dataproc.v1.BatchOperationMetadata
BatchesProto
CancelJobRequest
A request to cancel a job.
Protobuf type google.cloud.dataproc.v1.CancelJobRequest
CancelJobRequest.Builder
A request to cancel a job.
Protobuf type google.cloud.dataproc.v1.CancelJobRequest
Cluster
Describes the identifying information, config, and status of a Dataproc cluster
Protobuf type google.cloud.dataproc.v1.Cluster
Cluster.Builder
Describes the identifying information, config, and status of a Dataproc cluster
Protobuf type google.cloud.dataproc.v1.Cluster
ClusterConfig
The cluster config.
Protobuf type google.cloud.dataproc.v1.ClusterConfig
ClusterConfig.Builder
The cluster config.
Protobuf type google.cloud.dataproc.v1.ClusterConfig
ClusterControllerClient
Service Description: The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
String projectId = "projectId-894832108";
String region = "region-934795532";
String clusterName = "clusterName-1141738587";
Cluster response = clusterControllerClient.getCluster(projectId, region, clusterName);
}
Note: close() needs to be called on the ClusterControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of ClusterControllerSettings to create(). For example:
To customize credentials:
ClusterControllerSettings clusterControllerSettings =
ClusterControllerSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
ClusterControllerClient clusterControllerClient =
ClusterControllerClient.create(clusterControllerSettings);
To customize the endpoint:
ClusterControllerSettings clusterControllerSettings =
ClusterControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
ClusterControllerClient clusterControllerClient =
ClusterControllerClient.create(clusterControllerSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
ClusterControllerClient.ListClustersFixedSizeCollection
ClusterControllerClient.ListClustersPage
ClusterControllerClient.ListClustersPagedResponse
ClusterControllerGrpc
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterControllerGrpc.ClusterControllerBlockingStub
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterControllerGrpc.ClusterControllerFutureStub
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterControllerGrpc.ClusterControllerImplBase
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterControllerGrpc.ClusterControllerStub
The ClusterControllerService provides methods to manage clusters of Compute Engine instances.
ClusterControllerSettings
Settings class to configure an instance of ClusterControllerClient.
The default instance has everything set to sensible defaults:
- The default service address (dataproc.googleapis.com) and default port (443) are used.
- Credentials are acquired automatically through Application Default Credentials.
- Retries are configured for idempotent methods but not for non-idempotent methods.
The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.
For example, to set the total timeout of getCluster to 30 seconds:
ClusterControllerSettings.Builder clusterControllerSettingsBuilder =
ClusterControllerSettings.newBuilder();
clusterControllerSettingsBuilder
.getClusterSettings()
.setRetrySettings(
clusterControllerSettingsBuilder
.getClusterSettings()
.getRetrySettings()
.toBuilder()
.setTotalTimeout(Duration.ofSeconds(30))
.build());
ClusterControllerSettings clusterControllerSettings = clusterControllerSettingsBuilder.build();
ClusterControllerSettings.Builder
Builder for ClusterControllerSettings.
ClusterMetrics
Contains cluster daemon metrics, such as HDFS and YARN stats. Beta Feature: This report is available for testing purposes only. It may be changed before final release.
Protobuf type google.cloud.dataproc.v1.ClusterMetrics
ClusterMetrics.Builder
Contains cluster daemon metrics, such as HDFS and YARN stats. Beta Feature: This report is available for testing purposes only. It may be changed before final release.
Protobuf type google.cloud.dataproc.v1.ClusterMetrics
ClusterOperation
The cluster operation triggered by a workflow.
Protobuf type google.cloud.dataproc.v1.ClusterOperation
ClusterOperation.Builder
The cluster operation triggered by a workflow.
Protobuf type google.cloud.dataproc.v1.ClusterOperation
ClusterOperationMetadata
Metadata describing the operation.
Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata
ClusterOperationMetadata.Builder
Metadata describing the operation.
Protobuf type google.cloud.dataproc.v1.ClusterOperationMetadata
ClusterOperationStatus
The status of the operation.
Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus
ClusterOperationStatus.Builder
The status of the operation.
Protobuf type google.cloud.dataproc.v1.ClusterOperationStatus
ClusterSelector
A selector that chooses target cluster for jobs based on metadata.
Protobuf type google.cloud.dataproc.v1.ClusterSelector
ClusterSelector.Builder
A selector that chooses target cluster for jobs based on metadata.
Protobuf type google.cloud.dataproc.v1.ClusterSelector
ClusterStatus
The status of a cluster and its instances.
Protobuf type google.cloud.dataproc.v1.ClusterStatus
ClusterStatus.Builder
The status of a cluster and its instances.
Protobuf type google.cloud.dataproc.v1.ClusterStatus
ClustersProto
ConfidentialInstanceConfig
Confidential Instance Config for clusters using Confidential VMs
Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig
ConfidentialInstanceConfig.Builder
Confidential Instance Config for clusters using Confidential VMs
Protobuf type google.cloud.dataproc.v1.ConfidentialInstanceConfig
CreateAutoscalingPolicyRequest
A request to create an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest
CreateAutoscalingPolicyRequest.Builder
A request to create an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.CreateAutoscalingPolicyRequest
CreateBatchRequest
A request to create a batch workload.
Protobuf type google.cloud.dataproc.v1.CreateBatchRequest
CreateBatchRequest.Builder
A request to create a batch workload.
Protobuf type google.cloud.dataproc.v1.CreateBatchRequest
CreateClusterRequest
A request to create a cluster.
Protobuf type google.cloud.dataproc.v1.CreateClusterRequest
CreateClusterRequest.Builder
A request to create a cluster.
Protobuf type google.cloud.dataproc.v1.CreateClusterRequest
CreateWorkflowTemplateRequest
A request to create a workflow template.
Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest
CreateWorkflowTemplateRequest.Builder
A request to create a workflow template.
Protobuf type google.cloud.dataproc.v1.CreateWorkflowTemplateRequest
DeleteAutoscalingPolicyRequest
A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.
Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest
DeleteAutoscalingPolicyRequest.Builder
A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.
Protobuf type google.cloud.dataproc.v1.DeleteAutoscalingPolicyRequest
DeleteBatchRequest
A request to delete a batch workload.
Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest
DeleteBatchRequest.Builder
A request to delete a batch workload.
Protobuf type google.cloud.dataproc.v1.DeleteBatchRequest
DeleteClusterRequest
A request to delete a cluster.
Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest
DeleteClusterRequest.Builder
A request to delete a cluster.
Protobuf type google.cloud.dataproc.v1.DeleteClusterRequest
DeleteJobRequest
A request to delete a job.
Protobuf type google.cloud.dataproc.v1.DeleteJobRequest
DeleteJobRequest.Builder
A request to delete a job.
Protobuf type google.cloud.dataproc.v1.DeleteJobRequest
DeleteWorkflowTemplateRequest
A request to delete a workflow template. Currently started workflows will remain running.
Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest
DeleteWorkflowTemplateRequest.Builder
A request to delete a workflow template. Currently started workflows will remain running.
Protobuf type google.cloud.dataproc.v1.DeleteWorkflowTemplateRequest
DiagnoseClusterRequest
A request to collect cluster diagnostic information.
Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest
DiagnoseClusterRequest.Builder
A request to collect cluster diagnostic information.
Protobuf type google.cloud.dataproc.v1.DiagnoseClusterRequest
DiagnoseClusterResults
The location of diagnostic output.
Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults
DiagnoseClusterResults.Builder
The location of diagnostic output.
Protobuf type google.cloud.dataproc.v1.DiagnoseClusterResults
DiskConfig
Specifies the config of disk options for a group of VM instances.
Protobuf type google.cloud.dataproc.v1.DiskConfig
DiskConfig.Builder
Specifies the config of disk options for a group of VM instances.
Protobuf type google.cloud.dataproc.v1.DiskConfig
EncryptionConfig
Encryption settings for the cluster.
Protobuf type google.cloud.dataproc.v1.EncryptionConfig
EncryptionConfig.Builder
Encryption settings for the cluster.
Protobuf type google.cloud.dataproc.v1.EncryptionConfig
EndpointConfig
Endpoint config for this cluster
Protobuf type google.cloud.dataproc.v1.EndpointConfig
EndpointConfig.Builder
Endpoint config for this cluster
Protobuf type google.cloud.dataproc.v1.EndpointConfig
EnvironmentConfig
Environment configuration for a workload.
Protobuf type google.cloud.dataproc.v1.EnvironmentConfig
EnvironmentConfig.Builder
Environment configuration for a workload.
Protobuf type google.cloud.dataproc.v1.EnvironmentConfig
ExecutionConfig
Execution configuration for a workload.
Protobuf type google.cloud.dataproc.v1.ExecutionConfig
ExecutionConfig.Builder
Execution configuration for a workload.
Protobuf type google.cloud.dataproc.v1.ExecutionConfig
GceClusterConfig
Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.
Protobuf type google.cloud.dataproc.v1.GceClusterConfig
GceClusterConfig.Builder
Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.
Protobuf type google.cloud.dataproc.v1.GceClusterConfig
GetAutoscalingPolicyRequest
A request to fetch an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest
GetAutoscalingPolicyRequest.Builder
A request to fetch an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.GetAutoscalingPolicyRequest
GetBatchRequest
A request to get the resource representation for a batch workload.
Protobuf type google.cloud.dataproc.v1.GetBatchRequest
GetBatchRequest.Builder
A request to get the resource representation for a batch workload.
Protobuf type google.cloud.dataproc.v1.GetBatchRequest
GetClusterRequest
Request to get the resource representation for a cluster in a project.
Protobuf type google.cloud.dataproc.v1.GetClusterRequest
GetClusterRequest.Builder
Request to get the resource representation for a cluster in a project.
Protobuf type google.cloud.dataproc.v1.GetClusterRequest
GetJobRequest
A request to get the resource representation for a job in a project.
Protobuf type google.cloud.dataproc.v1.GetJobRequest
GetJobRequest.Builder
A request to get the resource representation for a job in a project.
Protobuf type google.cloud.dataproc.v1.GetJobRequest
GetWorkflowTemplateRequest
A request to fetch a workflow template.
Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest
GetWorkflowTemplateRequest.Builder
A request to fetch a workflow template.
Protobuf type google.cloud.dataproc.v1.GetWorkflowTemplateRequest
GkeClusterConfig
The cluster's GKE config.
Protobuf type google.cloud.dataproc.v1.GkeClusterConfig
GkeClusterConfig.Builder
The cluster's GKE config.
Protobuf type google.cloud.dataproc.v1.GkeClusterConfig
GkeNodePoolConfig
The configuration of a GKE NodePool used by a Dataproc-on-GKE cluster.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig
GkeNodePoolConfig.Builder
The configuration of a GKE NodePool used by a Dataproc-on-GKE cluster.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig
GkeNodePoolConfig.GkeNodeConfig
Parameters that describe cluster nodes.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig
GkeNodePoolConfig.GkeNodeConfig.Builder
Parameters that describe cluster nodes.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodeConfig
GkeNodePoolConfig.GkeNodePoolAcceleratorConfig
A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a NodePool.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig
GkeNodePoolConfig.GkeNodePoolAcceleratorConfig.Builder
A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a NodePool.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAcceleratorConfig
GkeNodePoolConfig.GkeNodePoolAutoscalingConfig
GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig
GkeNodePoolConfig.GkeNodePoolAutoscalingConfig.Builder
GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolConfig.GkeNodePoolAutoscalingConfig
GkeNodePoolTarget
GKE NodePools that Dataproc workloads run on.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget
GkeNodePoolTarget.Builder
GKE NodePools that Dataproc workloads run on.
Protobuf type google.cloud.dataproc.v1.GkeNodePoolTarget
HadoopJob
A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.
Protobuf type google.cloud.dataproc.v1.HadoopJob
HadoopJob.Builder
A Dataproc job for running Apache Hadoop MapReduce jobs on Apache Hadoop YARN.
Protobuf type google.cloud.dataproc.v1.HadoopJob
HiveJob
A Dataproc job for running Apache Hive queries on YARN.
Protobuf type google.cloud.dataproc.v1.HiveJob
HiveJob.Builder
A Dataproc job for running Apache Hive queries on YARN.
Protobuf type google.cloud.dataproc.v1.HiveJob
IdentityConfig
Identity related configuration, including service account based secure multi-tenancy user mappings.
Protobuf type google.cloud.dataproc.v1.IdentityConfig
IdentityConfig.Builder
Identity related configuration, including service account based secure multi-tenancy user mappings.
Protobuf type google.cloud.dataproc.v1.IdentityConfig
InstanceGroupAutoscalingPolicyConfig
Configuration for the size bounds of an instance group, including its proportional size to other groups.
Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig
InstanceGroupAutoscalingPolicyConfig.Builder
Configuration for the size bounds of an instance group, including its proportional size to other groups.
Protobuf type google.cloud.dataproc.v1.InstanceGroupAutoscalingPolicyConfig
InstanceGroupConfig
The config settings for Compute Engine resources in an instance group, such as a master or worker group.
Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig
InstanceGroupConfig.Builder
The config settings for Compute Engine resources in an instance group, such as a master or worker group.
Protobuf type google.cloud.dataproc.v1.InstanceGroupConfig
InstantiateInlineWorkflowTemplateRequest
A request to instantiate an inline workflow template.
Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest
InstantiateInlineWorkflowTemplateRequest.Builder
A request to instantiate an inline workflow template.
Protobuf type google.cloud.dataproc.v1.InstantiateInlineWorkflowTemplateRequest
InstantiateWorkflowTemplateRequest
A request to instantiate a workflow template.
Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest
InstantiateWorkflowTemplateRequest.Builder
A request to instantiate a workflow template.
Protobuf type google.cloud.dataproc.v1.InstantiateWorkflowTemplateRequest
Job
A Dataproc job resource.
Protobuf type google.cloud.dataproc.v1.Job
Job.Builder
A Dataproc job resource.
Protobuf type google.cloud.dataproc.v1.Job
JobControllerClient
Service Description: The JobController provides methods to manage jobs.
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
String projectId = "projectId-894832108";
String region = "region-934795532";
Job job = Job.newBuilder().build();
Job response = jobControllerClient.submitJob(projectId, region, job);
}
Note: close() needs to be called on the JobControllerClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of JobControllerSettings to create(). For example:
To customize credentials:
JobControllerSettings jobControllerSettings =
JobControllerSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);
To customize the endpoint:
JobControllerSettings jobControllerSettings =
JobControllerSettings.newBuilder().setEndpoint(myEndpoint).build();
JobControllerClient jobControllerClient = JobControllerClient.create(jobControllerSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
JobControllerClient.ListJobsFixedSizeCollection
JobControllerClient.ListJobsPage
JobControllerClient.ListJobsPagedResponse
JobControllerGrpc
The JobController provides methods to manage jobs.
JobControllerGrpc.JobControllerBlockingStub
The JobController provides methods to manage jobs.
JobControllerGrpc.JobControllerFutureStub
The JobController provides methods to manage jobs.
JobControllerGrpc.JobControllerImplBase
The JobController provides methods to manage jobs.
JobControllerGrpc.JobControllerStub
The JobController provides methods to manage jobs.
JobControllerSettings
Settings class to configure an instance of JobControllerClient.
The default instance has everything set to sensible defaults:
- The default service address (dataproc.googleapis.com) and default port (443) are used.
- Credentials are acquired automatically through Application Default Credentials.
- Retries are configured for idempotent methods but not for non-idempotent methods.
The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.
For example, to set the total timeout of submitJob to 30 seconds:
JobControllerSettings.Builder jobControllerSettingsBuilder = JobControllerSettings.newBuilder();
jobControllerSettingsBuilder
.submitJobSettings()
.setRetrySettings(
jobControllerSettingsBuilder
.submitJobSettings()
.getRetrySettings()
.toBuilder()
.setTotalTimeout(Duration.ofSeconds(30))
.build());
JobControllerSettings jobControllerSettings = jobControllerSettingsBuilder.build();
JobControllerSettings.Builder
Builder for JobControllerSettings.
JobMetadata
Job Operation metadata.
Protobuf type google.cloud.dataproc.v1.JobMetadata
JobMetadata.Builder
Job Operation metadata.
Protobuf type google.cloud.dataproc.v1.JobMetadata
JobPlacement
Dataproc job config.
Protobuf type google.cloud.dataproc.v1.JobPlacement
JobPlacement.Builder
Dataproc job config.
Protobuf type google.cloud.dataproc.v1.JobPlacement
JobReference
Encapsulates the full scoping used to reference a job.
Protobuf type google.cloud.dataproc.v1.JobReference
JobReference.Builder
Encapsulates the full scoping used to reference a job.
Protobuf type google.cloud.dataproc.v1.JobReference
JobScheduling
Job scheduling options.
Protobuf type google.cloud.dataproc.v1.JobScheduling
JobScheduling.Builder
Job scheduling options.
Protobuf type google.cloud.dataproc.v1.JobScheduling
JobStatus
Dataproc job status.
Protobuf type google.cloud.dataproc.v1.JobStatus
JobStatus.Builder
Dataproc job status.
Protobuf type google.cloud.dataproc.v1.JobStatus
JobsProto
KerberosConfig
Specifies Kerberos related configuration.
Protobuf type google.cloud.dataproc.v1.KerberosConfig
KerberosConfig.Builder
Specifies Kerberos related configuration.
Protobuf type google.cloud.dataproc.v1.KerberosConfig
KubernetesClusterConfig
The configuration for running the Dataproc cluster on Kubernetes.
Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig
KubernetesClusterConfig.Builder
The configuration for running the Dataproc cluster on Kubernetes.
Protobuf type google.cloud.dataproc.v1.KubernetesClusterConfig
KubernetesSoftwareConfig
The software configuration for this Dataproc cluster running on Kubernetes.
Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig
KubernetesSoftwareConfig.Builder
The software configuration for this Dataproc cluster running on Kubernetes.
Protobuf type google.cloud.dataproc.v1.KubernetesSoftwareConfig
LifecycleConfig
Specifies the cluster auto-delete schedule configuration.
Protobuf type google.cloud.dataproc.v1.LifecycleConfig
LifecycleConfig.Builder
Specifies the cluster auto-delete schedule configuration.
Protobuf type google.cloud.dataproc.v1.LifecycleConfig
ListAutoscalingPoliciesRequest
A request to list autoscaling policies in a project.
Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest
ListAutoscalingPoliciesRequest.Builder
A request to list autoscaling policies in a project.
Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesRequest
ListAutoscalingPoliciesResponse
A response to a request to list autoscaling policies in a project.
Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse
ListAutoscalingPoliciesResponse.Builder
A response to a request to list autoscaling policies in a project.
Protobuf type google.cloud.dataproc.v1.ListAutoscalingPoliciesResponse
ListBatchesRequest
A request to list batch workloads in a project.
Protobuf type google.cloud.dataproc.v1.ListBatchesRequest
ListBatchesRequest.Builder
A request to list batch workloads in a project.
Protobuf type google.cloud.dataproc.v1.ListBatchesRequest
ListBatchesResponse
A list of batch workloads.
Protobuf type google.cloud.dataproc.v1.ListBatchesResponse
ListBatchesResponse.Builder
A list of batch workloads.
Protobuf type google.cloud.dataproc.v1.ListBatchesResponse
ListClustersRequest
A request to list the clusters in a project.
Protobuf type google.cloud.dataproc.v1.ListClustersRequest
ListClustersRequest.Builder
A request to list the clusters in a project.
Protobuf type google.cloud.dataproc.v1.ListClustersRequest
ListClustersResponse
The list of all clusters in a project.
Protobuf type google.cloud.dataproc.v1.ListClustersResponse
ListClustersResponse.Builder
The list of all clusters in a project.
Protobuf type google.cloud.dataproc.v1.ListClustersResponse
ListJobsRequest
A request to list jobs in a project.
Protobuf type google.cloud.dataproc.v1.ListJobsRequest
ListJobsRequest.Builder
A request to list jobs in a project.
Protobuf type google.cloud.dataproc.v1.ListJobsRequest
ListJobsResponse
A list of jobs in a project.
Protobuf type google.cloud.dataproc.v1.ListJobsResponse
ListJobsResponse.Builder
A list of jobs in a project.
Protobuf type google.cloud.dataproc.v1.ListJobsResponse
ListWorkflowTemplatesRequest
A request to list workflow templates in a project.
Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest
ListWorkflowTemplatesRequest.Builder
A request to list workflow templates in a project.
Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesRequest
ListWorkflowTemplatesResponse
A response to a request to list workflow templates in a project.
Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse
ListWorkflowTemplatesResponse.Builder
A response to a request to list workflow templates in a project.
Protobuf type google.cloud.dataproc.v1.ListWorkflowTemplatesResponse
LocationName
LocationName.Builder
Builder for projects/{project}/locations/{location}.
LoggingConfig
The runtime logging config of the job.
Protobuf type google.cloud.dataproc.v1.LoggingConfig
LoggingConfig.Builder
The runtime logging config of the job.
Protobuf type google.cloud.dataproc.v1.LoggingConfig
ManagedCluster
Cluster that is managed by the workflow.
Protobuf type google.cloud.dataproc.v1.ManagedCluster
ManagedCluster.Builder
Cluster that is managed by the workflow.
Protobuf type google.cloud.dataproc.v1.ManagedCluster
ManagedGroupConfig
Specifies the resources used to actively manage an instance group.
Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig
ManagedGroupConfig.Builder
Specifies the resources used to actively manage an instance group.
Protobuf type google.cloud.dataproc.v1.ManagedGroupConfig
MetastoreConfig
Specifies a Metastore configuration.
Protobuf type google.cloud.dataproc.v1.MetastoreConfig
MetastoreConfig.Builder
Specifies a Metastore configuration.
Protobuf type google.cloud.dataproc.v1.MetastoreConfig
NodeGroupAffinity
Node Group Affinity for clusters using sole-tenant node groups.
Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity
NodeGroupAffinity.Builder
Node Group Affinity for clusters using sole-tenant node groups.
Protobuf type google.cloud.dataproc.v1.NodeGroupAffinity
NodeInitializationAction
Specifies an executable to run on a fully configured node and a timeout period for executable completion.
Protobuf type google.cloud.dataproc.v1.NodeInitializationAction
NodeInitializationAction.Builder
Specifies an executable to run on a fully configured node and a timeout period for executable completion.
Protobuf type google.cloud.dataproc.v1.NodeInitializationAction
OperationsProto
OrderedJob
A job executed by the workflow.
Protobuf type google.cloud.dataproc.v1.OrderedJob
OrderedJob.Builder
A job executed by the workflow.
Protobuf type google.cloud.dataproc.v1.OrderedJob
ParameterValidation
Configuration for parameter validation.
Protobuf type google.cloud.dataproc.v1.ParameterValidation
ParameterValidation.Builder
Configuration for parameter validation.
Protobuf type google.cloud.dataproc.v1.ParameterValidation
PeripheralsConfig
Auxiliary services configuration for a workload.
Protobuf type google.cloud.dataproc.v1.PeripheralsConfig
PeripheralsConfig.Builder
Auxiliary services configuration for a workload.
Protobuf type google.cloud.dataproc.v1.PeripheralsConfig
PigJob
A Dataproc job for running Apache Pig queries on YARN.
Protobuf type google.cloud.dataproc.v1.PigJob
PigJob.Builder
A Dataproc job for running Apache Pig queries on YARN.
Protobuf type google.cloud.dataproc.v1.PigJob
PrestoJob
A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.
Protobuf type google.cloud.dataproc.v1.PrestoJob
PrestoJob.Builder
A Dataproc job for running Presto queries. IMPORTANT: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.
Protobuf type google.cloud.dataproc.v1.PrestoJob
PySparkBatch
A configuration for running an Apache PySpark batch workload.
Protobuf type google.cloud.dataproc.v1.PySparkBatch
PySparkBatch.Builder
A configuration for running an Apache PySpark batch workload.
Protobuf type google.cloud.dataproc.v1.PySparkBatch
PySparkJob
A Dataproc job for running Apache PySpark applications on YARN.
Protobuf type google.cloud.dataproc.v1.PySparkJob
PySparkJob.Builder
A Dataproc job for running Apache PySpark applications on YARN.
Protobuf type google.cloud.dataproc.v1.PySparkJob
QueryList
A list of queries to run on a cluster.
Protobuf type google.cloud.dataproc.v1.QueryList
QueryList.Builder
A list of queries to run on a cluster.
Protobuf type google.cloud.dataproc.v1.QueryList
RegexValidation
Validation based on regular expressions.
Protobuf type google.cloud.dataproc.v1.RegexValidation
RegexValidation.Builder
Validation based on regular expressions.
Protobuf type google.cloud.dataproc.v1.RegexValidation
RegionName
RegionName.Builder
Builder for projects/{project}/regions/{region}.
ReservationAffinity
Reservation Affinity for consuming Zonal reservation.
Protobuf type google.cloud.dataproc.v1.ReservationAffinity
ReservationAffinity.Builder
Reservation Affinity for consuming Zonal reservation.
Protobuf type google.cloud.dataproc.v1.ReservationAffinity
RuntimeConfig
Runtime configuration for a workload.
Protobuf type google.cloud.dataproc.v1.RuntimeConfig
RuntimeConfig.Builder
Runtime configuration for a workload.
Protobuf type google.cloud.dataproc.v1.RuntimeConfig
RuntimeInfo
Runtime information about workload execution.
Protobuf type google.cloud.dataproc.v1.RuntimeInfo
RuntimeInfo.Builder
Runtime information about workload execution.
Protobuf type google.cloud.dataproc.v1.RuntimeInfo
SecurityConfig
Security related configuration, including encryption, Kerberos, etc.
Protobuf type google.cloud.dataproc.v1.SecurityConfig
SecurityConfig.Builder
Security related configuration, including encryption, Kerberos, etc.
Protobuf type google.cloud.dataproc.v1.SecurityConfig
SharedProto
ShieldedInstanceConfig
Shielded Instance Config for clusters using Compute Engine Shielded VMs.
Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig
ShieldedInstanceConfig.Builder
Shielded Instance Config for clusters using Compute Engine Shielded VMs.
Protobuf type google.cloud.dataproc.v1.ShieldedInstanceConfig
SoftwareConfig
Specifies the selection and config of software inside the cluster.
Protobuf type google.cloud.dataproc.v1.SoftwareConfig
SoftwareConfig.Builder
Specifies the selection and config of software inside the cluster.
Protobuf type google.cloud.dataproc.v1.SoftwareConfig
SparkBatch
A configuration for running an Apache Spark batch workload.
Protobuf type google.cloud.dataproc.v1.SparkBatch
SparkBatch.Builder
A configuration for running an Apache Spark batch workload.
Protobuf type google.cloud.dataproc.v1.SparkBatch
SparkHistoryServerConfig
Spark History Server configuration for the workload.
Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig
SparkHistoryServerConfig.Builder
Spark History Server configuration for the workload.
Protobuf type google.cloud.dataproc.v1.SparkHistoryServerConfig
SparkJob
A Dataproc job for running Apache Spark applications on YARN.
Protobuf type google.cloud.dataproc.v1.SparkJob
SparkJob.Builder
A Dataproc job for running Apache Spark applications on YARN.
Protobuf type google.cloud.dataproc.v1.SparkJob
SparkRBatch
A configuration for running an Apache SparkR batch workload.
Protobuf type google.cloud.dataproc.v1.SparkRBatch
SparkRBatch.Builder
A configuration for running an Apache SparkR batch workload.
Protobuf type google.cloud.dataproc.v1.SparkRBatch
SparkRJob
A Dataproc job for running Apache SparkR applications on YARN.
Protobuf type google.cloud.dataproc.v1.SparkRJob
SparkRJob.Builder
A Dataproc job for running Apache SparkR applications on YARN.
Protobuf type google.cloud.dataproc.v1.SparkRJob
SparkSqlBatch
A configuration for running Apache Spark SQL queries as a batch workload.
Protobuf type google.cloud.dataproc.v1.SparkSqlBatch
SparkSqlBatch.Builder
A configuration for running Apache Spark SQL queries as a batch workload.
Protobuf type google.cloud.dataproc.v1.SparkSqlBatch
SparkSqlJob
A Dataproc job for running Apache Spark SQL queries.
Protobuf type google.cloud.dataproc.v1.SparkSqlJob
SparkSqlJob.Builder
A Dataproc job for running Apache Spark SQL queries.
Protobuf type google.cloud.dataproc.v1.SparkSqlJob
StartClusterRequest
A request to start a cluster.
Protobuf type google.cloud.dataproc.v1.StartClusterRequest
StartClusterRequest.Builder
A request to start a cluster.
Protobuf type google.cloud.dataproc.v1.StartClusterRequest
StopClusterRequest
A request to stop a cluster.
Protobuf type google.cloud.dataproc.v1.StopClusterRequest
StopClusterRequest.Builder
A request to stop a cluster.
Protobuf type google.cloud.dataproc.v1.StopClusterRequest
SubmitJobRequest
A request to submit a job.
Protobuf type google.cloud.dataproc.v1.SubmitJobRequest
SubmitJobRequest.Builder
A request to submit a job.
Protobuf type google.cloud.dataproc.v1.SubmitJobRequest
TemplateParameter
A configurable parameter that replaces one or more fields in the template. Parameterizable fields:
- Labels
- File uris
- Job properties
- Job arguments
- Script variables
- Main class (in HadoopJob and SparkJob)
- Zone (in ClusterSelector)
Protobuf type google.cloud.dataproc.v1.TemplateParameter
TemplateParameter.Builder
A configurable parameter that replaces one or more fields in the template. Parameterizable fields:
- Labels
- File uris
- Job properties
- Job arguments
- Script variables
- Main class (in HadoopJob and SparkJob)
- Zone (in ClusterSelector)
Protobuf type google.cloud.dataproc.v1.TemplateParameter
UpdateAutoscalingPolicyRequest
A request to update an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest
UpdateAutoscalingPolicyRequest.Builder
A request to update an autoscaling policy.
Protobuf type google.cloud.dataproc.v1.UpdateAutoscalingPolicyRequest
UpdateClusterRequest
A request to update a cluster.
Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest
UpdateClusterRequest.Builder
A request to update a cluster.
Protobuf type google.cloud.dataproc.v1.UpdateClusterRequest
UpdateJobRequest
A request to update a job.
Protobuf type google.cloud.dataproc.v1.UpdateJobRequest
UpdateJobRequest.Builder
A request to update a job.
Protobuf type google.cloud.dataproc.v1.UpdateJobRequest
UpdateWorkflowTemplateRequest
A request to update a workflow template.
Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest
UpdateWorkflowTemplateRequest.Builder
A request to update a workflow template.
Protobuf type google.cloud.dataproc.v1.UpdateWorkflowTemplateRequest
ValueValidation
Validation based on a list of allowed values.
Protobuf type google.cloud.dataproc.v1.ValueValidation
ValueValidation.Builder
Validation based on a list of allowed values.
Protobuf type google.cloud.dataproc.v1.ValueValidation
VirtualClusterConfig
Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.
Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig
VirtualClusterConfig.Builder
Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster.
Protobuf type google.cloud.dataproc.v1.VirtualClusterConfig
WorkflowGraph
The workflow graph.
Protobuf type google.cloud.dataproc.v1.WorkflowGraph
WorkflowGraph.Builder
The workflow graph.
Protobuf type google.cloud.dataproc.v1.WorkflowGraph
WorkflowMetadata
A Dataproc workflow template resource.
Protobuf type google.cloud.dataproc.v1.WorkflowMetadata
WorkflowMetadata.Builder
A Dataproc workflow template resource.
Protobuf type google.cloud.dataproc.v1.WorkflowMetadata
WorkflowNode
The workflow node.
Protobuf type google.cloud.dataproc.v1.WorkflowNode
WorkflowNode.Builder
The workflow node.
Protobuf type google.cloud.dataproc.v1.WorkflowNode
WorkflowTemplate
A Dataproc workflow template resource.
Protobuf type google.cloud.dataproc.v1.WorkflowTemplate
WorkflowTemplate.Builder
A Dataproc workflow template resource.
Protobuf type google.cloud.dataproc.v1.WorkflowTemplate
WorkflowTemplateName
WorkflowTemplateName.Builder
Builder for projects/{project}/regions/{region}/workflowTemplates/{workflow_template}.
WorkflowTemplateName.ProjectLocationWorkflowTemplateBuilder
Builder for projects/{project}/locations/{location}/workflowTemplates/{workflow_template}.
WorkflowTemplatePlacement
Specifies workflow execution target.
Either managed_cluster or cluster_selector is required.
Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement
WorkflowTemplatePlacement.Builder
Specifies workflow execution target.
Either managed_cluster or cluster_selector is required.
Protobuf type google.cloud.dataproc.v1.WorkflowTemplatePlacement
WorkflowTemplateServiceClient
Service Description: The API interface for managing Workflow Templates in the Dataproc API.
This class provides the ability to make remote calls to the backing service through method calls that map to API methods. Sample code to get started:
try (WorkflowTemplateServiceClient workflowTemplateServiceClient =
WorkflowTemplateServiceClient.create()) {
LocationName parent = LocationName.of("[PROJECT]", "[LOCATION]");
WorkflowTemplate template = WorkflowTemplate.newBuilder().build();
WorkflowTemplate response =
workflowTemplateServiceClient.createWorkflowTemplate(parent, template);
}
Note: close() needs to be called on the WorkflowTemplateServiceClient object to clean up resources such as threads. In the example above, try-with-resources is used, which automatically calls close().
The surface of this class includes several types of Java methods for each of the API's methods:
- A "flattened" method. With this type of method, the fields of the request type have been converted into function parameters. It may be the case that not all fields are available as parameters, and not every API method will have a flattened method entry point.
- A "request object" method. This type of method only takes one parameter, a request object, which must be constructed before the call. Not every API method will have a request object method.
- A "callable" method. This type of method takes no parameters and returns an immutable API callable object, which can be used to initiate calls to the service.
See the individual methods for example code.
Many parameters require resource names to be formatted in a particular way. To assist with these names, this class includes a format method for each type of name, and additionally a parse method to extract the individual identifiers contained within names that are returned.
This class can be customized by passing in a custom instance of WorkflowTemplateServiceSettings to create(). For example:
To customize credentials:
WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
WorkflowTemplateServiceSettings.newBuilder()
.setCredentialsProvider(FixedCredentialsProvider.create(myCredentials))
.build();
WorkflowTemplateServiceClient workflowTemplateServiceClient =
WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);
To customize the endpoint:
WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
WorkflowTemplateServiceSettings.newBuilder().setEndpoint(myEndpoint).build();
WorkflowTemplateServiceClient workflowTemplateServiceClient =
WorkflowTemplateServiceClient.create(workflowTemplateServiceSettings);
Please refer to the GitHub repository's samples for more quickstart code snippets.
WorkflowTemplateServiceClient.ListWorkflowTemplatesFixedSizeCollection
WorkflowTemplateServiceClient.ListWorkflowTemplatesPage
WorkflowTemplateServiceClient.ListWorkflowTemplatesPagedResponse
WorkflowTemplateServiceGrpc
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateServiceGrpc.WorkflowTemplateServiceBlockingStub
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateServiceGrpc.WorkflowTemplateServiceFutureStub
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateServiceGrpc.WorkflowTemplateServiceImplBase
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateServiceGrpc.WorkflowTemplateServiceStub
The API interface for managing Workflow Templates in the Dataproc API.
WorkflowTemplateServiceSettings
Settings class to configure an instance of WorkflowTemplateServiceClient.
The default instance has everything set to sensible defaults:
- The default service address (dataproc.googleapis.com) and default port (443) are used.
- Credentials are acquired automatically through Application Default Credentials.
- Retries are configured for idempotent methods but not for non-idempotent methods.
The builder of this class is recursive, so contained classes are themselves builders. When build() is called, the tree of builders is called to create the complete settings object.
For example, to set the total timeout of createWorkflowTemplate to 30 seconds:
WorkflowTemplateServiceSettings.Builder workflowTemplateServiceSettingsBuilder =
WorkflowTemplateServiceSettings.newBuilder();
workflowTemplateServiceSettingsBuilder
.createWorkflowTemplateSettings()
.setRetrySettings(
workflowTemplateServiceSettingsBuilder
.createWorkflowTemplateSettings()
.getRetrySettings()
.toBuilder()
.setTotalTimeout(Duration.ofSeconds(30))
.build());
WorkflowTemplateServiceSettings workflowTemplateServiceSettings =
workflowTemplateServiceSettingsBuilder.build();
WorkflowTemplateServiceSettings.Builder
Builder for WorkflowTemplateServiceSettings.
WorkflowTemplatesProto
YarnApplication
A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>. Beta Feature: This report is available for testing purposes only. It may be changed before final release.
Protobuf type google.cloud.dataproc.v1.YarnApplication
YarnApplication.Builder
A YARN application created by a job. Application information is a subset of <code>org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto</code>. Beta Feature: This report is available for testing purposes only. It may be changed before final release.
Protobuf type google.cloud.dataproc.v1.YarnApplication
Interfaces
AcceleratorConfigOrBuilder
AutoscalingConfigOrBuilder
AutoscalingPolicyOrBuilder
AuxiliaryServicesConfigOrBuilder
BasicAutoscalingAlgorithmOrBuilder
BasicYarnAutoscalingConfigOrBuilder
Batch.StateHistoryOrBuilder
BatchOperationMetadataOrBuilder
BatchOrBuilder
CancelJobRequestOrBuilder
ClusterConfigOrBuilder
ClusterMetricsOrBuilder
ClusterOperationMetadataOrBuilder
ClusterOperationOrBuilder
ClusterOperationStatusOrBuilder
ClusterOrBuilder
ClusterSelectorOrBuilder
ClusterStatusOrBuilder
ConfidentialInstanceConfigOrBuilder
CreateAutoscalingPolicyRequestOrBuilder
CreateBatchRequestOrBuilder
CreateClusterRequestOrBuilder
CreateWorkflowTemplateRequestOrBuilder
DeleteAutoscalingPolicyRequestOrBuilder
DeleteBatchRequestOrBuilder
DeleteClusterRequestOrBuilder
DeleteJobRequestOrBuilder
DeleteWorkflowTemplateRequestOrBuilder
DiagnoseClusterRequestOrBuilder
DiagnoseClusterResultsOrBuilder
DiskConfigOrBuilder
EncryptionConfigOrBuilder
EndpointConfigOrBuilder
EnvironmentConfigOrBuilder
ExecutionConfigOrBuilder
GceClusterConfigOrBuilder
GetAutoscalingPolicyRequestOrBuilder
GetBatchRequestOrBuilder
GetClusterRequestOrBuilder
GetJobRequestOrBuilder
GetWorkflowTemplateRequestOrBuilder
GkeClusterConfigOrBuilder
GkeNodePoolConfig.GkeNodeConfigOrBuilder
GkeNodePoolConfig.GkeNodePoolAcceleratorConfigOrBuilder
GkeNodePoolConfig.GkeNodePoolAutoscalingConfigOrBuilder
GkeNodePoolConfigOrBuilder
GkeNodePoolTargetOrBuilder
HadoopJobOrBuilder
HiveJobOrBuilder
IdentityConfigOrBuilder
InstanceGroupAutoscalingPolicyConfigOrBuilder
InstanceGroupConfigOrBuilder
InstantiateInlineWorkflowTemplateRequestOrBuilder
InstantiateWorkflowTemplateRequestOrBuilder
JobMetadataOrBuilder
JobOrBuilder
JobPlacementOrBuilder
JobReferenceOrBuilder
JobSchedulingOrBuilder
JobStatusOrBuilder
KerberosConfigOrBuilder
KubernetesClusterConfigOrBuilder
KubernetesSoftwareConfigOrBuilder
LifecycleConfigOrBuilder
ListAutoscalingPoliciesRequestOrBuilder
ListAutoscalingPoliciesResponseOrBuilder
ListBatchesRequestOrBuilder
ListBatchesResponseOrBuilder
ListClustersRequestOrBuilder
ListClustersResponseOrBuilder
ListJobsRequestOrBuilder
ListJobsResponseOrBuilder
ListWorkflowTemplatesRequestOrBuilder
ListWorkflowTemplatesResponseOrBuilder
LoggingConfigOrBuilder
ManagedClusterOrBuilder
ManagedGroupConfigOrBuilder
MetastoreConfigOrBuilder
NodeGroupAffinityOrBuilder
NodeInitializationActionOrBuilder
OrderedJobOrBuilder
ParameterValidationOrBuilder
PeripheralsConfigOrBuilder
PigJobOrBuilder
PrestoJobOrBuilder
PySparkBatchOrBuilder
PySparkJobOrBuilder
QueryListOrBuilder
RegexValidationOrBuilder
ReservationAffinityOrBuilder
RuntimeConfigOrBuilder
RuntimeInfoOrBuilder
SecurityConfigOrBuilder
ShieldedInstanceConfigOrBuilder
SoftwareConfigOrBuilder
SparkBatchOrBuilder
SparkHistoryServerConfigOrBuilder
SparkJobOrBuilder
SparkRBatchOrBuilder
SparkRJobOrBuilder
SparkSqlBatchOrBuilder
SparkSqlJobOrBuilder
StartClusterRequestOrBuilder
StopClusterRequestOrBuilder
SubmitJobRequestOrBuilder
TemplateParameterOrBuilder
UpdateAutoscalingPolicyRequestOrBuilder
UpdateClusterRequestOrBuilder
UpdateJobRequestOrBuilder
UpdateWorkflowTemplateRequestOrBuilder
ValueValidationOrBuilder
VirtualClusterConfigOrBuilder
WorkflowGraphOrBuilder
WorkflowMetadataOrBuilder
WorkflowNodeOrBuilder
WorkflowTemplateOrBuilder
WorkflowTemplatePlacementOrBuilder
YarnApplicationOrBuilder
Enums
AutoscalingPolicy.AlgorithmCase
Batch.BatchConfigCase
Batch.State
The batch state.
Protobuf enum google.cloud.dataproc.v1.Batch.State
BatchOperationMetadata.BatchOperationType
Operation type for Batch resources
Protobuf enum google.cloud.dataproc.v1.BatchOperationMetadata.BatchOperationType
ClusterOperationStatus.State
The operation state.
Protobuf enum google.cloud.dataproc.v1.ClusterOperationStatus.State
ClusterStatus.State
The cluster state.
Protobuf enum google.cloud.dataproc.v1.ClusterStatus.State
ClusterStatus.Substate
The cluster substate.
Protobuf enum google.cloud.dataproc.v1.ClusterStatus.Substate
Component
Cluster components that can be activated.
Protobuf enum google.cloud.dataproc.v1.Component
ExecutionConfig.NetworkCase
FailureAction
Actions in response to failure of a resource associated with a cluster.
Protobuf enum google.cloud.dataproc.v1.FailureAction
GceClusterConfig.PrivateIpv6GoogleAccess
PrivateIpv6GoogleAccess controls whether and how Dataproc cluster nodes
can communicate with Google Services through gRPC over IPv6.
These values are directly mapped to corresponding values in the
Compute Engine Instance
fields.
Protobuf enum google.cloud.dataproc.v1.GceClusterConfig.PrivateIpv6GoogleAccess
GkeNodePoolTarget.Role
Role specifies whose tasks will run on the NodePool. The roles can be
specific to workloads. Exactly one GkeNodePoolTarget within the
VirtualCluster must have 'default' role, which is used to run all workloads
that are not associated with a NodePool.
Protobuf enum google.cloud.dataproc.v1.GkeNodePoolTarget.Role
HadoopJob.DriverCase
HiveJob.QueriesCase
InstanceGroupConfig.Preemptibility
Controls the use of preemptible instances within the group.
Protobuf enum google.cloud.dataproc.v1.InstanceGroupConfig.Preemptibility
Job.TypeJobCase
JobStatus.State
The job state.
Protobuf enum google.cloud.dataproc.v1.JobStatus.State
JobStatus.Substate
The job substate.
Protobuf enum google.cloud.dataproc.v1.JobStatus.Substate
KubernetesClusterConfig.ConfigCase
LifecycleConfig.TtlCase
ListJobsRequest.JobStateMatcher
A matcher that specifies categories of job states.
Protobuf enum google.cloud.dataproc.v1.ListJobsRequest.JobStateMatcher
LoggingConfig.Level
The Log4j level for job execution. When running an Apache Hive job, Cloud Dataproc configures the Hive client to an equivalent verbosity level.
Protobuf enum google.cloud.dataproc.v1.LoggingConfig.Level
OrderedJob.JobTypeCase
ParameterValidation.ValidationTypeCase
PigJob.QueriesCase
PrestoJob.QueriesCase
ReservationAffinity.Type
Indicates whether to consume capacity from an reservation or not.
Protobuf enum google.cloud.dataproc.v1.ReservationAffinity.Type
SparkBatch.DriverCase
SparkJob.DriverCase
SparkSqlJob.QueriesCase
VirtualClusterConfig.InfrastructureConfigCase
WorkflowMetadata.State
The operation state.
Protobuf enum google.cloud.dataproc.v1.WorkflowMetadata.State
WorkflowNode.NodeState
The workflow node state.
Protobuf enum google.cloud.dataproc.v1.WorkflowNode.NodeState
WorkflowTemplatePlacement.PlacementCase
YarnApplication.State
The application state, corresponding to <code>YarnProtos.YarnApplicationStateProto</code>.
Protobuf enum google.cloud.dataproc.v1.YarnApplication.State