API documentation for dataproc_v1.types package.
Classes
AcceleratorConfig
Specifies the type and number of accelerator cards attached to the
instances of an instance. See GPUs on Compute
Engine <https://cloud.google.com/compute/docs/gpus/>__.
AutoscalingConfig
Autoscaling Policy config associated with the cluster.
AutoscalingPolicy
Describes an autoscaling policy for Dataproc cluster autoscaler.
BasicAutoscalingAlgorithm
Basic algorithm for autoscaling.
BasicYarnAutoscalingConfig
Basic autoscaling configurations for YARN.
CancelJobRequest
A request to cancel a job.
Cluster
Describes the identifying information, config, and status of a cluster of Compute Engine instances.
ClusterConfig
The cluster config.
ClusterMetrics
Contains cluster daemon metrics, such as HDFS and YARN stats.
Beta Feature: This report is available for testing purposes only. It may be changed before final release.
ClusterOperation
The cluster operation triggered by a workflow.
ClusterOperationMetadata
Metadata describing the operation.
ClusterOperationStatus
The status of the operation.
ClusterSelector
A selector that chooses target cluster for jobs based on metadata.
ClusterStatus
The status of a cluster and its instances.
CreateAutoscalingPolicyRequest
A request to create an autoscaling policy.
CreateClusterRequest
A request to create a cluster.
CreateWorkflowTemplateRequest
A request to create a workflow template.
DeleteAutoscalingPolicyRequest
A request to delete an autoscaling policy. Autoscaling policies in use by one or more clusters will not be deleted.
DeleteClusterRequest
A request to delete a cluster.
DeleteJobRequest
A request to delete a job.
DeleteWorkflowTemplateRequest
A request to delete a workflow template. Currently started workflows will remain running.
DiagnoseClusterRequest
A request to collect cluster diagnostic information.
DiagnoseClusterResults
The location of diagnostic output.
DiskConfig
Specifies the config of disk options for a group of VM instances.
EncryptionConfig
Encryption settings for the cluster.
EndpointConfig
Endpoint config for this cluster
GceClusterConfig
Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.
GetAutoscalingPolicyRequest
A request to fetch an autoscaling policy.
GetClusterRequest
Request to get the resource representation for a cluster in a project.
GetJobRequest
A request to get the resource representation for a job in a project.
GetWorkflowTemplateRequest
A request to fetch a workflow template.
HadoopJob
A Dataproc job for running Apache Hadoop
MapReduce <https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html>
jobs on Apache Hadoop
YARN <https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html>.
HiveJob
A Dataproc job for running Apache
Hive <https://hive.apache.org/>__ queries on YARN.
InstanceGroupAutoscalingPolicyConfig
Configuration for the size bounds of an instance group, including its proportional size to other groups.
InstanceGroupConfig
The config settings for Compute Engine resources in an instance group, such as a master or worker group.
InstantiateInlineWorkflowTemplateRequest
A request to instantiate an inline workflow template.
InstantiateWorkflowTemplateRequest
A request to instantiate a workflow template.
Job
A Dataproc job resource.
JobMetadata
Job Operation metadata.
JobPlacement
Dataproc job config.
JobReference
Encapsulates the full scoping used to reference a job.
JobScheduling
Job scheduling options.
JobStatus
Dataproc job status.
KerberosConfig
Specifies Kerberos related configuration.
LifecycleConfig
Specifies the cluster auto-delete schedule configuration.
ListAutoscalingPoliciesRequest
A request to list autoscaling policies in a project.
ListAutoscalingPoliciesResponse
A response to a request to list autoscaling policies in a project.
ListClustersRequest
A request to list the clusters in a project.
ListClustersResponse
The list of all clusters in a project.
ListJobsRequest
A request to list jobs in a project.
ListJobsResponse
A list of jobs in a project.
ListWorkflowTemplatesRequest
A request to list workflow templates in a project.
ListWorkflowTemplatesResponse
A response to a request to list workflow templates in a project.
LoggingConfig
The runtime logging config of the job.
ManagedCluster
Cluster that is managed by the workflow.
ManagedGroupConfig
Specifies the resources used to actively manage an instance group.
NodeInitializationAction
Specifies an executable to run on a fully configured node and a timeout period for executable completion.
OrderedJob
A job executed by the workflow.
ParameterValidation
Configuration for parameter validation.
PigJob
A Dataproc job for running Apache Pig <https://pig.apache.org/>__
queries on YARN.
PrestoJob
A Dataproc job for running Presto <https://prestosql.io/>
queries. IMPORTANT: The Dataproc Presto Optional
Component <https://cloud.google.com/dataproc/docs/concepts/components/presto>
must be enabled when the cluster is created to submit a Presto job
to the cluster.
PySparkJob
A Dataproc job for running Apache
PySpark <https://spark.apache.org/docs/0.9.0/python-programming-guide.html>__
applications on YARN.
QueryList
A list of queries to run on a cluster.
RegexValidation
Validation based on regular expressions.
ReservationAffinity
Reservation Affinity for consuming Zonal reservation.
SecurityConfig
Security related configuration, including Kerberos.
SoftwareConfig
Specifies the selection and config of software inside the cluster.
SparkJob
A Dataproc job for running Apache
Spark <http://spark.apache.org/>__ applications on YARN.
SparkRJob
A Dataproc job for running Apache
SparkR <https://spark.apache.org/docs/latest/sparkr.html>__
applications on YARN.
SparkSqlJob
A Dataproc job for running Apache Spark
SQL <http://spark.apache.org/sql/>__ queries.
SubmitJobRequest
A request to submit a job.
TemplateParameter
A configurable parameter that replaces one or more fields in the template. Parameterizable fields:
- Labels
- File uris
- Job properties
- Job arguments
- Script variables
- Main class (in HadoopJob and SparkJob)
- Zone (in ClusterSelector)
UpdateAutoscalingPolicyRequest
A request to update an autoscaling policy.
UpdateClusterRequest
A request to update a cluster.
UpdateJobRequest
A request to update a job.
UpdateWorkflowTemplateRequest
A request to update a workflow template.
ValueValidation
Validation based on a list of allowed values.
WorkflowGraph
The workflow graph.
WorkflowMetadata
A Dataproc workflow template resource.
WorkflowNode
The workflow node.
WorkflowTemplate
A Dataproc workflow template resource.
WorkflowTemplatePlacement
Specifies workflow execution target.
Either managed_cluster or cluster_selector is required.
YarnApplication
A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
Beta Feature: This report is available for testing purposes only. It may be changed before final release.