- NAME
-
- gcloud dataproc jobs submit - submit Dataproc jobs to execute on a cluster
- SYNOPSIS
-
-
gcloud dataproc jobs submitCOMMAND[--async] [--bucket=BUCKET] [--region=REGION] [GCLOUD_WIDE_FLAG …]
-
- DESCRIPTION
- Submit Dataproc jobs to execute on a cluster.
- EXAMPLES
-
To submit a Hadoop MapReduce job, run:
gcloud dataproc jobs submit hadoop --cluster my-cluster --jar my_jar.jar -- arg1 arg2To submit a Spark Scala or Java job, run:
gcloud dataproc jobs submit spark --cluster my-cluster --jar my_jar.jar -- arg1 arg2To submit a PySpark job, run:
gcloud dataproc jobs submit pyspark --cluster my-cluster my_script.py -- arg1 arg2To submit a Spark SQL job, run:
gcloud dataproc jobs submit spark-sql --cluster my-cluster --file my_queries.qTo submit a Pig job, run:
gcloud dataproc jobs submit pig --cluster my-cluster --file my_script.pigTo submit a Hive job, run:
gcloud dataproc jobs submit hive --cluster my-cluster --file my_queries.q - FLAGS
-
--async- Return immediately, without waiting for the operation in progress to complete.
--bucket=BUCKET- The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--region=REGION-
Dataproc region to use. Each Dataproc region constitutes an independent resource
namespace constrained to deploying instances into Compute Engine zones inside
the region. Overrides the default
dataproc/regionproperty value for this command invocation.
- GCLOUD WIDE FLAGS
-
These flags are available to all commands:
--help.Run
$ gcloud helpfor details. - COMMANDS
-
COMMANDflink- Submit a Flink job to a cluster.
hadoop- Submit a Hadoop job to a cluster.
hive- Submit a Hive job to a cluster.
pig- Submit a Pig job to a cluster.
presto- Submit a Presto job to a cluster.
pyspark- Submit a PySpark job to a cluster.
spark- Submit a Spark job to a cluster.
spark-r- Submit a SparkR job to a cluster.
spark-sql- Submit a Spark SQL job to a cluster.
trino- Submit a Trino job to a cluster.
- NOTES
-
These variants are also available:
gcloud alpha dataproc jobs submitgcloud beta dataproc jobs submit
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-02-06 UTC.