gcloud dataproc jobs submit spark-sql

NAME
gcloud dataproc jobs submit spark-sql - submit a Spark SQL job to a cluster
SYNOPSIS
gcloud dataproc jobs submit spark-sql (--cluster=CLUSTER     | --cluster-labels=[KEY=VALUE,…]) (--execute=QUERY, -e QUERY     | --file=FILE, -f FILE) [--async] [--bucket=BUCKET] [--driver-log-levels=[PACKAGE=LEVEL,…]] [--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB] [--driver-required-vcores=DRIVER_REQUIRED_VCORES] [--jars=[JAR,…]] [--labels=[KEY=VALUE,…]] [--max-failures-per-hour=MAX_FAILURES_PER_HOUR] [--max-failures-total=MAX_FAILURES_TOTAL] [--params=[PARAM=VALUE,…]] [--properties=[PROPERTY=VALUE,…]] [--properties-file=PROPERTIES_FILE] [--region=REGION] [GCLOUD_WIDE_FLAG …]
DESCRIPTION
Submit a Spark SQL job to a cluster.
EXAMPLES
To submit a Spark SQL job with a local script, run: 
gcloud dataproc jobs submit spark-sql --cluster=my-cluster --file=my_queries.ql

To submit a Spark SQL job with inline queries, run: 

gcloud dataproc jobs submit spark-sql --cluster=my-cluster -e="CREATE EXTERNAL TABLE foo(bar int) LOCATION 'gs://my_bucket/'" -e="SELECT * FROM foo WHERE bar > 2"
REQUIRED FLAGS
Exactly one of these must be specified:
--cluster=CLUSTER
The Dataproc cluster to submit the job to.
--cluster-labels=[KEY=VALUE,…]
List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens (-), underscores (_), lowercase characters, and numbers. Values must contain only hyphens (-), underscores (_), lowercase characters, and numbers.

Labels of Dataproc cluster on which to place the job.

Exactly one of these must be specified:
--execute=QUERY, -e QUERY
A Spark SQL query to execute as part of the job.
--file=FILE, -f FILE
HCFS URI of file containing Spark SQL script to execute as the job.
OPTIONAL FLAGS
--async
Return immediately, without waiting for the operation in progress to complete.
--bucket=BUCKET
The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--driver-log-levels=[PACKAGE=LEVEL,…]
A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO
--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB
The memory allocation requested by the job driver in megabytes (MB) for execution on the driver node group (it is used only by clusters with a driver node group).
--driver-required-vcores=DRIVER_REQUIRED_VCORES
The vCPU allocation requested by the job driver for execution on the driver node group (it is used only by clusters with a driver node group).
--jars=[JAR,…]
Comma separated list of jar files to be provided to the executor and driver classpaths. May contain UDFs.
--labels=[KEY=VALUE,…]
List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens (-), underscores (_), lowercase characters, and numbers. Values must contain only hyphens (-), underscores (_), lowercase characters, and numbers.

--max-failures-per-hour=MAX_FAILURES_PER_HOUR
Specifies the maximum number of times a job can be restarted per hour in event of failure. Default is 0 (no retries after job failure).
--max-failures-total=MAX_FAILURES_TOTAL
Specifies the maximum total number of times a job can be restarted after the job fails. Default is 0 (no retries after job failure).
--params=[PARAM=VALUE,…]
A list of key value pairs to set variables in the Hive queries.
--properties=[PROPERTY=VALUE,…]
A list of key value pairs to configure Hive.
--properties-file=PROPERTIES_FILE
Path to a local file or a file in a Cloud Storage bucket containing configuration properties for the job. The client machine running this command must have read permission to the file.

Specify properties in the form of property=value in the text file. For example: 

  # Properties to set for the job:
  key1=value1
  key2=value2
  # Comment out properties not used.
  # key3=value3

If a property is set in both --properties and --properties-file, the value defined in --properties takes precedence.

--region=REGION
Dataproc region to use. Each Dataproc region constitutes an independent resource namespace constrained to deploying instances into Compute Engine zones inside the region. Overrides the default dataproc/region property value for this command invocation.
GCLOUD WIDE FLAGS
These flags are available to all commands: --access-token-file, --account, --billing-project, --configuration, --flags-file, --flatten, --format, --help, --impersonate-service-account, --log-http, --project, --quiet, --trace-token, --user-output-enabled, --verbosity.

Run $ gcloud help for details.

NOTES
These variants are also available: 
gcloud alpha dataproc jobs submit spark-sql
 
gcloud beta dataproc jobs submit spark-sql