REST Resource: projects.locations.batches

Stay organized with collections Save and categorize content based on your preferences.

Resource: Batch

A representation of a batch workload in the service.

JSON representation
{
  "name": string,
  "uuid": string,
  "createTime": string,
  "runtimeInfo": {
    object (RuntimeInfo)
  },
  "state": enum (State),
  "stateMessage": string,
  "stateTime": string,
  "creator": string,
  "labels": {
    string: string,
    ...
  },
  "runtimeConfig": {
    object (RuntimeConfig)
  },
  "environmentConfig": {
    object (EnvironmentConfig)
  },
  "operation": string,
  "stateHistory": [
    {
      object (StateHistory)
    }
  ],

  // Union field batch_config can be only one of the following:
  "pysparkBatch": {
    object (PySparkBatch)
  },
  "sparkBatch": {
    object (SparkBatch)
  },
  "sparkRBatch": {
    object (SparkRBatch)
  },
  "sparkSqlBatch": {
    object (SparkSqlBatch)
  }
  // End of list of possible types for union field batch_config.
}
Fields
name

string

Output only. The resource name of the batch.

uuid

string

Output only. A batch UUID (Unique Universal Identifier). The service generates this value when it creates the batch.

createTime

string (Timestamp format)

Output only. The time when the batch was created.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

runtimeInfo

object (RuntimeInfo)

Output only. Runtime information about batch execution.

state

enum (State)

Output only. The state of the batch.

stateMessage

string

Output only. Batch state details, such as a failure description if the state is FAILED.

stateTime

string (Timestamp format)

Output only. The time when the batch entered a current state.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

creator

string

Output only. The email address of the user who created the batch.

labels

map (key: string, value: string)

Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035. Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035. No more than 32 labels can be associated with a batch.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

runtimeConfig

object (RuntimeConfig)

Optional. Runtime configuration for the batch execution.

environmentConfig

object (EnvironmentConfig)

Optional. Environment configuration for the batch execution.

operation

string

Output only. The resource name of the operation associated with this batch.

stateHistory[]

object (StateHistory)

Output only. Historical state information for the batch.

Union field batch_config. The application/framework-specific portion of the batch configuration. batch_config can be only one of the following:
pysparkBatch

object (PySparkBatch)

Optional. PySpark batch config.

sparkBatch

object (SparkBatch)

Optional. Spark batch config.

sparkRBatch

object (SparkRBatch)

Optional. SparkR batch config.

sparkSqlBatch

object (SparkSqlBatch)

Optional. SparkSql batch config.

PySparkBatch

A configuration for running an Apache PySpark batch workload.

JSON representation
{
  "mainPythonFileUri": string,
  "args": [
    string
  ],
  "pythonFileUris": [
    string
  ],
  "jarFileUris": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ]
}
Fields
mainPythonFileUri

string

Required. The HCFS URI of the main Python file to use as the Spark driver. Must be a .py file.

args[]

string

Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission.

pythonFileUris[]

string

Optional. HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.

fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor.

archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

SparkBatch

A configuration for running an Apache Spark batch workload.

JSON representation
{
  "args": [
    string
  ],
  "jarFileUris": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ],

  // Union field driver can be only one of the following:
  "mainJarFileUri": string,
  "mainClass": string
  // End of list of possible types for union field driver.
}
Fields
args[]

string

Optional. The arguments to pass to the driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission.

jarFileUris[]

string

Optional. HCFS URIs of jar files to add to the classpath of the Spark driver and tasks.

fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor.

archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Union field driver. The specification of the main method to call to drive the Spark workload. Specify either the jar file that contains the main class or the main class name. To pass both a main jar and a main class in that jar, add the jar to jar_file_uris, and then specify the main class name in main_class. driver can be only one of the following:
mainJarFileUri

string

Optional. The HCFS URI of the jar file that contains the main class.

mainClass

string

Optional. The name of the driver main class. The jar file that contains the class must be in the classpath or specified in jarFileUris.

SparkRBatch

A configuration for running an Apache SparkR batch workload.

JSON representation
{
  "mainRFileUri": string,
  "args": [
    string
  ],
  "fileUris": [
    string
  ],
  "archiveUris": [
    string
  ]
}
Fields
mainRFileUri

string

Required. The HCFS URI of the main R file to use as the driver. Must be a .R or .r file.

args[]

string

Optional. The arguments to pass to the Spark driver. Do not include arguments that can be set as batch properties, such as --conf, since a collision can occur that causes an incorrect batch submission.

fileUris[]

string

Optional. HCFS URIs of files to be placed in the working directory of each executor.

archiveUris[]

string

Optional. HCFS URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

SparkSqlBatch

A configuration for running Apache Spark SQL queries as a batch workload.

JSON representation
{
  "queryFileUri": string,
  "queryVariables": {
    string: string,
    ...
  },
  "jarFileUris": [
    string
  ]
}
Fields
queryFileUri

string

Required. The HCFS URI of the script that contains Spark SQL queries to execute.

queryVariables

map (key: string, value: string)

Optional. Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

jarFileUris[]

string

Optional. HCFS URIs of jar files to be added to the Spark CLASSPATH.

RuntimeInfo

Runtime information about workload execution.

JSON representation
{
  "endpoints": {
    string: string,
    ...
  },
  "outputUri": string,
  "diagnosticOutputUri": string
}
Fields
endpoints

map (key: string, value: string)

Output only. Map of remote access endpoints (such as web interfaces and APIs) to their URIs.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

outputUri

string

Output only. A URI pointing to the location of the stdout and stderr of the workload.

diagnosticOutputUri

string

Output only. A URI pointing to the location of the diagnostics tarball.

State

The batch state.

Enums
STATE_UNSPECIFIED The batch state is unknown.
PENDING The batch is created before running.
RUNNING The batch is running.
CANCELLING The batch is cancelling.
CANCELLED The batch cancellation was successful.
SUCCEEDED The batch completed successfully.
FAILED The batch is no longer running due to an error.

RuntimeConfig

Runtime configuration for a workload.

JSON representation
{
  "version": string,
  "containerImage": string,
  "properties": {
    string: string,
    ...
  }
}
Fields
version

string

Optional. Version of the batch runtime.

containerImage

string

Optional. Optional custom container image for the job runtime environment. If not specified, a default container image will be used.

properties

map (key: string, value: string)

Optional. A mapping of property names to values, which are used to configure workload execution.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

EnvironmentConfig

Environment configuration for a workload.

JSON representation
{
  "executionConfig": {
    object (ExecutionConfig)
  },
  "peripheralsConfig": {
    object (PeripheralsConfig)
  }
}
Fields
executionConfig

object (ExecutionConfig)

Optional. Execution configuration for a workload.

peripheralsConfig

object (PeripheralsConfig)

Optional. Peripherals configuration that workload has access to.

ExecutionConfig

Execution configuration for a workload.

JSON representation
{
  "serviceAccount": string,
  "networkTags": [
    string
  ],
  "kmsKey": string,

  // Union field network can be only one of the following:
  "networkUri": string,
  "subnetworkUri": string
  // End of list of possible types for union field network.
}
Fields
serviceAccount

string

Optional. Service account that used to execute workload.

networkTags[]

string

Optional. Tags used for network traffic control.

kmsKey

string

Optional. The Cloud KMS key to use for encryption.

Union field network. Network configuration for workload execution. network can be only one of the following:
networkUri

string

Optional. Network URI to connect workload to.

subnetworkUri

string

Optional. Subnetwork URI to connect workload to.

PeripheralsConfig

Auxiliary services configuration for a workload.

JSON representation
{
  "metastoreService": string,
  "sparkHistoryServerConfig": {
    object (SparkHistoryServerConfig)
  }
}
Fields
metastoreService

string

Optional. Resource name of an existing Dataproc Metastore service.

Example:

  • projects/[projectId]/locations/[region]/services/[service_id]
sparkHistoryServerConfig

object (SparkHistoryServerConfig)

Optional. The Spark History Server configuration for the workload.

SparkHistoryServerConfig

Spark History Server configuration for the workload.

JSON representation
{
  "dataprocCluster": string
}
Fields
dataprocCluster

string

Optional. Resource name of an existing Dataproc Cluster to act as a Spark History Server for the workload.

Example:

  • projects/[projectId]/regions/[region]/clusters/[clusterName]

StateHistory

Historical state information.

JSON representation
{
  "state": enum (State),
  "stateMessage": string,
  "stateStartTime": string
}
Fields
state

enum (State)

Output only. The state of the batch at this point in history.

stateMessage

string

Output only. Details about the state at this point in history.

stateStartTime

string (Timestamp format)

Output only. The time when the batch entered the historical state.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

Methods

create

Creates a batch workload that executes asynchronously.

delete

Deletes the batch workload resource.

get

Gets the batch workload resource representation.

list

Lists batch workloads.