Class Task.Types.SparkTaskConfig (1.0.0)

public sealed class SparkTaskConfig : IMessage<Task.Types.SparkTaskConfig>, IEquatable<Task.Types.SparkTaskConfig>, IDeepCloneable<Task.Types.SparkTaskConfig>, IBufferMessage, IMessage

User-specified config for running a Spark task.

Inheritance

Object > Task.Types.SparkTaskConfig

Namespace

Google.Cloud.Dataplex.V1

Assembly

Google.Cloud.Dataplex.V1.dll

Constructors

SparkTaskConfig()

public SparkTaskConfig()

SparkTaskConfig(Task.Types.SparkTaskConfig)

public SparkTaskConfig(Task.Types.SparkTaskConfig other)
Parameter
NameDescription
otherTask.Types.SparkTaskConfig

Properties

ArchiveUris

public RepeatedField<string> ArchiveUris { get; }

Optional. Cloud Storage URIs of archives to be extracted into the working directory of each executor. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.

Property Value
TypeDescription
RepeatedField<String>

DriverCase

public Task.Types.SparkTaskConfig.DriverOneofCase DriverCase { get; }
Property Value
TypeDescription
Task.Types.SparkTaskConfig.DriverOneofCase

FileUris

public RepeatedField<string> FileUris { get; }

Optional. Cloud Storage URIs of files to be placed in the working directory of each executor.

Property Value
TypeDescription
RepeatedField<String>

InfrastructureSpec

public Task.Types.InfrastructureSpec InfrastructureSpec { get; set; }

Optional. Infrastructure specification for the execution.

Property Value
TypeDescription
Task.Types.InfrastructureSpec

MainClass

public string MainClass { get; set; }

The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in jar_file_uris. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
TypeDescription
String

MainJarFileUri

public string MainJarFileUri { get; set; }

The Cloud Storage URI of the jar file that contains the main class. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
TypeDescription
String

PythonScriptFile

public string PythonScriptFile { get; set; }

The Gcloud Storage URI of the main Python file to use as the driver. Must be a .py file. The execution args are passed in as a sequence of named process arguments (--key=value).

Property Value
TypeDescription
String

SqlScript

public string SqlScript { get; set; }

The query text. The execution args are used to declare a set of script variables (set key=&quot;value&quot;;).

Property Value
TypeDescription
String

SqlScriptFile

public string SqlScriptFile { get; set; }

A reference to a query file. This can be the Cloud Storage URI of the query file or it can the path to a SqlScript Content. The execution args are used to declare a set of script variables (set key=&quot;value&quot;;).

Property Value
TypeDescription
String