Class PigJob (2.2.0)

PigJob(mapping=None, *, ignore_unknown_fields=False, **kwargs)

A Dataproc job for running Apache Pig <https://pig.apache.org/>__ queries on YARN.

Attributes

NameDescription
query_file_uri str
The HCFS URI of the script that contains the Pig queries.
query_list .gcd_jobs.QueryList
A list of queries.
continue_on_failure bool
Optional. Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries.
script_variables Sequence[.gcd_jobs.PigJob.ScriptVariablesEntry]
Optional. Mapping of query variable names to values (equivalent to the Pig command: name=[value]).
properties Sequence[.gcd_jobs.PigJob.PropertiesEntry]
Optional. A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
jar_file_uris Sequence[str]
Optional. HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
logging_config .gcd_jobs.LoggingConfig
Optional. The runtime log config for job execution.

Classes

PropertiesEntry

PropertiesEntry(mapping=None, *, ignore_unknown_fields=False, **kwargs)

The abstract base class for a message.

Parameters
NameDescription
kwargs dict

Keys and values corresponding to the fields of the message.

mapping Union[dict, .Message]

A dictionary or message to be used to determine the values for this message.

ignore_unknown_fields Optional(bool)

If True, do not raise errors for unknown fields. Only applied if mapping is a mapping type or there are keyword parameters.

ScriptVariablesEntry

ScriptVariablesEntry(mapping=None, *, ignore_unknown_fields=False, **kwargs)

The abstract base class for a message.

Parameters
NameDescription
kwargs dict

Keys and values corresponding to the fields of the message.

mapping Union[dict, .Message]

A dictionary or message to be used to determine the values for this message.

ignore_unknown_fields Optional(bool)

If True, do not raise errors for unknown fields. Only applied if mapping is a mapping type or there are keyword parameters.