Class JobControllerClient (0.8.2)

JobControllerClient(
    transport=None,
    channel=None,
    credentials=None,
    client_config=None,
    client_info=None,
    client_options=None,
)

The JobController provides methods to manage jobs.

Methods

JobControllerClient

JobControllerClient(
    transport=None,
    channel=None,
    credentials=None,
    client_config=None,
    client_info=None,
    client_options=None,
)

Constructor.

Parameters
NameDescription
channel grpc.Channel

DEPRECATED. A Channel instance through which to make calls. This argument is mutually exclusive with credentials; providing both will raise an exception.

credentials google.auth.credentials.Credentials

The authorization credentials to attach to requests. These credentials identify this application to the service. If none are specified, the client will attempt to ascertain the credentials from the environment. This argument is mutually exclusive with providing a transport instance to transport; doing so will raise an exception.

client_config dict

DEPRECATED. A dictionary of call options for each method. If not specified, the default configuration is used.

client_info google.api_core.gapic_v1.client_info.ClientInfo

The client info used to send a user-agent string along with API requests. If None, then default info will be used. Generally, you only need to set this if you're developing your own client library.

client_options Union[dict, google.api_core.client_options.ClientOptions]

Client options used to set user options on the client. API Endpoint should be set through client_options.

cancel_job

cancel_job(project_id, region, job_id, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Starts a job cancellation request. To access the job resource after cancellation, call regions/{region}/jobs.list <https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.jobs/list> or regions/{region}/jobs.get <https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.jobs/get>.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job_id:

job_id = ''

response = client.cancel_job(project_id, region, job_id)

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job_id str

Required. The job ID.

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

delete_job

delete_job(project_id, region, job_id, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Deletes the job from the project. If the job is active, the delete fails, and the response returns FAILED_PRECONDITION.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job_id:

job_id = ''

client.delete_job(project_id, region, job_id)

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job_id str

Required. The job ID.

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

from_service_account_file

from_service_account_file(filename, *args, **kwargs)

Creates an instance of this client using the provided credentials file.

Parameter
NameDescription
filename str

The path to the service account private key json file.

Returns
TypeDescription
JobControllerClientThe constructed client.

from_service_account_json

from_service_account_json(filename, *args, **kwargs)

Creates an instance of this client using the provided credentials file.

Parameter
NameDescription
filename str

The path to the service account private key json file.

Returns
TypeDescription
JobControllerClientThe constructed client.

get_job

get_job(project_id, region, job_id, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Gets the resource representation for a job in a project.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job_id:

job_id = ''

response = client.get_job(project_id, region, job_id)

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job_id str

Required. The job ID.

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

list_jobs

list_jobs(project_id, region, page_size=None, cluster_name=None, job_state_matcher=None, filter_=None, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Lists regions/{region}/jobs in a project.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

Iterate over all results

for element in client.list_jobs(project_id, region): ... # process element ... pass

Alternatively:

Iterate over results one page at a time

for page in client.list_jobs(project_id, region).pages: ... for element in page: ... # process element ... pass

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

page_size int

The maximum number of resources contained in the underlying API response. If page streaming is performed per- resource, this parameter does not affect the return value. If page streaming is performed per-page, this determines the maximum number of resources in a page.

cluster_name str

Optional. If set, the returned jobs list includes only jobs that were submitted to the named cluster.

job_state_matcher JobStateMatcher

Optional. Specifies enumerated categories of jobs to list. (default = match ALL jobs). If filter is provided, jobStateMatcher will be ignored.

filter_ str

Optional. A filter constraining the jobs to list. Filters are case-sensitive and have the following syntax: [field = value] AND [field [= value]] ... where field is status.state or labels.[KEY], and [KEY] is a label key. value can be * to match all values. status.state can be either ACTIVE or NON_ACTIVE. Only the logical AND operator is supported; space-separated items are treated as having an implicit AND operator. Example filter: status.state = ACTIVE AND labels.env = staging AND labels.starred = *

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

submit_job

submit_job(project_id, region, job, request_id=None, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Submits a job to a cluster.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job:

job = {}

response = client.submit_job(project_id, region, job)

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job Union[dict, Job]

Required. The job resource. If a dict is provided, it must be of the same form as the protobuf message Job

request_id str

Optional. A unique id used to identify the request. If the server receives two SubmitJobRequest requests with the same id, then the second request will be ignored and the first Job created and stored in the backend is returned. It is recommended to always set this value to a UUID https://en.wikipedia.org/wiki/Universally_unique_identifier_. The id must contain only letters (a-z, A-Z), numbers (0-9), underscores (), and hyphens (-). The maximum length is 40 characters.

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

submit_job_as_operation

submit_job_as_operation(project_id, region, job, request_id=None, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Submits job to a cluster.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job:

job = {}

response = client.submit_job_as_operation(project_id, region, job)

def callback(operation_future): ... # Handle result. ... result = operation_future.result()

response.add_done_callback(callback)

Handle metadata.

metadata = response.metadata()

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job Union[dict, Job]

Required. The job resource. If a dict is provided, it must be of the same form as the protobuf message Job

request_id str

Optional. A unique id used to identify the request. If the server receives two SubmitJobRequest requests with the same id, then the second request will be ignored and the first Job created and stored in the backend is returned. It is recommended to always set this value to a UUID https://en.wikipedia.org/wiki/Universally_unique_identifier_. The id must contain only letters (a-z, A-Z), numbers (0-9), underscores (), and hyphens (-). The maximum length is 40 characters.

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.

update_job

update_job(project_id, region, job_id, job, update_mask, retry=<_MethodDefault._DEFAULT_VALUE: <object object>>, timeout=<_MethodDefault._DEFAULT_VALUE: <object object>>, metadata=None)

Updates a job in a project.

.. rubric:: Example

from google.cloud import dataproc_v1beta2

client = dataproc_v1beta2.JobControllerClient()

TODO: Initialize project_id:

project_id = ''

TODO: Initialize region:

region = ''

TODO: Initialize job_id:

job_id = ''

TODO: Initialize job:

job = {}

TODO: Initialize update_mask:

update_mask = {}

response = client.update_job(project_id, region, job_id, job, update_mask)

Parameters
NameDescription
project_id str

Required. The ID of the Google Cloud Platform project that the job belongs to.

region str

Required. The Dataproc region in which to handle the request.

job_id str

Required. The job ID.

job Union[dict, Job]

Required. The changes to the job. If a dict is provided, it must be of the same form as the protobuf message Job

update_mask Union[dict, FieldMask]

Required. Specifies the path, relative to Job, of the field to update. For example, to update the labels of a Job the update_mask parameter would be specified as labels, and the PATCH request body would specify the new value. Note: Currently, labels is the only field that can be updated. If a dict is provided, it must be of the same form as the protobuf message FieldMask

retry Optional[google.api_core.retry.Retry]

A retry object used to retry requests. If None is specified, requests will be retried using a default configuration.

timeout Optional[float]

The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata Optional[Sequence[Tuple[str, str]]]

Additional metadata that is provided to the method.

Exceptions
TypeDescription
google.api_core.exceptions.GoogleAPICallErrorIf the request failed for any reason.
google.api_core.exceptions.RetryErrorIf the request failed due to a retryable error and retry attempts failed.
ValueErrorIf the parameters are invalid.