Managing BigQuery Jobs

After you submit a BigQuery job, you can view job data, list jobs, cancel a job, or rerun a job.

When a job is submitted, it can be in one of three states:

  • PENDING — scheduled
  • RUNNING
  • DONE — reported as SUCCESS or FAILURE (if the job completed with errors)

Viewing job data

You can view job data and metadata by using the web UI, CLI, and API. This data includes details such as the job type, the job state, and the user who ran the job.

Required permissions

In order to get job data and metadata, you must have bigquery.jobs.get permissions. The following project-level, predefined IAM role includes bigquery.jobs.get permissions:

If you grant an account the bigquery.admin role, the user can view all job data in the project no matter who submitted the job.

The following roles are granted bigquery.jobs.get permissions for self-created jobs. These users can only view job data for jobs they submit:

For more information on IAM roles and permissions in BigQuery, see access control.

Viewing information about jobs

To view information about a job:

Web UI

  1. In the navigation pane, click Job History.

  2. In the Recent Jobs section, click the job to view the details.

CLI

Issue the bq show command with the -j flag and the job_id parameter. Supply the --location flag and set the value to your location.

The following command requests information about a job:

bq --location=[LOCATION] show -j [JOB_ID]

Where:

  • [LOCATION] is the name of the location where the job runs. For example, if you are using BigQuery in the Tokyo region, set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  • [JOB_ID] is the ID of the job.

Examples:

The following command gets summary information about job bquijob_123x456_123y123z123c running in myproject in the US multi-region location:

bq --location=US show -j myproject:bquijob_123x456_123y123z123c

The output looks like the following:

 Job Type    State      Start Time      Duration      User Email       Bytes Processed   Bytes Billed   Billing Tier   Labels


extract SUCCESS 06 Jul 11:32:10 0:01:41 user@example.com

To see full job details, enter:

bq --location=US show --format=prettyjson -j myproject:bquijob_123x456_789y123z456c

The output looks like the following:

{
  "configuration": {
    "extract": {
      "compression": "NONE",
      "destinationUri": "[URI removed]",
      "destinationUris": [
        "[URI removed]"
      ],
      "sourceTable": {
        "datasetId": "github_repos",
        "projectId": "bigquery-public-data",
        "tableId": "commits"
      }
    }
  },
  "etag": "\"[etag removed]\"",
  "id": "myproject:bquijob_123x456_789y123z456c",
  "jobReference": {
    "jobId": "bquijob_123x456_789y123z456c",
    "projectId": "[Project ID removed]"
  },
  "kind": "bigquery#job",
  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/federated-testing/jobs/bquijob_123x456_789y123z456c",
  "statistics": {
    "creationTime": "1499365894527",
    "endTime": "1499365894702",
    "startTime": "1499365894702"
  },
  "status": {
    "errorResult": {
      "debugInfo": "[Information removed for readability]",
      "message": "Operation cannot be performed on a nested schema. Field: author",
      "reason": "invalid"
    },
    "errors": [
      {
        "message": "Operation cannot be performed on a nested schema. Field: author",
        "reason": "invalid"
      }
    ],
    "state": "DONE"
  },
  "user_email": "user@example.com"
}

The following command gets summary information about job bquijob_123x456_123y123z123c running in myproject in the asia-northeast1 region:

bq --location=asia-northeast1 show -j myproject:bquijob_123x456_123y123z123c

API

Call jobs.get and provide the jobId and projectId parameters. Supply the location parameter and set the value to the location where the job runs.

Python

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

# TODO(developer): Uncomment the lines below and replace with your values.
# from google.cloud import bigquery
# client = bigquery.Client()
# job_id = 'bq-job-123x456-123y123z123c'  # replace with your job ID
# location = 'us'                         # replace with your location

job = client.get_job(job_id, location=location)  # API request

# Print selected job properties
print('Details for job {} running in {}:'.format(job_id, location))
print('\tType: {}\n\tState: {}\n\tCreated: {}'.format(
    job.job_type, job.state, job.created))

Listing jobs in a project

Your project maintains your job history for all jobs created in the past six months. To request automatic deletion of jobs that are more than 50 days old, contact support.

You can view your BigQuery job history via the Google Cloud Platform Console, the CLI, or the API. This history includes jobs that are in the RUNNING state and jobs that are DONE (indicated by reporting the state as SUCCESS or FAILURE).

Required permissions

In order to list jobs, you must have bigquery.jobs.list permissions. The following project-level, predefined IAM roles include bigquery.jobs.list permissions:

When you are granted bigquery.jobs.list permissions, you can list all jobs in a project, but the details and metadata are redacted for jobs submitted by other users. bigquery.jobs.list permissions allows you to see full details for self-created jobs.

To list all jobs, including details for jobs created by other users, you must have bigquery.jobs.listAll permissions. Only the bigquery.admin role has bigquery.jobs.listAll permissions.

The following role is granted bigquery.jobs.list permissions only for self-created jobs. These users can only list jobs they submit:

For more information on IAM roles and permissions in BigQuery, see access control.

Listing jobs

When you list jobs in a project, you do not need to provide a location. Currently, jobs are listed for all locations.

To list jobs in a project:

Web UI

  1. In the navigation pane, click Job History.

  2. In the Recent Jobs section, your jobs are listed by creation time with the most recent jobs at the top. The list includes jobs only for the current user. To see all jobs, use the command-line tool or the API.

CLI

Issue the bq ls command with one of the following flags:

  • -j is used to identify jobs as the resource to list.
  • --all or -a lists jobs from all users. To see full (unredacted) details for all jobs, you must have bigquery.jobs.listAll permissions.
  • -n limits the results.

    bq ls -j -a -n [INTEGER] [PROJECT_ID]

Where:

  • [INTEGER] is an integer that indicates the number of jobs returned.
  • [PROJECT_ID] is the ID of the project that contains the jobs you're listing. If you set a default project, you do not need to provide the [PROJECT_ID] parameter.

Examples:

The following command lists all jobs for the current user. Running this command requires bigquery.jobs.list permissions.

bq ls -j myproject

The following command lists all jobs for all users. Running this command requires bigquery.jobs.listAll permissions.

bq ls -j -a myproject

The following command lists the 10 most recent jobs in myproject:

bq ls -j -a -n 10 myproject

API

Call jobs.list and provide the projectId parameter. To list jobs for all users, set the allUsers parameter to true. Setting allUsers to true requires bigquery.jobs.listAll permissions.

Go

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

it := client.Jobs(ctx)
for i := 0; i < 10; i++ {
	j, err := it.Next()
	if err == iterator.Done {
		break
	}
	if err != nil {
		return err
	}
	state := "Unknown"
	switch j.LastStatus().State {
	case bigquery.Pending:
		state = "Pending"
	case bigquery.Running:
		state = "Running"
	case bigquery.Done:
		state = "Done"
	}
	fmt.Printf("Job %s in state %s\n", j.ID(), state)
}

Java

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

Page<Job> jobs = bigquery.listJobs(JobListOption.pageSize(100));
for (Job job : jobs.iterateAll()) {
  // do something with the job
}

Python

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

# TODO(developer): Uncomment the lines below and replace with your values.
# from google.cloud import bigquery
# project = 'my_project'  # replace with your project ID
# client = bigquery.Client(project=project)
import datetime

# List the 10 most recent jobs in reverse chronological order.
# Omit the max_results parameter to list jobs from the past 6 months.
print("Last 10 jobs:")
for job in client.list_jobs(max_results=10):  # API request(s)
    print(job.job_id)

# The following are examples of additional optional parameters:

# Use min_creation_time and/or max_creation_time to specify a time window.
print("Jobs from the last ten minutes:")
ten_mins_ago = datetime.datetime.utcnow() - datetime.timedelta(minutes=10)
for job in client.list_jobs(min_creation_time=ten_mins_ago):
    print(job.job_id)

# Use all_users to include jobs run by all users in the project.
print("Last 10 jobs run by all users:")
for job in client.list_jobs(max_results=10, all_users=True):
    print("{} run by user: {}".format(job.job_id, job.user_email))

# Use state_filter to filter by job state.
print("Jobs currently running:")
for job in client.list_jobs(state_filter='RUNNING'):
    print(job.job_id)

Cancelling jobs

You can cancel a RUNNING or PENDING job in the web UI, CLI, or API. However, not all job types can be cancelled. If the job cannot be cancelled, an error is returned.

Even if the job can be cancelled, success is not guaranteed. The job might have completed by the time the cancel request is submitted, or the job might be in a stage where it cannot be cancelled.

Required permissions

In order to cancel a job, you must have bigquery.jobs.update permissions. The following project-level, predefined IAM role includes bigquery.jobs.update permissions:

If you grant an account the bigquery.admin role, the user can cancel any eligible job, no matter who submitted it.

The following roles can cancel self-created jobs. These users can only cancel jobs they submit:

For more information on IAM roles and permissions in BigQuery, see access control.

Cancelling a job

To cancel a job:

Web UI

  1. In the navigation pane, click Job History.

  2. In the Recent Jobs section, click the job you're cancelling. The most recent jobs appear at the top of the list.

  3. In the job details, click Cancel Job.

    Cancel job

CLI

Issue the bq cancel command with the job_id parameter. You can request cancellation and return immediately by using the --nosync flag. By default, cancellation requests wait for completion.

When you supply the job ID, you can use the fully-qualified ID or the short form. For example, job IDs listed in the BigQuery web UI are fully-qualified — they include the project and location:

my-project-1234:US.bquijob_123x456_123y123z123c

Job IDs in the command-line tool are listed using the short form — project ID and location are not included:

bquijob_123x456_123y123z123c

To specify the job location, supply the --location flag and set the value to your location. This flag is optional if you use the fully-qualified job ID. If you include the --location flag and you're using the fully-qualified job ID, the --location flag is ignored.

The following command requests job cancellation and waits for completion. If the fully-qualified job ID is supplied, the --location flag is ignored:

bq --location=[LOCATION] cancel [JOB_ID]

The following command requests job cancellation and returns immediately. If the fully-qualified job ID is supplied, the --location flag is ignored:

bq --location=[LOCATION] --nosync cancel [JOB_ID]

Where:

  • [LOCATION] is the name of the location where the job runs. For example, if you are using BigQuery in the Tokyo region, set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  • [JOB_ID] is the ID of the job you're cancelling. If you copy the job ID from the BigQuery web UI, the project ID and location are included in the job ID. For example, my-project-1234:US.bquijob_123x456_123y123z123c.

Examples:

The following command cancels job my-project-1234:US.bquijob_123x456_123y123z123c running in the US multi-region location in my-project-1234 and waits for completion. Because the fully-qualified job ID is used, the location flag is not supplied.

bq cancel my-project-1234:US.bquijob_123x456_123y123z123c

The following command cancels job bquijob_123x456_123y123z123c running in the US multi-region location in my-project-1234 and waits for completion. Because the short form of the job ID is used, the --location flag is supplied.

bq --location=US cancel bquijob_123x456_123y123z123c

The following command cancels job bquijob_123x456_123y123z123c running in the US multi-region location in my-project-1234 and returns immediately. Because the fully-qualified job ID is used, the --location flag is not supplied.

bq --nosync cancel my-project-1234:US.bquijob_123x456_123y123z123c

API

Call jobs.cancel and provide the jobId and projectId parameters. Supply the location parameter and set the value to the location where the job runs.

Python

For more on installing and creating a BigQuery client, refer to BigQuery Client Libraries.

# TODO(developer): Uncomment the lines below and replace with your values.
# from google.cloud import bigquery
# client = bigquery.Client()
# job_id = 'bq-job-123x456-123y123z123c'  # replace with your job ID
# location = 'us'                         # replace with your location

job = client.cancel_job(job_id, location=location)

Repeating a job

It is not possible to rerun a job using the same job ID. Instead you create a new job with the same configuration. When you submit the new job in the web UI or CLI, a new job ID is assigned. When you submit the job using the API or client libraries, you must generate a new job ID.

Required permissions

In order to run a job, you must have bigquery.jobs.create permissions. The following project-level, predefined IAM roles include bigquery.jobs.create permissions:

For more information on IAM roles and permissions in BigQuery, see access control.

Rerunning a job

To repeat a job:

Web UI

To repeat a query job:

  1. In the navigation pane, click Query History.

  2. In the Queries section, to the right of the query, click Open Query.

  3. Click Run Query.

To repeat a load job:

  1. In the navigation pane, click Job History.

  2. In the Recent Jobs section, click the job you want to repeat. The most recent jobs appear at the top of the list.

  3. In the job details, click Repeat Load Job.

CLI

Issue your command again and BigQuery automatically generates a job with a new job ID.

API

There is no single-call method to repeat a job; if you want to repeat a specific job:

  1. Call jobs.get to retrieve the resource for the job to repeat.

  2. Remove the id, status, and statistics field. Change the jobId field to a new value generated by your client code. Change any other fields as necessary.

  3. Call jobs.insert with the modified resource and the new job ID to start the new job.

Cette page vous a-t-elle été utile ? Évaluez-la :

Envoyer des commentaires concernant…