Detect Labels

The Vision API can detect and extract information about entities within an image, across a broad group of categories.

Labels can identify objects, locations, activities, animal species, products, and more.

Labels are returned in English only. The Cloud Translation API can translate English labels into any of a number of other languages.

For example, the image above may return the following list of labels:

Description	Score
ferris wheel	0.84832066
amusement park	0.8101249
night	0.8036025
outdoor recreation	0.68825835
fair	0.6566326

Note: Using this API in a mobile app? Try ML Kit for Firebase, which provides native Android and iOS SDKs for using Cloud Vision services, as well as on-device ML Vision APIs and on-device inference using custom ML models.

Code samples

For samples in a number of programming languages, see:

• Label detection code samples

Label detection requests

Set up your GCP project and authentication

If you have not created a Google Cloud Platform (GCP) project and service account credentials, do so now. Expand this section for instructions.

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. Set up a GCP Console project.

   Set up a project

   Click to:

   • Create or select a project.
   • Enable the Cloud Vision API for that project.
   • Create a service account.
   • Download a private key as JSON.

   You can view and manage these resources at any time in the GCP Console.

3. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the file path of the JSON file that contains your service account key. This variable only applies to your current shell session, so if you open a new session, set the variable again.

   Example: Linux or macOS

   Replace [PATH] with the file path of the JSON file that contains your service account key.

   export GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

   For example:

   export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"

   Example: Windows

   Replace [PATH] with the file path of the JSON file that contains your service account key, and [FILE_NAME] with the filename.

   With PowerShell:

   $env:GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

   For example:

   $env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\[FILE_NAME].json"

   With command prompt:

   set GOOGLE_APPLICATION_CREDENTIALS=[PATH]

4. Install and initialize the Cloud SDK.

Detect Labels

Powershell

To make a label detection request using Windows Powershell, make a POST request to the https://vision.googleapis.com/v1/images:annotate endpoint and specify LABEL_DETECTION as the value of features.type, as shown in the following example:

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Post `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body "{
      'requests': [
        {
          'image': {
            'source': {
              'imageUri': 'https://cloud.google.com/vision/docs/images/ferris-wheel.jpg'
            }
          },
          'features': [
            {
              'type': 'LABEL_DETECTION'
            }
          ]
        }
      ]
    }" `
  -Uri "https://vision.googleapis.com/v1/images:annotate" | Select-Object -Expand Content

Images can be passed in one of three ways: as a base64-encoded string; as a Google Cloud Storage URI; or as a publicly-accessible HTTPS or HTTP URL. See Making requests for more information.

See the AnnotateImageRequest reference documentation for more information on configuring the request body.

curl command

To make a label detection request using curl from the Linux or MacOS command line, make a POST request to the https://vision.googleapis.com/v1/images:annotate endpoint and specify LABEL_DETECTION as the value of features.type, as shown in the following example:

curl -X POST \
     -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
     -H "Content-Type: application/json; charset=utf-8" \
     --data "{
      'requests': [
        {
          'image': {
            'source': {
              'imageUri': 'https://cloud.google.com/vision/docs/images/ferris-wheel.jpg'
            }
          },
          'features': [
            {
              'type': 'LABEL_DETECTION'
            }
          ]
        }
      ]
    }" "https://vision.googleapis.com/v1/images:annotate"

Images can be passed in one of three ways: as a base64-encoded string; as a Google Cloud Storage URI; or as a publicly-accessible HTTPS or HTTP URL. See Making requests for more information.

See the AnnotateImageRequest reference documentation for more information on configuring the request body.

GCLOUD COMMAND

To detect labels in an image, use the gcloud ml vision detect-labels command as shown in the following example:

gcloud ml vision detect-labels https://cloud.google.com/vision/docs/images/ferris-wheel.jpg

Label detection responses

If the request is successful, the server returns a 200 OK HTTP status code and the response in JSON format.

A LABEL_DETECTION response includes the detected labels, their score, and an opaque label ID:

{
  "responses": [
    {
      "labelAnnotations": [
        {
          "mid": "/m/017rgb",
          "description": "ferris wheel",
          "score": 0.84832066
        },
        {
          "mid": "/m/010jjr",
          "description": "amusement park",
          "score": 0.8101249
        },
        {
          "mid": "/m/01d74z",
          "description": "night",
          "score": 0.8036025
        },
        {
          "mid": "/m/05b0n7k",
          "description": "outdoor recreation",
          "score": 0.68825835
        },
        {
          "mid": "/m/02jf28",
          "description": "fair",
          "score": 0.6566326
        }
      ]
    }
  ]
}

Where:

• mid, if present, contains a machine-generated identifier (MID) corresponding to the entity's Google Knowledge Graph entry. Note that mid values remain unique across different languages, so you can use these values to tie entities together from different languages. To inspect MID values, refer to the Google Knowledge Graph API documentation.

• description is the label description.

• score is the confidence score, which ranges from 0 (no confidence) to 1 (very high confidence).