Web detection tutorial

Audience

The goal of this tutorial is help you develop applications using the Google Cloud Vision API Web detection feature. It assumes you are familiar with basic programming constructs and techniques, but even if you are a beginning programmer, you should be able to follow along and run this tutorial without difficulty, then use the Cloud Vision API reference documentation to create basic applications.

This tutorial steps through a Vision API application, showing you how to make a call to the Vision API to use its Web detection feature.

Prerequisites

Overview

This tutorial walks you through a basic Vision API application that uses a Web detection request. A Web detection response annotates the image sent in the request with:

  • labels obtained from the Web
  • site URLs that have matching images
  • URLs to Web images that partially or fully match the image in the request

Code listing

As you read the code, we recommend that you follow along by referring to the Cloud Vision API Python reference.

import argparse
import io

from google.cloud import vision


def annotate(path):
    """Returns web annotations given the path to an image."""
    image = None
    vision_client = vision.Client()

    if path.startswith('http') or path.startswith('gs:'):
        image = vision_client.image(source_uri=path)

    else:
        with io.open(path, 'rb') as image_file:
            content = image_file.read()

        image = vision_client.image(content=content)

    return image.detect_web()


def report(annotations):
    """Prints detected features in the provided web annotations."""
    if annotations.pages_with_matching_images:
        print('\n{} Pages with matching images retrieved')

        for page in annotations.pages_with_matching_images:
            print('Score : {}'.format(page.score))
            print('Url   : {}'.format(page.url))

    if annotations.full_matching_images:
        print ('\n{} Full Matches found: '.format(
               len(annotations.full_matching_images)))

        for image in annotations.full_matching_images:
            print('Score:  {}'.format(image.score))
            print('Url  : {}'.format(image.url))

    if annotations.partial_matching_images:
        print ('\n{} Partial Matches found: '.format(
               len(annotations.partial_matching_images)))

        for image in annotations.partial_matching_images:
            print('Score: {}'.format(image.score))
            print('Url  : {}'.format(image.url))

    if annotations.web_entities:
        print ('\n{} Web entities found: '.format(
            len(annotations.web_entities)))

        for entity in annotations.web_entities:
            print('Score      : {}'.format(entity.score))
            print('Description: {}'.format(entity.description))


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    path_help = str('The image to detect, can be web URI, '
                    'Google Cloud Storage, or path to local file.')
    parser.add_argument('image_url', help=path_help)
    args = parser.parse_args()

    report(annotate(args.image_url))

This simple application performs the following tasks:

  • Imports the libraries necessary to run the application
  • Takes an image path as an argument and passes it to the main() function
  • Uses the Google Cloud API Client to perform Web detection
  • Loops over the response and prints out the results
  • Prints list of Web entities with description and score
  • Prints a list of matching pages
  • Prints a list of partially-matching images
  • Prints a list of fully-matching images

A closer look

Importing libraries

import argparse
import io

from google.cloud import vision

We import standard libraries:

  • argparse to allow the application to accept input filenames as arguments
  • io for reading from files

Other imports:

  • The vision module within the google.cloud library for accessing the Vision API

Running the application

parser = argparse.ArgumentParser(
    description=__doc__,
    formatter_class=argparse.RawDescriptionHelpFormatter)
path_help = str('The image to detect, can be web URI, '
                'Google Cloud Storage, or path to local file.')
parser.add_argument('image_url', help=path_help)
args = parser.parse_args()

report(annotate(args.image_url))

Here, we simply parse the passed-in argument that specifies the URL of the Web image, and pass it to the main() function.

Authenticating to the API

Before communicating with the Vision API service, you must authenticate your service using previously acquired credentials. Within an application, the simplest way to obtain credentials is to use Application Default Credentials (ADC). The client library obtains the credentials automatically. By default, this is done by obtaining credentials from the GOOGLE_APPLICATION_CREDENTIALS environment variable, which should be set to point to your service account's JSON key file (see Set Up a Service Account for more information.)

Constructing the request

image = None
vision_client = vision.Client()

if path.startswith('http') or path.startswith('gs:'):
    image = vision_client.image(source_uri=path)

else:
    with io.open(path, 'rb') as image_file:
        content = image_file.read()

    image = vision_client.image(content=content)

return image.detect_web()

Now that our Vision API service is ready, we can construct a request to the service. Requests to the Google Cloud Vision API are provided as JSON objects. See the Vision API Reference for complete information on the structure of a request.

This code snippet performs the following tasks:

  1. Constructs an image object for making requests using the client library
  2. Uses the image object to retrieve the Web annotations
  3. Returns the annotations

Printing the response

if annotations.pages_with_matching_images:
    print('\n{} Pages with matching images retrieved')

    for page in annotations.pages_with_matching_images:
        print('Score : {}'.format(page.score))
        print('Url   : {}'.format(page.url))

if annotations.full_matching_images:
    print ('\n{} Full Matches found: '.format(
           len(annotations.full_matching_images)))

    for image in annotations.full_matching_images:
        print('Score:  {}'.format(image.score))
        print('Url  : {}'.format(image.url))

if annotations.partial_matching_images:
    print ('\n{} Partial Matches found: '.format(
           len(annotations.partial_matching_images)))

    for image in annotations.partial_matching_images:
        print('Score: {}'.format(image.score))
        print('Url  : {}'.format(image.url))

if annotations.web_entities:
    print ('\n{} Web entities found: '.format(
        len(annotations.web_entities)))

    for entity in annotations.web_entities:
        print('Score      : {}'.format(entity.score))
        print('Description: {}'.format(entity.description))

Once the operation has been completed, our response will contain an AnnotateImageResponse, which consists of a list of Image Annotation results, one for each image sent in the request. Because we sent only one image in the request, we walk through the WebDetection, and print the entities and URLs contained in the annotation (the top two results from each annotation type are shown in the next section).

Running the application

To run the application, we pass in the Web URL (http://wallppr.net/wp-content/uploads/2016/10/Car-4K-Wallpaper-10.jpeg) of the following car image.

Here is the Python command with the passed-in Web URL of the car image, followed by console output. Note that a relevancy score is added (for example, :1.1068367) after the listed entities and URLs. Note that scores are not normalized or comparable across different image queries.

python web_detect.py "http://wallppr.net/wp-content/uploads/2016/10/Car-4K-Wallpaper-10.jpeg"
Car:1.47328
Ferrari F430:1.1068367
Ferrari S.p.A.:0.75906
Auto racing:0.73261
Sports car:0.5722
pagesWithMatchingImages
https://www.pexels.com/search/race%20car/:7.068652
https://www.pexels.com/photo/road-blue-car-vehicle-50704/:3.7592764
...
partialMatchingImages
https://i1.wp.com/representeveryone.com/wp-content/uploads/2016/05/car_new.jpg?fit=5016%2C3344&ssl=1:1
http://wallppr.net/wp-content/uploads/2016/10/Car-4K-Wallpaper-10.jpeg:1
...
fullMatchingImages
https://i1.wp.com/representeveryone.com/wp-content/uploads/2016/05/car_new.jpg?fit=5016%2C3344&ssl=1:1
http://wallppr.net/wp-content/uploads/2016/10/Car-4K-Wallpaper-10.jpeg:1
...

Congratulations! You've performed Web detection using the Google Cloud Vision API!

Send feedback about...

Google Cloud Vision API Documentation