Putting a Groovy twist on Cloud Vision

Powerful machine learning APIs are at your fingertips if you’re developing with Google Cloud Platform, as client libraries are available for various programming languages. Today, we’re investigating the Cloud Vision API and its Java SDK, using the Apache Groovy programming language—a multi-faceted language for the Java platform that aims to improve developer productivity thanks to a concise, familiar and easy to learn syntax.

At GR8Conf Europe, in Denmark, the conference dedicated to the Apache Groovy ecosystem, I spoke about the machine learning APIs provided by Google Cloud Platform: Vision, Natural Language, Translate, and Speech (both recognition and synthesis). Since it’s a groovy conference, we presented samples and demos using a pretty Groovy language. I wanted to share the underlying examples with a wider audience, so here’s the first of a series of blog posts covering the demos I presented. I’ll start with the Google Cloud Vision API, and I will cover the other APIs in future posts.

Google Cloud Vision API lets you:

• Get labels of objects and places from your pictures
• Detect faces, with precise location of facial features
• Determine if the picture is a particular landmark
• Check for inappropriate content
• Obtain some image attributes information
• Find out if the picture is already available elsewhere on the internet
• Detect brand logos
• Extract text that appears in your images (OCR)

In this first installment, I focus on two aspects: labels and OCR. But before diving in, I want to illustrate some of the use cases that all of the Vision API features enable, when you want to enhance your apps by integrating the Vision API.

Label detection

Example use cases:

• If you run a photography website, you might want to let your users search for particular pictures on specific topics. Rather than having someone manually label each and every picture uploaded, you can store labels generated by the API as keywords for your search engine.
• If you’re building the next friend-finder app for animals, perhaps you want to check that the picture indeed contains a dog. Again, labels will help to tell you if that’s the case without manually checking each submitted image.
• If you need to find pictures that match a certain theme, you can use the labels to help with automatic categorization.

Face detection

Example use cases:

• You are building a light-hearted app in which users add a mustache or silly hat to uploaded pictures, you now know how to initially position and size the added objects.
• If you want to know (or estimate) the number of attendees in a meeting or presentation at a conference, you can use the API to provide a count of the number of faces in the picture.

Landmark detection

Example use cases:

• If you want users of your app or site to upload only pictures of a particular location, you could use those coordinates (provided in latitude and longitude) to automatically check that the place photographed is within the right geo-fenced bounding box.
• If you want to automatically show pictures on a map in your app, you could take advantage of the GPS coordinates, or automatically enrich tourism websites with pictures from the visitors of specific locales.

Inappropriate content detection

Example use case:

• If you want to avoid the unwelcome surprise of inappropriate user-generated content showing up in your site or app, you can use this capability to filter images automatically.

Image attributes and web annotations

Example use cases:

• If you want to make your app or website responsive, before loading the full images you may want to show colored placeholders for them. You can get that information with the palette information from the API.
• If you’d like to automatically crop pictures while keeping their essential aspects, you can use the crop hints.
• If you allow photographers to upload their pictures on your website and you want to check that no one is stealing pictures and posting them without proper attribution, you can use the API to check if each picture can be found elsewhere on the web.
• For the picture of me in the introduction of this post, Vision API recognized entities like “Guillaume Laforge” (me!), Groovy (the programming language I’ve been working on since 2003), JavaOne (a conference I’ve often spoken at), “Groovy in Action” (the book I’ve co-authored), “Java Champions” (I’ve recently been nominated!), and “Software Developer” (yes, I do code!). Thanks to those entities, you are able to automatically recognize famous people—more than just me!

Brand or logo detection

Example use case:

• If you want your company’s brand or products to be displayed on supermarket shelves, you might have people take pictures of those shelves and confirm automatically that your logo is being displayed.

OCR or text recognition

Example use case:

• When you want to automatically scan expense receipts, enter text rapidly from pictures, or tackle any of the usual use cases for OCR, you can use the API to find and extract any text identified within.

Time to get Groovy!

Using the Apache Groovy programming language, I will illustrate two approaches: the first one uses a REST client library like groovy-wslite, and the second one uses the Java client library provided for the API.

Prerequisites

In order to use Vision API, you’ll need to have an account on Google Cloud Platform (GCP). You can benefit from the $300 free trial credits, as well as free quotas. For instance, for Vision API, without even consuming your credits, you can make 1000 calls for free every month. You can also take a look at the API’s pricing details, once you exceed the quota or your credits.

Briefly, if you’re not already using GCP or still don’t have an account, please register and create one.

Once you’ve set up your GCP account, create a cloud project and enable access to Vision API for the project. That’s it—now you’re ready to follow the steps detailed below.

Note that after you have enabled the API, you still need to authenticate your client to use the service. There are different possibilities here. In the OCR example, I call the REST API and use an API key passed as a query parameter. In the label detection sample, I use a service account with application default credentials. You can learn more about those approaches in the authentication documentation.

Okay, let’s get started!

OCR using the Vision REST endpoint

In Apache Groovy, when calling REST APIs, I often use the groovy-wslite library. But there are other similar great libraries like HTTPBuilder-NG, which offer similar capabilities via a nice syntax too.

To start, let’s grab the REST client library and instantiate it:

Here’s the URL of the picture with the text I’m interested in:

Be sure to change this dummy text for the API key with a proper API key that you can generate from the “APIs & services > Credentials” section in the cloud console, as explained here.

Next, I’m going to send a post request to the /images:annotate path with the API key as a query parameter. My request is in JSON, and I’m using Groovy’s nice list and maps syntax to represent that JSON request, providing the image URL and the feature I want to use (i.e. text detection):

This corresponds to the following JSON payload:

Thanks to Apache Groovy’s flexible nature, it’s then easy to go through the returned JSON payload to get the list and `println` all the text annotations and their descriptions (which correspond to the recognized text):

In spite of the low quality of the image and some odd font kerning, the API was able to find allergens like “châtaignier” (with the correct circumflex accent) and “urticacées” (acute accent). On the allergen “cupressacées,” the diacritical mark was missed though, and a space is intertwined, but the font seems to be adding extra space between some letters.

Label detection with the Java client library

Let’s grab the Java client library for the vision API:

Here’s the URL of the picture:

Now, let’s instantiate the ImageAnnotatorClient class. It’s a Closeable object, so we can use Groovy’s withCloseable{} method:

ImageAnnotatorClient.create().withCloseable { vision ->

We need the bytes of the picture, which we obtain via the .bytes shortcut, and we create a ByteString object from the protobuf library used by the Vision API:

def imgBytes = ByteString.copyFrom(imgUrl.bytes)

To create the request with the AnnotateImageRequest builder, we can employ the Groovy tap{} method to simplify usage of the builder design pattern:

In that request, we ask for the label detection feature and pass the image bytes. Next, we call the API with our request and then iterate over the resulting label annotations and their confidence score:

The labels found (and their confidence) are as follows:

That looks pretty accurate to me!

Conclusion