Adding computer vision to your Android app
Contributed by Google employees.
Sara Robinson has authored an excellent post explaining one approach to Adding computer vision to your app.
In the post, she introduces the following pattern:
- Use Firebase Auth native client to upload to Firebase Storage.
- Use Cloud Functions to access Vision API and store the result in Firestore.
- Access Cloud Firestore from device to retrieve the API Result.
Sara uses iPhone X in her application, but you can get similar results with Android. This tutorial, based on the Firebase storage quickstart for Android, provides a quick proof of concept.
Start by checking out a short demo of how the proof of concept works.
First, you upload an image to Firebase storage from your Android device:
After the image is uploaded, a link to the uploaded file is presented in the app, exactly the same as in the sample app. Then the app retrieves the detected labels for the image and presents them.
Now take a closer look at how each of the steps is performed.
If you want to follow along, start by getting the Firebase quickstart samples.
git clone https://github.com/firebase/quickstart-android
The storage sample is in the quickstart-android/storage folder, and you can find
the instructions for configuring it on the Firebase Cloud Storage site.
You also need to initialize the project sources folder by installing the
Firebase SDK and calling firebase init with Storage, Functions, and
Firestore.
Step 1: Upload a file to Firebase Storage
This step uses the functionality of the sample app. In the sample
app, a service named MyUploadService is implemented to upload a file outside of the
main application thread.
The uploadFromUri function in the sample app
illustrates the bulk of the operations performed by the provided service.
After the file is uploaded, the fileUri is passed in an Intent so that
the MainActivity class can retrieve the file data. This is done in the sample
app in the MyUploadService activity's broadcastUploadFinished method.
You can verify that the storage operation works by visiting the Firebase console for storage and seeing the uploaded files:
Step 2: Analyze the image and publish label data to Firestore
Now that the files are successfully uploading to Firebase Storage, it's time to process the file using a Cloud Functions call. This operation is virtually identical to the Cloud Functions API call made by Sara in her post, but retrieves only the label data, instead of the label and web annotation data.
The following code, specified in index.js in the functions folder, loads the
required libraries for Firebase and Google Cloud Vision, and then transfers the
API call result to Firestore.
You can see the results in the Firestore section of the console:
Of note, you can now see the indexed label data on the right-most column of the console.
Step 3: Manually retrieve the label data from Firestore
This tutorial reuses and relabels the Download button from the Firebase Storage sample
app to manually trigger the retrieval of the Label data from Firestore. To do
this, the button name is changed to button_detections in the app resources
and the UI strings are replaced as appropriate. A new method named
retrieveMetadata is added to the click handler for the button.
The following code shows how to retrieve the metadata for the last uploaded image by using the Firestore API:
It might be best to do this in a separate service, but for the
purposes of this proof of concept, this should be sufficient. Also, this replaces
the proto-style object characters with JSON-style object characters because of
manually filtering the result data in UpdateUI.
When UpdateUI is called, the sample app checks the stored member variable
mResponse and then filters the label description strings from the result
data.
With the help of Sara's blog post, it was incredibly easy to update the Firebase Storage sample app to work with the Vision API and return results to an Android device. This approach to accessing the Google Cloud Machine Learning features is not restricted to the Vision API. For example, if you wanted to use the translation API with the NMT model, you could employ a similar approach but by storing text data instead of photo data.
Next steps
If you want to prepare this app for production, think about the following items:
- Fix all the UI - Create label bubbles instead of just formatted text.
- Enable user auth to prevent abuse of your API quota.
- Move the operations done with Firestore to a separate service.
- Eliminate data polling for Firestore data.
See the following sites for more information:
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.