Announcing Accuracy Evaluation for Cloud Speech-to-Text

We are thrilled to introduce Accuracy Evaluation, the newest feature in our Cloud Speech UI, to allow for easy and seamless benchmarking of our Speech-to-Text (STT) API models and configurations. The STT API covers a wide variety of use cases, from dictation and short commands, to captioning and subtitles. Getting the most of STT, however, can be a complicated process. To achieve the highest accuracy on any AI use case requires careful testing and tuning to find just the right configuration. 

We have been diligently listening to  customer feedback, and looking for a quick and effective way to benchmark our current and future STT API offerings. Previously, our customers and enterprise users had to do this work manually. This included invoking the API to generate the transcripts and save the result, then using a command-line tool, relying on a third-party library, or writing code to compare the STT system results with a ground-truth file. For every model and configuration, this process had to be redone, which was cumbersome,time-consuming and error prone.

A 3-step process to measure accuracy

Today’s announcement significantly simplifies the process. Now, the user-friendly interface in the Accuracy Evaluation feature in our Cloud Speech UI makes it easy for anyone on your team to evaluate the accuracy of our STT API against your own datasets. To begin, customers upload audio files, specify the desired STT API configurations and ground-truth, and the benchmarking is done automatically for you. To ensure maximum privacy and security, audio files uploaded are only processed inside your own Google Cloud Tenant Project.  

To measure and compare the accuracy of our STT API, we use the industry standard of Word Error Rate (WER), a simple, easy-to-understand metric that can be compared across different models and datasets. It is defined as the ratio of the total number of errors (Insertions, Deletions, and Substitutions) to the total number of words in the reference transcript, and it ranges from 0%, when the output of the STT system matches exactly the ground-truth, to 100%, when there is no match at all. Our tool calculates WER for the STT output and the ground-truth, while also providing a detailed breakdown on the Insertion, Substitution and Deletion errors, giving scientists and application developers exactly the information they need to be successful in their workflow.

To access Accuracy Evaluation, log in to our Speech-to-text User Interface and navigate to the "Transcriptions'' tab. After you have successfully transcribed your audio file, use the Transcription Accuracy section. Click the Upload Ground Truth button at the top of the section to begin calculating accuracy.

Learn more about Accuracy

Detailed instructions on how to use the new feature can be found here, and if you are curious to learn more about how accuracy is measured in production-facing Speech Transcription systems, you can find our documentation here. 

We are excited to see the insights and improvements you can achieve with Accuracy Evaluation on Cloud Speech UI and we look forward to supporting you with the best in-class Speech-to-Text systems.