Chefkoch whips up handwritten recipes in the cloud with text detector

Editor’s note: When German cooking platform Chefkoch was looking to bring treasured hand-me-down recipes into the 21st century it found a scaleable, well-supported solution with Google’s data cloud. Here’s how it was cooked up. 

Whether it’s salad dressing or chicken soup, most households have a favorite dish passed down across the generations. These recipes are often scribbled on scraps of paper and this personal culinary heritage is heavily guarded. Recognizing the significance of handwritten or printed recipes, German cooking platform Chefkoch wanted to make it possible to quickly and easily parse, extract and digitalize these time-honored tasty morsels using Google Cloud augmented analytics and machine learning (ML) capabilities, to make it easy for users to share and access these recipes in a digital form.

When considering the best way to develop the technology, Chefkoch undertook extensive market research of the global food market. It identified best practices within the industry, looked at working business models and upcoming food-tech trends, and applied this to an in-depth study of its own users’ motivations for using its platform. Finally, Chefkoch created a Kano Model that prioritized different features based on how likely they would be to satisfy its users.

Chefkoch users each have their own Kochbuch (cookbook) on the platform where they can save, sort, and manage their Chefkoch recipes. On the back of its research, Chefkoch decided that this was the best place for its new proposition. It opted to develop the Kochbuch to make it possible to store any recipe within it, including offline ones. “To do this we needed to extract the text, be it handwritten or printed, and then separate the recipe title, the ingredients and the instructions,” explains Tim Adler, CTO of Chefkoch.

APIs get reading recipes

To enable this, Chefkoch began assessing various text importing tools. In May 2021, it settled on Google Cloud’s ML services and  Google Cloud Functions, because it offers scalable functions as a service (FaaS) with a number of APIs that enable code to be run without server management. 

“We screened the market for solutions in order to recognize text in scanned handwritten recipes,” says Adler. “Google's solution convinced us, not only because we could work with the APIs and documentation easily, but also because Google's team presented us with an impressive proof of concept with our own test-data."

Chefkoch chose to build this recipe-reading tool using the Vision and Natural Language services offered by Google Cloud  because it can run across devices and be scaled cost-effectively.  As seen in the diagram above, it uses the Cloud Vision API optical character recognition (OCR) tool, which is optimized for German and English text detection, to extract the text from a written or printed page. It then applies AutoML Natural Language Entity Extraction  Models 1 and 2  and the Cloud Natural Language API to identify and segregate the different sections of the recipes to perfect the resulting on screen recipe, as shown below.

Chefkoch worked closely with Google to perfect the solution. Our Google team initially made a demo for Chefkoch to help the team understand how everything works together, demonstrating, for example, how a dataset has to be structured to optimize the desired results after the model training. They presented a working end-to-end demo of a functioning API which takes in the image of a handwritten or printed recipe, and outputs the desired results: with the various components of the recipes cleanly extracted and separated. 

This offline-to-online recipe upload service is now being trialed on Chefkoch’s Kochbuch. “We are working on testing, improving, extending and producing the solution,” reveals Adler.

Tweaking the recipe 

Results from early-stage testing are encouraging, with users rating the OCR feature with an A or B grade. In response to this feedback, small adjustments have already been made to the training of the model to get it to align with audience needs. The tool, which has been unofficially named the Handwritten Recipe Parser, can now pick up contextual spelling mistakes, for example, where a word is spelled incorrectly for the context, such as “meet” instead of meat.

Cooking up users with analog to digital offering

There are plans to expand the menu of features on Handwritten Recipe Parser too. Chefkoch is now developing a manual recipe extraction solution, where users can upload their own recipe images and add the title, ingredients and method and there are plans to enable users to amend existing Chefkoch recipes by adding their own text and written annotations. 

To learn more about Cloud AutoML and Vision API, visit our site.