The life-changing magic of making with ML
Dale Markowitz
Applied AI Engineer
One of the most fun ways to learn machine learning is by building projects for yourself. In this post, I’ll show you how.
An eon ago, I decided to learn to code by building my own personal website. I knew nothing about computers then, let alone what a server was, but it felt like I had infinite tech topics left to learn and an endless appetite to learn them. I laid awake at night thinking about all the ways I’d trick out my website—scrolling parallax stars, an obscene number of Google Fonts—and dreamt of all the personal projects I’d build next.
Years later, I’m a professional engineer who works on objectively cooler, more sophisticated technological problems than my hack-y personal website. Yet it’s hard to say I ever feel that, ahem, “childlike wonder” I did back when I was first learning to code.
If you work in tech, you surely know you’ve committed yourself to a lifetime of learning. Blink too long and every piece of software you thought was state-of-the-art will be entirely replaced (unless that software is Vim, which I still haven’t managed to exit).
In computer science, one great example is machine learning (ML). Most of us never learned this topic in university (if we studied CS in school at all), but it’ll soon be ubiquitous, transforming software development in every domain. It’s no surprise that the question I’m most frequently asked is, “Where should I start learning machine learning?”
Typically I’ll recommend resources like Google’s Machine Learning Crash Course, the book Hands-On Machine Learning, or Andrew Ng’s classic Coursera course, which cover the fundamentals from theory to practice.
But if you’re like me and your favorite way to learn is by building, consider learning ML by building software for yourself. Not only are personal projects a fun (and potentially useful) way to learn a new technology, they’re also a great way to learn about some of the challenges you’ll face deploying machine learning in production, rather than in the quixotic setting of a homework assignment.
At the beginning of the pandemic, when I suddenly found myself swimming in free time, this was the challenge I gave myself—can I learn more about machine learning by building personal projects that solve problems in my own life? Since then, I’ve used ML to intelligently search my family videos, improve my tennis serve, translate videos, invent new baking recipes, and a whole lot more.
Below you’ll find a roundup of these projects along with their source code, tutorial YouTube videos, and step-by-step blog posts. From introducing you to new techniques and tools to showing you how to weave this stuff together into functioning apps, I hope you’ll find these projects as fun to build as they are to learn from. Or, better yet, I hope they’ll inspire your own endeavors, made with ML. If you do build something neat, let me know. Until then, as always, happy making.
Making with machine learning projects
Smart family video archive
What you’ll build: An archive that makes your home videos searchable by transcript (what people say) and by image (search for “birthday,” “bicycle,” “video game”).
What you’ll learn:
Discord moderator bot
What you’ll build: A bot for the chat platform Discord that allows you to flag toxic, offensive, profane, or spam messages
What you’ll learn:
How to use the Perspective API to analyze text
How to use ML in a chat application
How to think about using machine learning for tricky problems that aren’t so black-and-white
Can AI make you a better athlete?
What you’ll build: A Jupyter notebook that can track your tennis serve and the trajectory of a tennis ball (or golf swing or baseball pitch, etc) and that analyzes this data to give you tips to improve. Follow along in the Qwiklab.
What you’ll learn:
Hacks for doing sophisticated ML with small datasets
How to combine pose tracking with simple math to understand human movement
How to use the Video Intelligence API
How to use AutoML Vision
Making smarter video game worlds with NLP
aka Build Apps Powered by Language with Semantic ML
What you’ll build:
A simple language-based system for having video game worlds respond to users’ free-text input
What you’ll learn:
How to use sentence embeddings, one of the most useful techniques in all of NLP
How to implement semantic text search
How to cluster text
How to implement a basic conversational bot
How to do all this stuff from a Google Sheet
PDF-to-audiobook converter
What you’ll build: Code that converts PDFs to MP3 audiobooks
What you’ll learn:
How to extract text from PDFs using the Vision API
How to speak words using the Text-to-Speech API
How to use math hacks to parse document layouts
Translating and dubbing videos with ML
What you’ll build: Code that can automatically transcribe, translate, and dub videos.
What you’ll learn:
How to combine Speech-to-Text, Translation, and Text-to-Speech
How to improve transcription and translation quality
How to work with videos and audio in Python
Can AI generate a good baking recipe?
What you’ll build: A no-code machine learning model that classifies recipes and that can even generate new ones
What you’ll learn:
How to build a no-code machine learning model for tabular (“spreadsheet”) data using AutoML Tables
How to ML explainability features to understand why a model is making the decisions it is
ML without code in the browser
What you’ll build: A super quick machine learning model based on pose, vision, or sound
What you’ll learn:
What it takes to build a simple neural network (without coding)
How to build a super fast model that runs in the browser with Teachable Machines
Making outfits with AI
What you’ll build: An app that uses pictures of your own wardrobe and photos from fashionable social media influencers to recommend you outfits/
What you’ll learn:
How to use the Product Search and Vision API
How to architect ML apps with React and Firebase
