Using machine learning to improve road maintenance

There’s a new way to look out for potholes in the road and it doesn’t involve better eyeglasses or dispatching costly repair crews. Bus-mounted cameras and machine learning can do it for you, as the City of Memphis discovered.    

Staying on top of deteriorating roads when you can't add more personnel is a never ending cycle of patching holes as increasing traffic only worsens the problem.     

Google Cloud Partner, SpringML worked with the City of Memphis to tackle this problem, assisting in repairing 63,000 potholes in one year, a massive improvement in pothole detection over previous manual efforts.  

Advances in analytics and machine learning are making it possible for authorities to not only fix roads faster but actually prevent damage from occurring in the first place.

Using machine learning for road maintenance 

The City of Memphis struggled with a problem many cities have to face: the continuous degradation of paved roads and the formation, through usage and weather, of potholes. These gaps in the road not only frustrate drivers, they slow down traffic, delaying commutes and mass transit, and they lead to greater wear and tear on vehicles. They're just no good. 

Potholes are inevitable, so the challenge for Memphis, and other cities, becomes how to keep up, putting repair resources where they can be most helpful. With limited hardware and staff, they can't tackle every report from citizens. And those public reports don't always present a full picture of the problem either.

Enter SpringML, who partners with public sector customers to solve problems with technology in creative ways. As the SpringML team joined with Memphis to figure this out, they first looked at what sorts of data they could get access to. And voila: bus cameras!

“Look for data you already have that can fuel your decision making, before you go out and try to acquire new data sets.” Eric Clark, AI Practice, SpringML

The city buses in Memphis all have front-mounted cameras, gathering data the entire time that the bus is running, mostly for traffic purposes. Every bus in the city was watching the roads every day! 

Immediately the team had a treasure trove of data: every road covered by the mass transit system has daily recordings being captured. The bus routes are well defined and each bus has GPS to help correlate the footage with precise locations. The team set to work.

At the end of the day they retrieved videos from each bus and uploaded them to on-prem storage —a fairly manual process.

Then a script checked for new files in the video directory nightly, and uploaded the new videos to Google Cloud Storage, to begin processing.

From there the Google Cloud Video Intelligence API could start to work, running its detection model on the new videos to look for possible pothole images. To make the initial pothole detection AI model the SpringML team took existing images and manually picked out potholes. They also used data from higher quality cameras to improve detection and accuracy of the model, and continued to feed new data from the bus routes to improve the model over time. 

Results from the Video Intelligence model inference were sent to BigQuery, where the images, annotations, file metadata, location and scoring were kept and easily sorted or queried.

The custom web-app presented possible potholes to public works employees, who could help correct the model when it made mistakes (frequently caused by stains, shadows or animals), or confirm a pothole and then trigger the next automated flow. Once a pothole is detected and confirmed, the team needs a work ticket to track the actual repair. So the web-app submits information about confirmed potholes to the city's 311 information system, which can then generate a ticket, which will dispatch a work crew and repair vehicle to actually repair the road.

A smooth road ahead

As well as detecting and fixing potholes faster, this effort has paved the way for future projects that can improve public infrastructure, as more of the data gets gathered and applied to decision making. 

Want to learn more? Read this Video Intelligence API quickstart to try it out. Listen to the interview with SpringML’s Eric Clark on the GCP Podcast, and check out more machine learning tools in our AI Platform.