Descartes Labs: Advancing global food security
About Descartes Labs
Descartes Labs created a cloud-based supercomputing platform to apply machine intelligence to massive datasets. Its mission is to better understand the planet, for good and for profit. Today, Descartes Labs uses satellite imagery and advanced data and analytics processing in Google Cloud to model complex systems on the planet, like forestry and agriculture, to enable organizations to "see" and understand the world in a whole new way.
Tell us your challenge. We're here to help.
Contact us
About Intel Corporation
Intel Corporation, a Google Cloud Technology Alliance partner, is known for being a market leader and one of the world's best-known chip producers, but Intel does much more. Harnessing the capability of the cloud and the ubiquity of the Internet of Things, Intel is disrupting industries and solving global challenges. Leading on policy, diversity, inclusion, education, and sustainability, Intel creates value for its stockholders, customers, partners, and society.
Descartes Labs can cost effectively process years of satellite imagery to help businesses and governments make accurate predictions about global food supplies and detect early warnings of famine, using Google Cloud with the latest Intel Xeon Scalable processors.
Google Cloud results
- Accelerates time to market for research results by more than six months with Google Cloud
- Able to process constantly increasing data volumes while avoiding large capital expenditures
- Improves speed and accuracy of crop yield data
Handling PB-scale datasets in hours versus months
Feeding a growing world population in a changing climate requires highly accurate agricultural forecasting beyond what traditional survey data can provide. Descartes Labs, a research- and analytics-driven company in the Southwestern United States, is helping the world address food security crises and even identify early signs of famine. By using machine learning to analyze years of scientifically calibrated satellite imagery, the company can successfully predict changes in crop health and yield.
Named after the French mathematician René Descartes — who discovered that the position of a point can be determined by coordinates — Descartes Labs provides instant, programmatic access to satellite images of any geographic location. Using environmental change analysis, the company gives customers information about the global food supply through deep learning, remote sensing, and large-scale, high-performance computing. Governments, academic researchers, and food producers can use its forecasts to help ensure crop harvests are sufficient and that critical links in the food chain remain economically healthy.
Descartes Labs' forecasting platform uses satellite imagery to capture insights about more than just crop health. It offers insights into human populations, natural resources, the growth of cities, the spread of forest fires, and the state of available drinking water across the globe. The company also makes its platform available to organizations that want to gain insights from their own data to optimize pricing and better understand their customers.
"We are constantly getting petabytes of new data from hundreds of satellites. With Google Cloud, we don't worry about whether the compute, network, or storage can scale and, instead, can focus on improving models and analyzing larger datasets for better forecasts."
—Tim Kelton, Co-founder and Cloud Architect, Descartes LabsBuilding a timely, dynamic atlas of the world with deep historical coverage of the entire planet involves massive datasets — and the amount of data grows all the time. When Descartes Labs learned that Google Earth Engine hosts all NASA Landsat satellite imagery since 1973 natively on Cloud Storage, the company jumped at the chance to use images — the longest continuous observations of the Earth ever captured — to back-test models over many years. However, it needed a practical way to process more than 1 petabyte of satellite imagery of U.S. corn production data (4 quadrillion pixels) without setting up a large physical infrastructure.
As a startup company in a highly competitive space, Descartes Labs could not afford to wait months to prove its viability in the agricultural research market. By using Compute Engine and leveraging high-end Intel Xeon Scalable processors for added performance, Descartes Labs is able to scale compute, networking, and storage to process the entire Landsat image archive in just over 15 hours. By enabling historical back-testing, the company indicates it can now predict corn yields faster and more accurately than government organizations. The trading markets believe this to be true and on the first day that Descartes Labs published its forecasting models the company received major media coverage, and the corn futures market moved by 3 percent.
"Just a few years ago, we would have needed the world's largest supercomputers to do what we can now do with Google Cloud," says Mark Johnson, CEO, Descartes Labs. "Compared with other cloud solutions, Google Cloud offers unmatched scalability, which is a business requirement as we push toward exascale computing."
"We are constantly getting petabytes of new data from hundreds of satellites," says Tim Kelton, Co-founder and Cloud Architect, Descartes Labs. "With Google Cloud, we don't worry about whether the compute, network, or storage can scale and, instead, can focus on improving models and analyzing larger datasets for better forecasts."
Supercomputing in the cloud with Google Cloud
Descartes Labs now has the ability to scale its proprietary machine learning tools on demand by using Google Cloud to process even the largest datasets, including the European Space Agency's Sentinel satellite constellation. It has developed the first-ever global composite views from some of these satellites, showing different frequency bands to monitor changes in vegetation and the Earth's surface.
To help keep costs low, it uses preemptible VMs, Compute Engine instances that are extremely affordable because they are short-lived. "Using Intel Xeon Scalable processors helps us achieve even better price/performance in Compute Engine," says Tim. To help keep performance high, Descartes Labs uses tens of thousands of CPUs to ingest the imagery and high-bandwidth links to Cloud Storage, where the compressed imagery is stored. Soon, Descartes Labs expects to have nearly 15 petabytes of processed imagery on Cloud Storage, where it can be analyzed at any time.
"We use Cloud Storage as our large distributed file system because we know it will scale as our datasets grow beyond multiple petabytes," says Tim. "We saw aggregate read bandwidth of 230 gigabytes per second using 512 compute nodes, which is comparable to the best HPC storage systems in existence."
"Google Kubernetes Engine lets us get code into production faster and provide better APIs to our customers," adds Tim. "It's like playing Tetris with our workloads — everything automatically goes in the best slot, and it's constantly checking to make sure services are always available."
"We wouldn't be able to handle the amount of data we're managing daily without Google and Intel. We've already seen file compression rates improve by 38 percent and image extractions speed up by 24 percent."
—Tim Kelton, Co-founder and Cloud Architect, Descartes LabsFor Descartes Labs, the productivity improvements were supported by the company's use of Intel hardware. On Google Cloud, the company's solution is powered by Intel Xeon Scalable processors, which provide best-in-class hardware for many applications. Along with faster performance, the Xeon Scalable series also features Advanced Vector Extensions 512 support, which effectively doubles the amount of data that the processor can process at one time and improves compression speeds.
"We wouldn't be able to handle the amount of data we're managing daily without Google and Intel," says Tim. "We've already seen file compression rates improve by 38 percent and image extractions speed up by 24 percent. The combination of the Intel Xeon Scalable processors and preemptible VMs has lowered our costs from $10,000 to $2,700 per petabyte analyzed in Google Cloud."
Building on Google innovation with BigQuery GIS
As satellite images are analyzed, Descartes Labs captures information about each scene as vector data, and uses Cloud Pub/Sub in conjunction with a microservice hosted on Google Kubernetes Engine to persist that information into BigQuery for geospatial analysis. By bringing large GIS workloads directly into its data warehouse in BigQuery, Descartes Labs is able to simplify some of its GIS processes and take advantage of the serverless scalability of BigQuery.
"With BigQuery GIS we can provide a highly scalable, no ops API layer to our customers for geospatial analysis of any size." says Kelton. "Plus, its billing and cost structure aligns to one that matches our use patterns."
Descartes Labs also uses BigQuery to analyze logs from its applications and the APIs it provides to customers, gaining valuable insight to improve products. For monitoring and alerting it uses Stackdriver Monitoring, helping engineers resolve issues quickly. To store visual signatures that identify landmarks or structures such as wind turbines, solar farms, golf courses, or airport runways for the company's GeoVisual Search tool, it uses Cloud Bigtable, which returns search results much faster than a standard relational database.
"A major advantage of Google Cloud is that we get access to the same tools Google uses to power its own services, which helps us improve developer efficiency," says Tim. "Cloud Bigtable is a perfect example. Google has expertise in search, and we can benefit from the same technology."
Just a few years ago, Descartes Labs was a new company needing to maximize productivity on a limited budget. Google Workspace was the natural choice to help the company grow, and today it supports ten times the number of original employees. Descartes Labs depends on Gmail, Drive, Sheets, Docs, and Calendar to keep business moving, and regularly uses Google Meet to meet with customers and connect employees across offices.
"For a company with dreams to change the world, our collaboration with Google is invaluable. The Google Cloud team has been eager to work with us, helping us scale and get maximum benefit from Google Cloud."
—Mark Johnson, CEO, Descartes Labs"Google Workspace offers excellent security with multi-factor authentication, which is important to us. We started our company using Google Workspace and never looked back," says Tim. "It's an effective and affordable way to communicate and collaborate on the devices our teams rely on."
Scaling for the future of forecasting
As the cost of satellites and satellite launches comes down dramatically, the amount of image data observing the earth is dramatically increasing. With Google Cloud and Intel, Descartes Labs is enabling an understanding of the world at a scale, cost, and level of granularity that would have been impossible just a few years ago. The company expects to have more than 30 petabytes on Google Cloud within the next few years.
As datasets grow, it is exploring new ways to make model training even more efficient, including using Tensor Processing Units (TPUs) from Google to accelerate machine learning. Current machine learning applications for Descartes include geographic data analysis from satellite imaging and customer-designed models, and the company continues to explore future AI-driven uses on Google Cloud. Intel Xeon Scalable processors have played a vital part in running this machine learning model training, which can often require more than 1,000 iterations in initial calibration.
Descartes Labs collaborates with Google and runs Intel Xeon Scalable processors to learn about the latest innovations that can help maximize the value of its forecasting platform. The results of this collaboration could have enormous and lasting impact, helping people worldwide avoid food shortages and other potentially catastrophic events.
"For a company with dreams to change the world, our collaboration with Google is invaluable," says Mark. "The Google Cloud team has been eager to work with us, helping us scale and get maximum benefit from Google Cloud. We look forward to continue working with Google to push the limits of cloud computing."
Tell us your challenge. We're here to help.
Contact us
About Descartes Labs
Descartes Labs created a cloud-based supercomputing platform to apply machine intelligence to massive datasets. Its mission is to better understand the planet, for good and for profit. Today, Descartes Labs uses satellite imagery and advanced data and analytics processing in Google Cloud to model complex systems on the planet, like forestry and agriculture, to enable organizations to "see" and understand the world in a whole new way.
About Intel Corporation
Intel Corporation, a Google Cloud Technology Alliance partner, is known for being a market leader and one of the world's best-known chip producers, but Intel does much more. Harnessing the capability of the cloud and the ubiquity of the Internet of Things, Intel is disrupting industries and solving global challenges. Leading on policy, diversity, inclusion, education, and sustainability, Intel creates value for its stockholders, customers, partners, and society.