Google supports launch of DAOS Foundation, for next-generation HPC and AI storage

Today, we’re proud to announce that Google Cloud is a founding member of the DAOS Foundation, launched today by the Linux Foundation. Created in collaboration with partners, the DAOS Foundation will accelerate investment and development of the world-record-setting open-source DAOS storage stack, which supports next-generation HPC and AI/ML workloads.

Intel Corporation initiated investment in DAOS in 2012 to drive a new paradigm in storage for high performance computing (HPC). The combination of fully-distributed metadata management, a key-value store architecture, and the lack of dependency on kernel modifications has led to notable breakthroughs, such as:

• impressive gains in throughput, I/O, and metadata operations performance
• easier container/Kubernetes integration
• increased architectural flexibility and decreased infrastructure overhead (eliminating the need for metadata servers)

The upcoming deployment at Argonne National Laboratories, which is targeted to deliver >25TB/s of throughput for HPC, demonstrates the significant impact DAOS has had on the world of scientific computing.

Further, we recently launched Parallelstore, a scratch storage service based on DAOS that delivers up to 6.3x read-throughput performance compared to competitive Lustre scratch offerings. Both large language models (LLMs) and HPC have demanding IOPS, throughput, MDOPS, and ultra-low latency requirements. Customer demand for AI/ML training based on large data sets with small files that require low-latency operations aligns extremely well with the performance characteristics and architecture of DAOS.

“Google is delighted to be a founding member of the DAOS Foundation,” said Sameet Agarwal, Google Storage VP/GM. “This industry-leading architecture underpins our new storage service, Parallelstore, and we are excited to accelerate the growth of HPC and AI/ML workloads in the cloud.”

The DAOS Foundation will be established as a Linux Foundation project, which provides a structure for shared governance and continued investment in the open source community. The collaboration across companies has been strong and Google Cloud is excited to continue this investment.