Announcing FiberFS
FiberFS is a new filesystem designed for high-performance computing and AI workloads, leveraging the Linux kernel's io_uring interface for asynchronous I/O. It aims to provide low-latency, high-throughput data access by bypassing traditional page cache overhead. The project is open-source and targets modern storage hardware like NVMe SSDs.
Background
FiberFS is a new open-source filesystem designed to dramatically improve performance for AI/ML workloads by making GPUs the primary consumer of data — rather than traditional CPUs. It targets a core bottleneck in distributed training: feeding data fast enough to keep hundreds or thousands of GPUs busy. Distributed training involves splitting a model across many GPUs that must constantly exchange data; slow filesystem I/O creates "GPU starvation" where expensive compute units sit idle waiting for data. FiberFS employs a "p2p pull" architecture (GPUs directly fetch data from each other or from storage nodes over RDMA networks) instead of the traditional client-server model orchestrated by a CPU. The project is led by a team from MIT's Computer Science and AI Lab and builds on ideas from their earlier research (NoPFS and Skyloft). It competes in a space that includes systems like Alluxio, JuiceFS, and specific filesystem optimization efforts for deep learning frameworks.