Object store built for AI workloads

Anvil is an open-source object store specifically engineered for modern data processing needs. It addresses the limitations of traditional storage solutions like Git LFS, Hugging Face repositories, and S3 when handling large model files. Key features include model-aware indexing, enabling it to understand formats such as safetensors, gguf, and ONNX. It supports tensor-level streaming for efficient data access, and utilizes erasure-coded storage for enhanced data integrity and space efficiency. Anvil is fully S3-compatible and features a gRPC-native interface, offering both broad interoperability and high-performance communication. This storage system is ideal for those managing large models, needing to version fine-tunes, or running local inference. It's built to overcome challenges such as slow downloads of multi-gigabyte files and the inefficient duplication of base checkpoints common in model development. Anvil helps maintain developer velocity and enables execution of models on systems with limited resources, even facilitating serving models from edge devices. Anvil is available as an open-source project under the Apache-2.0 license, promoting transparency and community contribution. It empowers developers and researchers who require robust, efficient, and specialized storage for their complex data assets, allowing them to focus on innovation rather than storage bottlenecks.