oMLX

oMLX transforms your Mac into a powerful local language model server, accessible directly from your menu bar. It's engineered to accelerate performance for text, vision, OCR, embedding, and reranker models. Key capabilities include continuous batching for efficient processing of multiple requests and a unique RAM+SSD tiered KV cache that retains data even after restarts. This innovative caching mechanism significantly reduces response times, allowing applications like Claude Code and Cursor to respond in approximately 5 seconds instead of 90 seconds. The server offers OpenAI and Anthropic compatible APIs, enabling seamless integration with existing applications without modifications. Built natively in Swift, oMLX ensures optimal performance and a lightweight footprint, avoiding the overhead of Electron-based solutions. This robust tool is ideal for developers, researchers, and power users who require high-performance local model inference on Apple Silicon. It supports simultaneous loading of multiple model types, with smart LRU eviction for memory management and a web dashboard for effortless model administration and real-time monitoring. The open-source Apache 2.0 license promotes transparency and community contribution. oMLX empowers users to run advanced models locally with unprecedented speed and efficiency, perfect for intensive coding assistance, content understanding, and data processing tasks.