Fabric 1.5

Fabric 1.5 is a compact yet powerful causal language model, designed for processing extensive contexts efficiently. Key capabilities include: * 32,768-token context window * Sub-quadratic memory consumption * Integration of Chunked Fabric Memory * Support for long-form content generation * Built for high-performance processing This model introduces Chunked Fabric Memory, a novel mechanism that intelligently compresses past contextual data into learned summaries. These summaries are then seamlessly blended with local attention using a learned gating function, allowing for significantly larger context windows (up to 32,768 tokens) without the exponential memory costs typically associated with such scales. Its architecture combines LocalBlock (Grouped Query Attention) and FabricMemoryBlock components, where every third layer augments standard local attention with a memory attention branch over these learned chunk summaries. Trained on a diverse 12 billion token dataset, including high-quality educational text (FineWeb-edu), deduplicated web crawls (DCLM), and mathematical content (OpenWebMath), Fabric 1.5 is suitable for a wide range of text processing tasks. Developers can leverage its capabilities for applications requiring deep contextual understanding and generation. It's an excellent choice for engineering teams and researchers seeking performant language models for document analysis, content expansion, and advanced text processing when memory efficiency and large context handling are critical.