tldt — Too Long, Didn't Tokenize

Too Long; Didn't Tokenize (tl;dt) is a command-line interface and library designed to summarize extensive texts efficiently. It addresses common challenges encountered with large volumes of text data. Key features include: • Extractive summarization using LexRank and TextRank • Protection against Unicode confusables and prompt manipulation • Conversion of HTML content to Markdown • Sanitization of text, including removal of PII and API keys • No external API keys or subscription costs required This tool is ideal for developers and teams working with large documents, crawled sites, or extensive transcripts. It helps in reducing token costs and preventing issues like prompt injection and side instructions by providing concise, context-preserving summaries. The output consists of exact quotes from the original source, ensuring accuracy and preventing fabricated content. It integrates as a Go library, making it suitable for developers building applications that process and condense text. Perfect for engineering teams, content processors, and developers who need to condense large textual data without external service dependencies, ensuring data privacy and cost efficiency.