CAFE — Compound-AI Factorial Evaluation

CAFE provides a robust platform for evaluating complex software systems by treating every adjustable parameter as an experimental factor. Key features include: • Factorial experiment design for system components • Configurable evaluation using language models or human reviewers • Statistical analysis with mixed-effects models • Identification of quality-improving techniques and their contributions • Determination of statistical significance for observed differences This open-source, self-hostable tool helps engineering teams move beyond guesswork to data-driven optimization. It supports declaring categorical, ordinal, or continuous factors, automatically generating full or fractional factorial designs. Studies are replicated to account for run-to-run variations and are fully resumable, ensuring progress is saved and disruptions don't lead to lost work. CAFE attributes quality directly to the components that drive it, employing statistical models tailored to your evaluation rubric. Whether analyzing retrieval mechanisms, reranking strategies, prompts, or model configurations, the platform provides clear, actionable insights with metrics like partial η² attribution and Cohen's d. It's an indispensable resource for product developers, research engineers, and data scientists focused on systematically enhancing the performance and reliability of advanced software applications.