EvalCore

EvalCore revolutionizes the evaluation of advanced textual applications and agents by providing a robust, single-binary test runner. Key features include: • Define test cases and scorers using YAML configurations • Run local targets on every pull request for immediate feedback • Replay model or judge calls offline at no cost • Support for REST and shell targets, baselines, and trials • Detailed model comparisons and OpenTelemetry traces for deep insights EvalCore enables developers to precisely track application behavior. It records every request and response to a local SQLite cassette, ensuring that changes to prompts, models, or dependencies are rigorously checked against established baselines. This snapshot testing approach guarantees deterministic results and eliminates flaky tests, crucial for maintaining application quality. This tool is ideal for engineering teams focused on continuous integration and delivery. It offers a comprehensive suite for agent performance evaluation, allowing for rapid iteration and confidence in deployments. By replaying tests offline, EvalCore ensures that performance validation is both efficient and cost-effective. Seamlessly integrate into existing CI pipelines to catch regressions before they impact users.