Cipherra

Cipherra offers a robust platform for evaluating agent performance at scale. Key capabilities include: * Support for customized models * Integration with Harbor task format * Ready for GitHub Actions * Scalable evaluation infrastructure * Prioritized diagnostic reports This platform addresses common challenges in agent evaluation, such as flaky runs, manual pipelines that don't scale, and the lack of actionable insights from raw scores. It provides built-in redundancy, running the same task multiple times to account for non-deterministic model outputs, ensuring reliable results. From submission to diagnosis, Cipherra streamlines the evaluation process. Users can upload Harbor-format task bundles, configure model settings, and define redundancy N and step limits. The platform then executes N independent runs per task using the specified model endpoint, averages the results, and surfaces score variance. Crucially, every failure is classified by its root cause, offering prioritized issues with severity rankings and specific remediation steps. Cipherra is ideal for engineering teams focused on reinforcement learning post-training and large-scale agent evaluations or benchmarks. It integrates seamlessly into existing development pipelines via GitHub Actions, webhooks, or CLI, providing a web dashboard and REST API for comprehensive job submission and live results monitoring. The platform supports any OpenAI-compatible endpoint, including self-hosted vLLM or local Ollama instances, allowing users to leverage their preferred model infrastructure.