BenchSpan provides a hosted platform that runs AI‑agent benchmarks in isolated Docker containers, allowing developers to launch agents with a single shell command and execute large benchmark suites in parallel. The service automatically retries failed runs, records scores, timings, and error traces, and stores results centrally tagged by the agent’s commit hash for easy comparison and sharing across teams.
Funding
Funding not disclosed
Founders
Product
Problem
AI agents need reliable evaluation, but existing benchmarks require custom integration code, run sequentially on local hardware, and produce results that are hard to reproduce or share. Slow execution, high token costs on failures, and fragmented result storage impede rapid iteration and team collaboration.
Solution
BenchSpan offers a hosted platform that runs AI‑agent benchmarks in isolated Docker containers, eliminating the need for bespoke harnesses. Users provide a single shell command to launch their agent, select from a curated library of benchmarks or upload custom ones, and the system executes all instances in parallel. The platform automatically retries only the failed runs, captures scores, timings, and error traces, and stores every result centrally with the exact commit hash of the agent. A searchable dashboard lets teams compare runs side‑by‑side, trace changes across commits, and share validated metrics without manual data handling. Optional white‑glove onboarding containers custom evaluations while preserving the same parallel, reproducible execution model.
Target Audience
BenchSpan is aimed at AI‑agent developers, research labs, and ML engineering teams that need fast, reproducible performance evaluation and collaborative result tracking across multiple commits and team members. It also serves enterprises that run large‑scale agent assessments and require centralized auditability of benchmark outcomes.
Features
- One‑time onboarding: a bash script that starts the agent is the sole integration point; no framework lock‑in or interface conformance required.
- Parallelized execution: each benchmark instance runs in its own Docker container, reducing a 500‑instance suite from hours to minutes.
- Selective retry: only failed instances are re‑executed and merged with prior results, avoiding duplicate token costs.
- Deterministic environments: identical Docker images and benchmark versions are used for every run, with results tagged by the agent’s commit hash.
- Centralized result store: scores, trajectories, and error logs are automatically aggregated, searchable, and visualized in a web dashboard.
- Extensive benchmark library: 27 ready‑to‑run benchmarks covering coding, math, reasoning, knowledge, data science, tool use, and safety.
- Custom benchmark onboarding: white‑glove service creates Dockerized harnesses for proprietary evaluations within 1–2 business days.
- Smoke‑test mode: run a small subset of instances to validate setup before launching large‑scale evaluations.