Ashr provides a testing platform for AI agents that centralizes evaluation data, showing detailed conversation timelines, speaker inputs, tool calls, and responses. It lets developers replay runs, compare expected versus actual behavior with versioned prompt diffs, and automatically detect regressions, while its Python and TypeScript SDK integrates evaluations into CI/CD pipelines for continuous testing.
Funding
Funding not disclosed
Founders
Product
Problem
Developers of AI agents often lack comprehensive tools to automatically evaluate, trace, and compare agent behavior against expected outcomes, making it difficult to identify regressions and production failures before deployment.
Solution
Ashr offers a testing platform that aggregates evaluation data from every run of an AI agent, presenting detailed timelines of speaker inputs, tool calls, and responses. Users can replay conversations, view side‑by‑side diffs of expected versus actual actions, and pinpoint specific failure modes. The platform tracks versioned prompts and pass rates, highlighting which code changes introduced regressions. An SDK for Python and TypeScript enables teams to integrate evaluation runs directly into their CI/CD pipelines, allowing continuous testing and rapid iteration before shipping to production.
Target Audience
Primary customers are AI developers and engineering teams building conversational or tool‑using agents who need systematic testing and regression monitoring, as well as product teams deploying agents at scale.
Features
- Centralized dashboard displaying status, traces, and scores for all evaluation runs
- Full conversation timelines with replay capability to inspect each step of the agent’s execution
- Expected vs. actual diff view with inline prompt versioning and pass‑rate metrics
- Automated regression detection highlighting new failures introduced by code changes
- Python and TypeScript SDK for programmatic execution of evaluations within existing workflows
- Exportable metrics and detailed reports to support debugging and compliance needs