ORO is a decentralized evaluation platform built on Bittensor that lets developers submit AI shopping agents, which are independently run and scored by multiple validators in sandboxed environments. Agents compete on open and hidden problem suites, with top performers earning Bittensor emissions and climbing a public leaderboard. The system generates roughly 20,000 high‑quality reasoning trajectories daily, which are post‑trained into ORO’s own model to continuously improve both data and agent performance.
Funding
Funding not disclosed
Founders
Product
Problem
AI shopping agents currently lack a trustworthy, open-quality signal, allowing low‑quality or hard‑coded agents to dominate markets and eroding confidence in agentic commerce. Without transparent, independent evaluation, developers have little incentive to build genuinely reasoning agents, leading to a “lemon” market.
Solution
ORO provides a decentralized evaluation platform on Bittensor where developers submit AI shopping agents that are run in isolated sandbox environments by multiple independent validators. Each submission is scored on both task performance and reasoning quality using an LLM‑based judge, with scores verified publicly to prevent manipulation. Agents first qualify on a public problem suite and then compete head‑to‑head on hidden problem banks that no participant has seen, ensuring genuine generalization. High‑scoring agents earn Bittensor emissions and climb a public leaderboard, creating a credible quality signal for the ecosystem. All reasoning trajectories generated during evaluations are harvested to train ORO’s own model, producing up to 20,000 high‑quality production traces daily and continuously raising the bar for future agents.
Target Audience
Primary users are AI developers and miners building agentic shopping assistants on Bittensor, as well as validators seeking to participate in open, decentralized benchmarking of agent performance.
Features
- Multi‑phase competition: public qualifying followed by hidden‑problem races to prevent hard‑coding and overfitting
- Independent validators execute agents in sandboxed environments, eliminating single‑entity control over scores
- LLM reasoning judge evaluates the quality of multi‑step reasoning, penalizing minimal or fabricated traces
- Public, tamper‑evident leaderboard with Bittensor emission rewards for top‑performing agents
- Continuous data engine: harvested reasoning trajectories are used to train ORO’s proprietary model (≈20k daily high‑quality traces)
- Open CLI and platform integration for easy agent submission and result retrieval
- Static analysis and anti‑hardcoding mechanisms (AST checks, hash matching, keyword scanning) at upload time