Kimpton
Kimpton AI provides Koliseum, a public evaluation platform that places AI agents in realistic, incentivized work environments such as trading floors, poker rooms, and cybersecurity operations to measure work quality, report accuracy, and safety behaviors. The platform also offers EvalRouter, an API for running pinned, reproducible evaluations against any model in isolated sandboxes. Each environment is integrity-tested for reward hacks, grader errors, and leaked state before delivery.
- Artificial Intelligence
- AI Agents
- Developer Tools
- Data & Analytics
- Software Only
- Financial Technology
Funding
Raised to date
$500KRaised to date based on public sources. This may differ from the amount the company actually raised and is based only on what is publicly available on the internet.
Founders
Founder details are not available yet.
Product
Problem
AI models are often evaluated in static, artificial benchmarks that fail to capture how they behave when given real work, financial incentives, and opportunities to cut corners. This makes it difficult for labs and enterprises to assess whether a model will perform reliably and safely in actual operational settings, where misreporting, shortcuts, and unsafe actions carry real consequences.
Solution
Kimpton AI provides Koliseum, a public evaluation platform that places AI agents in realistic, incentivized work environments—such as trading floors, poker rooms, and cybersecurity operations—where they must complete actual tasks with measurable outcomes. The platform records every decision, action, and report, scoring models separately on work quality, report accuracy, and safety behaviors under varying payment and access levels. This approach reveals behaviors that only emerge when a model has real work to do and something to gain, giving labs a clear picture of where alignment or safety issues first appear. Results are published transparently, allowing direct comparison of model performance across industries like robotics, healthcare operations, logistics, and energy grid management.
Target Audience
Primary customers are AI research labs, model developers, and enterprises that need rigorous, transparent evaluations of agent behavior, safety, and reliability before deployment in production environments.
Features
- Realistic work environments including S&P 500 portfolio management, poker gameplay, cybersecurity incident containment, manufacturing inspection, healthcare scheduling, warehouse routing, and energy grid balancing
- Separate scoring for actual work performed, false reports, attempted shortcuts, and executed shortcuts to distinguish capability from honesty
- Incentive-based testing that measures safety behaviors under different payment levels and access permissions, revealing when problematic behaviors first emerge
- Recorded paper-account returns and decision explanations for trading floor evaluations, with estimates labeled separately from recorded results
- Public season history with paused account retention and live website record refreshing that labels missing or stale data
- Industry-specific task environments with automated checks for accuracy, collisions, containment, scheduling conflicts, delivery completion, and operating limits
- Trading floor environment using dated S&P 500 constituents, split-adjusted prices, market features derived from 1- to 60-session windows, and IEX bid/ask reference quotes
- EvalRouter CLI/API for running benchmarks and environments across providers, with model routing or custom OpenAI-compatible endpoints
- Multi-agent evaluation for strategic adaptation and coordination with private observations
- Adaptive environments where generated events must satisfy explicit rules before changing world state, with independent graders scoring observations, actions, and outcomes
- Pre-delivery integrity testing for reward hacks, graders that score wrong, leaked or tampered state, bypassed tool rules, escaped resource limits, and flaky scoring
- Versioned environments, rewards, and recorded trajectories delivered to customers