OneTriangle.ai provides a model-agnostic inference platform that separates expensive prefill from fast decode by transferring KV caches between models of different sizes language models. The company's technology enables significant reductions in time-to-first-token, cost, and latency by letting a small model read a prompt and a larger model generate answers from that transferred context. Their platform offers production-ready endpoints for models like DeepSeek V4 Flash, Llama 3.3 70B, and specialized audio and safety classifiers at per-token pricing.
Funding
Funding not disclosed
Founders
Product
Problem
Large language models incur high prefill costs because reading prompt context scales with the size of the model performing the read, even before generation begins. This makes serving long-context requests expensive and slow, particularly when a large model must process a long input before producing a single output token. The cost and latency problem is amplified for high-volume production workloads that repeatedly process substantial prompts.
Solution
OneTriangle.ai separates prefill from decode by building KV caches with a smaller, cheaper model and transferring that cache into a larger target model for generation. A learned mapper projects the small model's K/V tensors into the target model's attention space, while a residual correction layer repairs quality gaps. This "prefill small, decode large" approach cuts time-to-first-token by up to 6.3× and reduces prefill cost by 84% for models like Llama 3.1 8B to 70B. Transfer quality is gated on held-out evaluations, and the system falls back to ordinary prefill when a model pair does not pass quality or latency thresholds.
Target Audience
Primary customers are AI engineering teams operating production inference workloads at scale, including enterprises using LLM APIs for assistants, retrieval, tool use, and audio applications; the pricing model targets high-volume users needing cost-effective long-context processing.
Features
- KV-cache transfer mechanism that maps a small model's attention tensors into a larger target model's geometry, eliminating redundant prefill computation
- Learned ridge mapper plus rank-16 residual MLP corrections that preserve task quality, demonstrated with 91.2% quality retention and up to 8.7× faster prefill handoffs
- DFlash parallel drafting that predicts 16-token blocks in a single forward pass, achieving 2.154× geometric-mean decode speedup with exact token-ID agreement against native decoding
- Sealed test-set evaluation discipline using SHA-256 pre-registration, frozen checkpoints, and checksum-verified serving bridges to avoid overfitting in reported benchmarks
- Audio-specific KV transfer for voice models like Ultravox, enabling 31.2% accuracy versus 17.2% native 1B at comparable decode speeds of 133-150 tokens per second
- Production endpoints with per-token pricing and cached-input discounts, including DeepSeek V4 Flash, Llama 3.3 70B Fast with 1.61× measured decode improvement, and Vision/action-policy models in limited beta