Thaw provides a fork primitive that snapshots a live LLM inference session—including model weights, KV cache, scheduler state, and prompt prefix—so multiple child branches can continue generation without re‑prefilling the prompt. This enables sub‑second fork latency and high‑throughput branching for RL rollouts, tool‑call fan‑out, and tree‑search workloads on CUDA‑compatible GPUs.
Funding
Funding not disclosed
Founders
Product
Problem
Running large language models for tasks like reinforcement‑learning rollouts, tool‑call fan‑out, or best‑of‑N generation requires re‑prefilling the same prompt prefix for each branch, incurring significant GPU compute and latency. This limits throughput and makes large‑scale inference inefficient, especially on high‑end GPUs where the prefill step dominates runtime.
Solution
Thaw introduces a fork primitive that snapshots a live LLM inference session—including model weights, KV cache, scheduler state, and prefix hash table—without tearing down the engine. By restoring this snapshot, multiple child branches can continue generation from the exact token where the fork occurred, skipping the costly prefill step. The implementation leverages pinned memory, pipelined DMA, and O_DIRECT writes to achieve sub‑second per‑fork latency (median 0.88 s) and high restore bandwidth (≈14 GB/s). Thaw integrates with vLLM and SGLang and supports major LLM architectures (Llama, Qwen, Mistral, Gemma, Phi, DeepSeek, GLM, Yi) on CUDA‑compatible GPUs, enabling efficient RL rollouts, tool‑call fan‑out, and tree‑search workloads.
Target Audience
Primary users are developers and researchers building reinforcement‑learning pipelines, coding‑assistant systems, or any application that requires branching multiple generations from a shared LLM context.
Features
- Snapshot and restore of full inference state (weights, KV cache, scheduler, prefix hash) without engine shutdown
- Sub‑second fork latency with 14 GB/s peak restore bandwidth via pipelined DMA and CRC32C checksums
- Zero‑prefill for all child branches, eliminating repeated prompt processing
- Compatible with vLLM and SGLang, supporting a wide range of LLM architectures on CUDA 12+ GPUs
- Scales across multiple workers or within a single process; can hydrate N children from one fork point
- Open‑source Apache‑2.0 license, pip‑installable, runs on any CUDA‑compatible GPU without licensing fees