CarbonForge provides a runtime optimization loop that measures sub-millisecond power and latency at the kernel level, then automatically searches for and locks the operating point that maximizes token throughput on each GPU while staying within a fixed power envelope.
Funding
Funding not disclosed
Founders
Product
Problem
Inference workloads on GPU fleets often operate under a fixed power envelope, but existing monitoring tools only provide coarse, averaged telemetry, leaving most optimization opportunities undiscovered. As model updates, traffic patterns, and hardware changes shift the optimal operating point, performance degrades without a mechanism to continuously identify and lock the best settings.
Solution
CarbonForge introduces a runtime optimization loop that continuously measures power consumption and latency at sub‑millisecond granularity with kernel‑level attribution. An on‑device optimization engine searches the operating point that maximizes token throughput while respecting the fixed power budget. The loop then re‑locks the GPU to this optimal point, delivering more tokens per GPU without any changes to the inference stack. CarbonForge runs alongside popular inference frameworks such as vLLM and TensorRT, integrating transparently into existing deployments. By exposing fine‑grained telemetry and automated control, it captures performance gains that traditional coarse monitoring misses.
Target Audience
Primary customers are organizations that run large‑scale GPU inference fleets, such as AI service providers, cloud operators, and enterprises deploying high‑throughput language models.
Features
- Sub‑millisecond power and latency telemetry with per‑kernel attribution
- Automated optimization engine that searches for the highest‑throughput operating point within a fixed power envelope
- Runtime controller that re‑locks GPUs to the discovered optimal settings in real time
- Seamless integration with vLLM, TensorRT, and other inference stacks without code modifications
- Continuous loop that adapts to model updates, traffic shifts, and hardware refreshes