MK1 Flywheel is a high-performance LLM inference engine that integrates directly into existing software stacks, allowing businesses to manage GPU resources efficiently while keeping customer data and model weights secure. It enables faster response times and higher request processing rates, optimizing token costs by allowing users to utilize their own GPUs and cloud contracts without vendor lock-in.
Funding
Funding not disclosed
Founders
Product
Problem
Existing large language model (LLM) inference engines often lead to vendor lock-in, inefficient GPU resource utilization, and concerns about the security of customer data and model weights. Businesses struggle to optimize token costs and maintain control over their AI infrastructure.
Solution
MK1 Flywheel is a high-performance LLM inference engine designed for direct integration into existing software stacks, enabling businesses to efficiently manage GPU resources while ensuring the security of sensitive data. It allows for faster response times and increased request processing rates, optimizing token costs by providing the flexibility to leverage existing GPU resources and cloud contracts without vendor dependence. MK1 Flywheel offers a seamless replacement for vLLM, TensorRT-LLM, and HuggingFace TGI, delivering high performance without complex configurations and the option for tight integration within existing infrastructure.
Target Audience
The primary target audience includes businesses utilizing LLMs that require high-performance inference, efficient resource management, and control over data security and infrastructure.
Features
- Drop-in replacement for vLLM, TensorRT-LLM, and HuggingFace TGI
- High performance without configuration
- Option for tight integration within existing software stacks
- Flexibility to utilize existing GPUs and cloud contracts
- Seamless switching between NVIDIA and AMD backends
- Supports running open-source 1M context Llama3-70B model on AMD MI300X hardware