Co-Founder/YC W24
nCompass Technologies
nCompass provides an IDE extension that unifies code profiling, trace viewing, and trace analysis directly within the developer environment. This platform allows engineers to easily identify performance bottlenecks by integrating runtime profiling data with coding workflows. The tool aims to streamline the process of optimizing code for speed and efficiency on complex systems.
- Artificial Intelligence
- Developer Tools
- Enterprise Software
- Software Only
Updated 23 months ago
Funding
Raised to date
$500KRaised to date based on public sources. This may differ from the amount the company actually raised and is based only on what is publicly available on the internet.
Funding rounds are not available yet.
Founders
Product
Problem
Serving AI models at scale is expensive, and existing solutions struggle to maintain quality of service (QoS) when handling high request volumes, leading to increased infrastructure costs. Current state-of-the-art serving systems can experience significant response time degradation when stressed with request rates exceeding their capacity. Scaling up the number of GPUs is often the only recourse, further driving up expenses.
Solution
nCompass Technologies offers a hardware-aware request scheduler and Kubernetes autoscaler designed to optimize GPU utilization for AI model serving. Their technology enables a single GPU to handle significantly more requests per second without compromising QoS, resulting in substantial cost savings. By intelligently managing requests and dynamically scaling resources, nCompass Technologies ensures low latency and high throughput for AI inference workloads. This approach allows users to achieve greater efficiency and responsiveness compared to traditional scaling methods.
Target Audience
The primary customers are organizations that deploy AI models at scale and seek to reduce infrastructure costs while maintaining low latency and high throughput.
Features
- Hardware-aware request scheduler that optimizes GPU utilization
- Kubernetes autoscaler for dynamic resource allocation
- Enables a single GPU to handle over 100 requests per second
- Maintains a time-to-first-token (TTFT) of less than one second
- Improves AI model responsiveness by up to 18x compared to existing solutions
- Reduces the cost of serving AI models at scale by 50%