AutoScale.AI provides a GPU control plane that adds multi‑tenant, fractional GPU scheduling and utilization optimization to Kubernetes clusters running AI workloads. By integrating with Ray and popular frameworks such as PyTorch and vLLM, it enables seamless orchestration of training and inference jobs on bare‑metal GPU farms, improving resource efficiency and reducing contention across competing workloads.
Funding
Funding not disclosed
Founders
Product
Problem
Kubernetes can schedule containers but lacks native support for multi‑tenant GPU allocation, fractional GPU sharing, and real‑time utilization optimization, leading to under‑utilized bare‑metal GPU clusters and contention between training and inference jobs.
Solution
AutoScale.AI provides an eBPF‑based GPU control plane that integrates between Kubernetes orchestration and Ray distributed compute. By attaching kernel‑level probes to GPU drivers, it supplies Kubernetes with real‑time metrics on VRAM pressure, compute occupancy, and thermal state. The control plane uses this data to apply dynamic taints and perform scheduling decisions on a 500 ms reconciliation cycle, enabling fractional GPU sharing and multi‑tenant scheduling. It supports major ML frameworks such as PyTorch and vLLM, allowing both training and LLM inference workloads to run efficiently on bare‑metal GPU clusters across cloud providers. The solution abstracts GPU management, letting operators deploy AI workloads at production scale without custom scripting or manual tuning.
Target Audience
Primary customers are AI/ML engineering teams and DevOps groups that run large‑scale training or LLM inference workloads on Kubernetes‑managed bare‑metal GPU clusters, particularly in enterprises adopting Ray and vLLM.
Features
- eBPF‑driven control plane that injects kernel probes into NVIDIA GPU drivers for live resource visibility
- Dynamic Kubernetes tainting and 500 ms reconciliation loop for real‑time GPU allocation and fractional sharing
- Native integration with Ray to coordinate distributed compute across training and inference jobs
- Compatibility with PyTorch and vLLM, supporting model quantization, KV‑cache optimization, and OpenAI‑compatible APIs
- Cloud‑agnostic deployment via Terraform, Helm, and ArgoCD on GKE, EKS, AKS, and OpenShift clusters
- Support for multi‑tenant environments, enabling secure isolation of GPU resources among different teams or workloads