Mirai provides an on-device inference layer that allows AI model makers to deploy and run models of any architecture directly on user devices. This SDK enables local execution for near-zero latency and enhanced data privacy, freeing up cloud GPUs for training and scaling. Developers can use a unified API to intelligently route inference requests between the device and the cloud based on custom rules.
Funding
Funding not disclosed
Founders
Product
Problem
Developers of AI-powered applications often rely on cloud-based inference, which introduces latency, incurs GPU costs, and raises data‑privacy concerns. Mobile and desktop devices now have powerful GPUs and Neural Engine units, but there is no unified layer that lets any model architecture run efficiently on‑device without extensive engineering effort.
Solution
Mirai delivers an on‑device inference layer for iOS and macOS that lets developers deploy and run models of any architecture directly on user devices. The Apple Inference SDK taps the GPU and ANE to provide the fastest native inference, while the Smart Routing engine dynamically decides whether a request should be handled locally or sent to the cloud based on latency, privacy, or cost policies. Mirai’s tooling automatically converts, quantizes, and optimizes models (via the lalamo toolkit) and offers pre‑built bindings (uzu) for Swift and TypeScript, enabling a drop‑in integration that can be set up in minutes. Local inference yields near‑zero response time and full data confidentiality, freeing cloud GPUs for large‑scale training or heavy contextual tasks. Developers control access with granular permissions and can mirror existing pricing models (tokens, licenses, rev‑share).
Target Audience
The primary customers are iOS and macOS application developers and product teams building AI‑enhanced consumer or enterprise apps that require low latency, on‑device privacy, and cost‑effective scaling.
Features
- Apple Inference SDK: hardware‑aware kernels that exploit GPU and Apple Neural Engine for up to 3× faster token generation compared to existing runtimes.
- Smart Routing engine: programmable policies to route inference to device or cloud in real time based on prompt length, confidence thresholds, device capability, or user segment.
- Model conversion & quantization pipeline (lalamo) that automatically prepares SOTA LLMs, vision, and audio models for on‑device execution.
- uzu inference engine with built‑in speculative decoding, delivering up to 3× speed‑up without external libraries.
- Drop‑in SDK with API key management, supporting Swift, Objective‑C, and TypeScript bindings; includes automatic model download and versioning.
- Support for a broad catalog of open‑source models (Gemma, Llama, Qwen, DeepSeek, etc.) and custom user models.
- Structured JSON output and schema‑aligned results for reliable downstream processing.
- Free tier covering up to 10 000 devices; paid usage beyond the free quota with volume‑based pricing.