Daft is a multimodal‑native data engine that lets ML teams treat video, images, audio, sensor data, and structured metadata as columns in a single Python dataframe, using familiar Pandas‑ or Spark‑style operations.
Funding
Funding not disclosed
Founders
Product
Problem
Machine learning teams often need to process large volumes of raw video, image, audio, sensor, and structured data together, but existing tools require separate pipelines, extensive glue code, and consume excessive memory, making scaling from a laptop to a cloud cluster difficult.
Solution
Daft offers a multimodal‑native data engine that lets users treat all data types as columns in a single Python dataframe, using familiar Pandas or Spark‑style operations. The system automatically schedules CPU decoding and GPU inference in one pipeline, handling batching, retries, and error handling without custom code. Built in Rust, Daft executes at low overhead and can run on a laptop or scale across distributed clusters while maintaining the same pipeline definition. By leveraging Apache Arrow for zero‑copy data movement, Daft reduces memory usage by up to five times compared with traditional frameworks, enabling workloads that would otherwise exceed memory limits. The engine also provides native column types for embeddings and model operators, simplifying AI search, data enrichment, and multimodal ETL tasks.
Target Audience
Primary users are data engineers, machine‑learning engineers, and AI researchers who need to build scalable, multimodal data pipelines for training, inference, and analytics.
Features
- Multimodal column support for images, video, audio, text, and embeddings, all manipulable with standard dataframe operations
- Integrated CPU decoding and GPU inference scheduling with automatic batching and error handling for model UDFs
- Rust‑based core with Arrow zero‑copy execution for high‑performance processing at TB scale
- Consistent pipeline code that runs unchanged from a local notebook to a distributed cloud cluster
- Memory‑efficient execution that uses up to 5× less RAM than comparable Pandas or Spark workflows
- Built‑in observability with per‑operator memory attribution and OpenTelemetry endpoints