xorq Labs offers an open-source computational framework that simplifies the development and deployment of machine learning pipelines for Python developers. Built on Ibis and Apache DataFusion, xorq enables the creation of multi-engine ML pipelines with intelligent data caching and seamless deployment across various platforms. This allows developers to build, run, and serve ML pipelines more efficiently, from local environments to production.
Funding
Funding not disclosed
Founders
Product
Problem
Developing and deploying machine learning pipelines often requires juggling multiple query engines, manual data caching, and platform-specific code, leading to inefficiencies and increased complexity for Python developers. This fragmented approach slows down iteration and hinders the seamless transition from local development to production environments.
Solution
xorq Labs offers an open-source computational framework designed to streamline the development and deployment of multi-engine ML pipelines for Python developers. Built on Ibis and Apache DataFusion, xorq enables the creation of portable, multi-engine pipelines with intelligent data caching and seamless deployment across various platforms. The framework simplifies ML workflows by providing a unified interface for orchestrating data flows, managing data movement between engines, and building portable user-defined functions (UDxFs). xorq's built-in performance optimizations, such as in-memory data transfer and automated cache invalidation, ensure fast execution and rapid iteration during development and testing.
Target Audience
The primary users are Python developers and data scientists building ML pipelines who require the flexibility to work with multiple data engines, accelerate iteration through intelligent caching, and seamlessly deploy pipelines across different environments.
Features
- Declarative Pythonic scripting language for orchestrating multi-engine data flows
- Seamless data movement between query engines, including Snowflake, Trino, Pandas, DuckDB, and Postgres
- Built-in caching and in-memory data transfer for accelerated iteration
- Support for scalar UDFs, UDAF aggregation, and UDWF windowing
- Ability to compose end-to-end ML pipelines into single, executable expressions
- Command-line interface (CLI) for building and running serialized expressions
- Integration with Apache Arrow and Arrow Flight for data microservices
- YAML-based pipeline serialization for version control and deployment