LETSQL is a data processing library that enables the creation of declarative and efficient multi-engine pipelines in Python, utilizing a portable, multi-engine runtime with first-class support for user-defined functions (UDFs). It addresses the challenge of integrating diverse data processing engines by providing a composable framework that enhances performance and portability across platforms.
Funding
Funding not disclosed
Founders
Product
Problem
Integrating diverse data processing engines in Python often requires complex and inefficient pipelines. Existing solutions lack composability, portability, and performance, hindering the ability to seamlessly connect to various data sources and apply machine learning models.
Solution
LETSQL is a data processing library designed for building declarative and efficient multi-engine pipelines in Python. It provides a composable framework that enhances performance and portability across platforms by leveraging a multi-engine runtime and user-defined functions (UDFs). With LETSQL, data practitioners can connect to SQL sources and apply machine learning models without switching between SQL and Python or rewriting code for performance optimization. The library includes built-in UDFs for XGBoost model scoring, enabling users to score models directly from a DataFrame API.
Target Audience
LETSQL is designed for data scientists, data engineers, and machine learning practitioners who need to build efficient data processing pipelines that integrate diverse data sources and machine learning models.
Features
- Multi-engine support for connecting to various data sources from a pandas-like DataFrame API
- Built-in UDF for XGBoost model scoring, eliminating the need for code translation libraries
- Rust-based execution engine (via DataFusion) for optimized performance
- SQL transpilation (via Ibis) for seamless integration with SQL databases
- Feature selector for intelligently filtering features required for the scoring pipeline
- Ability to push down pre-processing tasks to the SQL source, reducing data transfer overhead