NannyML is an open-source Python library that estimates the performance of machine learning models in production without requiring access to target data, utilizing techniques like Confidence-Based Performance Estimation and Direct Loss Estimation. It addresses the issue of model degradation by detecting data drift and linking performance changes to specific features, enabling data scientists to maintain model accuracy and business value effectively.
Funding
$2.3M raised to dateRaised to date based on public sources. This may differ from the amount the company actually raised and is based only on what is publicly available on the internet.
Founders
Product
Problem
Machine learning models deployed in production often degrade over time due to data drift and concept drift, leading to inaccurate predictions and reduced business value. Traditional model monitoring relies on data drift detection, which can trigger numerous false alarms and overwhelm data science teams. Furthermore, many organizations lack access to target data in production, making it difficult to assess model performance accurately.
Solution
NannyML is an open-source Python library designed to estimate the performance of machine learning models in production without requiring access to target data. It utilizes techniques like Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE) to provide insights into model health. NannyML detects data drift and intelligently links performance changes to specific features, enabling data scientists to identify the root causes of model degradation. By focusing on performance-impacting issues, NannyML helps data science teams maintain model accuracy, avoid alert fatigue, and maximize the business value of their deployed models. NannyML Cloud solves infrastructure nuances and brings extra capabilities to the table.
Target Audience
NannyML is designed for data scientists and machine learning engineers who need to monitor and maintain the performance of their deployed models in production environments.
Features
- Performance estimation for classification and regression models without requiring target data
- Multivariate and univariate data drift detection using PCA-based data reconstruction and statistical tests (Jensen-Shannon Distance, L-Infinity Distance)
- Intelligent alerting system that links data drift to performance changes, reducing false alarms
- Alert ranking to prioritize issues based on impact on model performance
- Interactive visualizations for analyzing data drift and model performance over time
- Model-agnostic design compatible with various machine learning frameworks
- Open-source and extensible architecture, allowing for customization and integration with existing MLOps pipelines
- Effortless observability, start monitoring in minutes
- Concept drift detection to measure the impact of concept drift on model's performance
- Automated monitoring data collection with the SDK