Data Mechanics provides a serverless platform for data engineering and data science, utilizing dynamic scaling and automatic configuration tuning for Apache Spark applications on Kubernetes. This approach achieves 50-75% cost reductions for customers by optimizing cloud infrastructure and resource allocation based on historical data.
Funding
$150K raised to dateRaised to date based on public sources. This may differ from the amount the company actually raised and is based only on what is publicly available on the internet.
AERGLFounders
Product
Problem
Managing Apache Spark workloads on Kubernetes can be complex and costly due to the intricacies of resource allocation and configuration. Data engineers often struggle with optimizing infrastructure and application settings, leading to inefficient resource utilization and increased cloud expenses.
Solution
Data Mechanics provides a managed Apache Spark platform on Kubernetes that simplifies data engineering workflows and reduces cloud infrastructure costs. The platform automates the deployment, scaling, and configuration of Spark applications, dynamically adjusting resources based on workload demands. By leveraging historical data and machine learning, Data Mechanics automatically tunes Spark configurations, container memory/CPU allocations, and instance types, optimizing resource allocation for each pipeline run. This approach enables data teams to focus on development and analysis while achieving significant cost savings compared to unoptimized or manually managed Spark deployments.
Target Audience
The primary target audience includes data engineers, data scientists, and data platform teams who are running Apache Spark workloads and want to simplify management, optimize performance, and reduce cloud infrastructure costs.
Features
- Automated deployment and management of Apache Spark on Kubernetes clusters
- Dynamic scaling of applications and Kubernetes nodes based on real-time load
- Automatic tuning of Spark configurations, instance types, and container resource allocation
- Integration with Jupyter notebooks, REST API, and Airflow for flexible application submission
- Support for Docker images, allowing users to package dependencies and customize environments
- Monitoring dashboard for tracking application logs, metrics, and costs over time
- Node pool management with support for spot instances to further reduce costs
- Integration with cloud provider IAM roles, enabling secure access to cloud resources