OLake is an open-source, Iceberg-first EL engine that moves data from databases like MySQL, Postgres, MongoDB, and Kafka into Apache Iceberg or Parquet at high speed. It combines a Go-based ingestion engine with a maintenance engine for automated compaction, delivering up to 500× faster ingest than common alternatives while remaining self-hostable on any infrastructure.
- Artificial Intelligence
- Software Only
Funding
Funding not disclosed
Founders
Product
Problem
Traditional data ingestion pipelines are often brittle, slow, and require extensive manual scripting to move data from operational databases into data lakes. Teams struggle with fragile connectors, complex CDC setups, and performance bottlenecks that slow down analytics and require constant maintenance.
Solution
OLake provides a fast, open-source EL engine purpose-built for Apache Iceberg, simplifying the movement of data from sources like MySQL, Postgres, MongoDB, Oracle, Kafka, and S3 into modern lakehouse formats. The platform uses a Go-based ingestion engine with parallelized chunking, incremental sync, and resilient CDC to replicate data at scale, while a separate maintenance engine, OLake Fusion, handles scheduled compaction and cleanup to keep Iceberg tables fast as data grows. OLake writes directly to open standards like Iceberg and Parquet, supports multiple catalogs including AWS Glue, Hive Metastore, and REST catalogs, and can be fully self-hosted on any cloud or on-premises infrastructure. Recent benchmarks show up to 500× faster ingest than common alternatives, with a clean UI/CLI that reduces operational overhead.
Target Audience
Primary users are data engineers and analytics teams building modern lakehouse architectures who need reliable, high-performance data replication from operational databases into Apache Iceberg without vendor lock-in.
Features
- Go-based ingestion engine supporting full load, CDC, and incremental sync from MySQL, Postgres, MongoDB, Oracle, Kafka, S3, DB2 LUW, and MSSQL
- OLake Fusion maintenance engine for scheduled compaction and delete-file cleanup to sustain query performance and control storage costs
- Direct writes to Apache Iceberg or Parquet with support for AWS Glue, Hive Metastore, JDBC/SQL, and REST catalogs like Nessie, Polaris, and Unity
- Parallelized chunking and snapshot-based architecture enabling ACID transactions, schema evolution, and time travel for reliable concurrent reads and writes
- Self-hosted deployment on customer infrastructure, keeping full control over regulated data
- Clean UI and CLI for pipeline management, with observable metrics and logs for monitoring