Lance

// Overview

Lance is an open lakehouse format for multimodal AI. It is not a database you run but a file format, table format and catalog spec you build on top of object storage, so images, video, audio, text and embeddings all live in one dataset that your training and search code reads directly.

Its defining trade-off is random access. Columnar formats like Parquet are built for full scans, which makes fetching individual rows — exactly what shuffled ML training and point lookups do — expensive. The project reports Lance is 100x faster than Parquet for random access without giving up scan performance, which is what lets the same copy of the data serve both analytics and training.

Around that core sit the features a retrieval stack usually bolts on separately: accelerated secondary indices for vector similarity and BM25 full-text search alongside SQL analytics, blob encoding with lazy loading for large media, data evolution that adds backfilled columns without rewriting the table, and automatic zero-copy versioning with ACID transactions, time travel, tags and branches. It reads through Apache Arrow, Pandas, Polars, DuckDB, Spark, Ray and Trino, and is the storage layer beneath LanceDB.

// What it does

• Hybrid search on one dataset: vector similarity, BM25 full-text and SQL analytics backed by accelerated secondary indices
• Random access the project benchmarks at 100x faster than Parquet or Iceberg, without sacrificing scan throughput
• Native multimodal storage for images, video, audio, text and embeddings with blob encoding and lazy loading
• Data evolution: add columns with backfilled values without a full table rewrite
• Zero-copy versioning with ACID transactions, time travel, tags and branches — no extra infrastructure
• Reads from Apache Arrow, Pandas, Polars, DuckDB, Spark, Ray, Trino and open catalogs such as Apache Polaris and Unity Catalog

// Getting started

Lance ships as the `pylance` Python package. Converting an existing Parquet dataset is the fastest way to see the format, and the result is readable as a normal PyArrow dataset.

pip install pylance

import lance

import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")

dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)

df = dataset.to_table().to_pandas()

import duckdb

duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Store a training corpus of images or video alongside its embeddings so shuffled random-access reads stay fast
• Build a search engine or feature store that needs vector similarity, full-text and SQL over one copy of the data
• Add engineered feature columns to an existing table without rewriting terabytes of data
• Use time travel and branches to reproduce exactly which rows a model was trained on

// How Lance compares

Lance alongside other open-source data wrangling tools AI/TLDR tracks, ranked by GitHub stars.