Hugging Face Datasets

// Overview

Datasets is the Hugging Face library for getting data into and out of a machine-learning pipeline. It does two things. The first is one-line loading: `load_dataset("rajpurkar/squad")` pulls any dataset from the Hugging Face Hub — text in hundreds of languages, images, audio, video, PDFs, 3D medical scans, agent traces — ready to hand to NumPy, Pandas, Polars, PyTorch, TensorFlow, JAX or Spark. The same function reads local CSV, JSON, JSONL, Parquet, Arrow, HDF5, XML, text, image, audio and PDF files, and `Dataset.from_dict`, `from_list`, `from_pandas` and `from_generator` build one from Python objects.

The second is preprocessing that does not fall over on size. Storage is Apache Arrow, memory-mapped and zero-copy, so a dataset larger than RAM is still ordinary to work with, and `map()` transformations are cached so the same processing never runs twice. `map(num_proc=N)` parallelises across processes, and `streaming=True` iterates a dataset without downloading it at all — useful when the data is bigger than the disk.

Around that core sit the things a training or evaluation pipeline tends to need: multimodal feature types including audio, image, video, PDF and NIfTI, a flexible `Json()` feature type, reading and writing Hugging Face Storage Buckets for large mutable raw data, and built-in FAISS and Elasticsearch indexes for similarity search over a dataset. Optional extras (`datasets[audio]`, `[vision]`, `[pdfs,nibabel]`, `[torch,tensorflow,jax]`) pull in only the decoders and framework bridges you actually use. It is Apache-2.0 licensed.

// What it does

• `load_dataset()` for Hub datasets, local files in a dozen formats, or a whole directory with format auto-detection
• Apache Arrow backend — memory-mapped, zero-copy, so datasets are not bounded by RAM
• Streaming mode that iterates a dataset without downloading it
• Cached, multi-process `map()` / `filter()` preprocessing
• Multimodal feature types: text, audio, image, video, PDF and NIfTI 3D medical data
• Zero-copy conversion to NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX and Spark, plus FAISS and Elasticsearch indexes

// Getting started

Install into a virtual environment, load something from the Hub, then map your preprocessing over it.

pip install datasets

# optional extras
pip install datasets[audio]
pip install datasets[vision]
pip install datasets[torch,tensorflow,jax]

from datasets import load_dataset

squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)

image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
    print(example["image"])
    break

dataset = load_dataset('csv', data_files='my_data.csv')
dataset = load_dataset('parquet', data_files='data/*.parquet')

from datasets import Dataset
dataset = Dataset.from_dict({"text": ["Hello world", "How are you?"]})

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Pull a public benchmark or training corpus into a fine-tuning run without writing a loader
• Preprocess a corpus larger than memory — tokenize, filter, chunk — with caching and multiprocessing
• Stream a huge image or audio dataset for training when the disk cannot hold it
• Turn your own CSV, Parquet or JSONL into a Dataset and publish it to the Hub for reuse

// How Hugging Face Datasets compares

Hugging Face Datasets alongside other open-source data wrangling tools AI/TLDR tracks, ranked by GitHub stars.