VideoRAG

// Overview

VideoRAG is a retrieval-augmented generation framework from HKUDS built for one hard case: video collections far too long to stuff into a context window. Rather than chunking a transcript, it uses a dual-channel architecture — graph-driven textual knowledge grounding that models semantic relationships across videos, combined with hierarchical multimodal context encoding that preserves spatiotemporal visual patterns. Hundreds of hours are distilled into a structured knowledge graph, and retrieval aligns a text query against both the textual and the visual channel before an answer is generated.

The practical claim is efficiency: the authors report comprehending hundreds of hours of video on a single NVIDIA RTX 3090 with 24 GB. To measure it they built LongerVideos, a benchmark of 164 videos across 22 collections and roughly 134.6 hours — lectures, documentaries and entertainment — with 602 queries. On the Video-MME long-video track the README reports 60.2% accuracy for VideoRAG against 52.2% and 56.3% for MiniCPM-o without and with subtitles, and 51.8% and 56.3% for MiniCPM-V. The work is published as VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos (arXiv 2502.01549).

The same repository ships Vimo, an Electron desktop app that puts a chat interface over the framework: drag videos in, ask questions in natural language, analyse several videos at once, and export the references. Vimo is in beta and currently runs from source — a Python backend serving VideoRAG plus the Electron frontend — with packaged macOS builds announced as coming first. Licensing is dual: the framework architecture, interfaces and documentation are MIT, while the current implementation carries the licences of the third-party models and libraries it pulls in (MiniCPM-V, faster-distil-whisper, ImageBind and others), so check those before commercial use.

// What it does

• Dual-channel indexing — a multimodal knowledge graph for cross-video semantics plus hierarchical encoding that keeps spatiotemporal visual detail
• Designed for unbounded video length: hundreds of hours indexed and queried on one 24 GB RTX 3090
• Multimodal retrieval that aligns a textual query against visual and audio content, not just a transcript
• Cross-video understanding, so a question can be answered from a whole collection rather than one file
• The LongerVideos benchmark — 164 videos, 22 collections, ~134.6 hours and 602 queries across lectures, documentaries and entertainment
• Ollama support for running the language-model side locally
• Vimo, an Electron desktop front end with drag-and-drop upload, natural-language chat, multi-video analysis and reference export

// Getting started

VideoRAG is a Python framework with pinned dependencies and three model checkpoints to download. Set the environment up first; Vimo then runs on top of it.

conda create --name videorag python=3.11
conda activate videorag

pip install numpy==1.26.4
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2
pip install accelerate==0.30.1 bitsandbytes==0.43.1

pip install moviepy==1.0.3
pip install git+https://github.com/facebookresearch/pytorchvideo.git@28fe037d212663c6a24f373b94cc5d478c8c1a1d
pip install --no-deps git+https://github.com/facebookresearch/ImageBind.git@3fcf5c9039de97f6ff5528ee4a9dce903c5979b3

pip install timm ftfy regex einops fvcore eva-decord==0.6.1 iopath matplotlib types-regex cartopy
pip install ctranslate2==4.4.0 faster_whisper==1.0.3 neo4j hnswlib xxhash nano-vectordb
pip install transformers==4.37.1
pip install tiktoken openai tenacity
pip install ollama==0.5.3

git lfs install

git lfs clone https://huggingface.co/openbmb/MiniCPM-V-2_6-int4
git lfs clone https://huggingface.co/Systran/faster-distil-whisper-large-v3

mkdir .checkpoints
cd .checkpoints
wget https://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pth
cd ../

conda create --name vimo python=3.11
conda activate vimo
# ...core VideoRAG dependencies as above...
pip install flask psutil flask_cors setproctitle

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Ask questions across a lecture series or conference archive that no context window could hold
• Search a documentary or long-form video library for the moment a specific thing was said or shown
• Compare several long videos in one query, rather than summarising each and reconciling by hand
• Use it as a research baseline for long-context video understanding, with the LongerVideos benchmark and reproduce scripts included

// How VideoRAG compares

VideoRAG alongside other open-source rag frameworks & platforms tools AI/TLDR tracks, ranked by GitHub stars.