NeuroVFM

// Overview

NeuroVFM is a visual foundation model for neuroimaging developed by the MLiNS Lab at the University of Michigan. It is trained on 5.24 million clinical MRI and CT volumes from 566,915 studies at Michigan Medicine using a self-supervised approach called Vol-JEPA — no radiology report labels or manual curation are required.

The model uses a 3D ViT-Base encoder that learns brain anatomy and pathology by predicting masked volumetric patches from context, trained in under 1,000 GPU hours on 8 NVIDIA L40S GPUs. Diagnostic heads covering 156 tasks (74 MRI and 82 CT) and a lightweight LLaVA-based findings LLM are then fine-tuned on top for report generation and clinical triage.

In a prospective one-week trial at Michigan Medicine involving 1,155 studies in January 2026, NeuroVFM achieved 92.6% balanced triage accuracy versus 71.2% for GPT-5 — a 21.4 percentage-point gap — while generating radiology reports 24× cheaper and with 23× lower carbon footprint. Code is released under MIT; pretrained weights are on Hugging Face under CC-BY-NC-SA 4.0 for non-commercial research use.

// What it does

• Self-supervised pretraining via Vol-JEPA on 5.24M uncurated clinical MRI and CT volumes
• 3D ViT-Base encoder that predicts masked volumetric patches from context
• 156 diagnostic task heads spanning 74 MRI and 82 CT tasks
• Lightweight LLaVA-based findings LLM for radiology report generation
• Pretrained weights on Hugging Face (CC-BY-NC-SA 4.0, non-commercial research use)
• Trained in under 1,000 GPU hours — far less compute than comparable models

// Getting started

Clone the repository, create a Python environment, install dependencies, then download the pretrained encoder weights from Hugging Face.

git clone https://github.com/MLNeurosurg/neurovfm
cd neurovfm
pip install -e .

from huggingface_hub import snapshot_download
snapshot_download(repo_id="mlinslab/neurovfm-encoder", local_dir="./weights")

import torch
from neurovfm import load_encoder

encoder = load_encoder("./weights")
volume = torch.randn(1, 1, 96, 96, 96)  # B C D H W
embeddings = encoder(volume)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Extract volumetric embeddings from MRI and CT scans for downstream classification or regression tasks
• Fine-tune the encoder for institution-specific neuroimaging diagnostics across 74 MRI or 82 CT tasks
• Generate structured radiology reports from brain scans using the bundled LLaVA-based findings LLM
• Benchmark self-supervised 3D medical imaging models against the Vol-JEPA pretraining recipe

// How NeuroVFM compares

NeuroVFM alongside other open-source medical imaging tools AI/TLDR tracks, ranked by GitHub stars.