MediaPipe

// Overview

MediaPipe is Google's open-source framework for running machine learning on device. It packages common perception problems — object detection, image and gesture recognition, pose and hand tracking, text classification, audio classification — as ready-to-run tasks with a consistent cross-platform API, so you call a solution rather than assembling a model, a preprocessor and a postprocessor yourself.

The project has two layers. MediaPipe Solutions is the high-level layer: MediaPipe Tasks provides the cross-platform APIs and libraries for deployment, and MediaPipe models provides pre-trained checkpoints for each task. Underneath sits MediaPipe Framework, a lower-level graph engine built from packets, graphs and calculators, which is what you drop to when you need a custom on-device pipeline in C++, Android or iOS.

Two supporting tools round it out: MediaPipe Model Maker retrains the bundled models on your own data, and MediaPipe Studio lets you visualise, evaluate and benchmark solutions in the browser. Because MediaPipe Tasks processes input on device, image, video and text data never leaves the machine — though the APIs do report usage and performance metrics back to Google.

// What it does

• Ready-to-run tasks across three domains: vision (detection, segmentation, tracking), text (classification, embedding) and audio (classification)
• One task API deployed across Android, iOS, web, desktop, edge devices and IoT
• Pre-trained models bundled per solution, with MediaPipe Model Maker to retrain them on your own data
• MediaPipe Framework underneath — a packets/graphs/calculators engine for building custom on-device pipelines in C++
• MediaPipe Studio for visualising, evaluating and benchmarking solutions in the browser
• On-device processing: input data is not sent to Google servers, so it suits data that must not leave the device

// Getting started

The Python path is the quickest way in. Google's developer site is the primary documentation, with separate setup guides for Android, web apps and Python, and per-task developer guides for vision, text and audio.

python -m pip install mediapipe

import mediapipe as mp
from mediapipe.tasks.python import vision

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Add hand, face or pose tracking to a mobile or web app without training a model or standing up an inference server
• Run object detection or segmentation on device where latency, offline operation or data residency rules out a cloud API
• Retrain a bundled detector or classifier on your own images with Model Maker, then deploy the same task to Android, iOS and web
• Build a custom on-device perception pipeline in C++ using MediaPipe Framework's calculator graphs

// How MediaPipe compares

MediaPipe alongside other open-source vision & understanding tools AI/TLDR tracks, ranked by GitHub stars.