What Is Milvus? The Vector Database Built for Billion-Scale Search

// In plain English

Milvus is an open-source vector database designed to store and search huge numbers of embeddings — the lists of numbers that AI models produce to represent the meaning of text, images, or audio. You hand Milvus millions or even billions of these vectors, and when a query vector arrives it returns the closest matches in milliseconds. Its claim to fame is scale: Milvus was built from day one to run as a distributed system across many machines, so it keeps working when your data outgrows a single server.

An analogy: think of a national postal sorting network rather than a single mailroom. A small mailroom (a lightweight vector store) is perfect when you have a few thousand letters — one person can find any envelope fast. But once you handle billions of letters a day, you can't pile them in one room. You split the work across regional hubs, each holding part of the mail, with separate teams for receiving, indexing, and looking things up. Milvus is that sorting network: storage, indexing, and search are separate moving parts that you can scale independently as the volume grows.

Milvus started in 2019 at the company Zilliz and is now a graduated project of the LINUX Foundation's LF AI & Data. It lives on GitHub at github.com/milvus-io/milvus and is one of the most widely adopted open-source vector databases. You can run it three ways: Milvus Lite (a tiny embedded version you pip install for prototyping), Milvus Standalone (a single-node Docker deployment), and Milvus Distributed (the full cluster for production-scale workloads).

// Why it matters for builders

Most AI applications that use embeddings hit the same wall eventually: the data gets too big for one machine. A RAG system over a company's entire document history, a product-recommendation engine across a giant catalog, image search over a stock-photo library, or long-term agent memory that never forgets — all of these can grow into hundreds of millions of vectors. At that size, the question stops being "which database is easiest" and becomes "which one won't fall over."

Milvus matters because it answers that scaling question directly. Its design separates storage from compute, which is the architectural choice that makes billion-scale search practical:

• Independent scaling. If search traffic spikes but your data is stable, you add query nodes without touching storage. If data grows but queries are light, you add storage cheaply. You pay for what each part actually needs.
• Durability through object storage. Vectors and indexes ultimately live in cheap, durable object storage (such as S3-compatible buckets). A compute node can crash and restart without losing data, because the data was never only in that node's memory.
• Multiple index types in one system. Different workloads want different speed-versus-accuracy tradeoffs. Milvus lets you pick from several index families (HNSW, IVF variants, DiskANN, and quantized indexes) per collection, instead of locking you into one algorithm.
• Rich filtered search. Like other modern stores, Milvus stores structured fields ('scalar fields') next to each vector, so you can combine semantic similarity with hard filters — for example, search by meaning but only within a date range or a category.

The honest flip side, which the rest of this article keeps coming back to: that same distributed design is overkill for small projects. Below a certain scale, the operational weight of running a cluster buys you nothing, and a lighter store like Chroma or pgvector will get you to a working product faster. Milvus earns its keep when scale is the actual problem you have.

// How Milvus works

To understand Milvus you need two pictures: the data model (how you organize vectors) and the architecture (the components that do the work). Start with the data model, because it's the part you touch as a developer.

The data model: collections, fields, and partitions

A collection is the top-level container, roughly the equivalent of a table. Each collection has a schema that defines its fields. Every collection has at least one vector field (the embeddings) plus a primary-key field, and you can add scalar fields for metadata such as category, price, or created_at. A single row — one vector plus its metadata — is called an entity. Within a collection you can create partitions to physically group related entities (for example, one partition per tenant or per month), so a search can skip the partitions it doesn't need and run faster.

The architecture: separated storage and compute

Under the hood, Milvus Distributed is not one process — it is several specialized roles, grouped into four layers. This is the design that makes independent scaling possible: each layer can grow on its own.

• Access layer (proxy). A stateless front door. Clients connect here; it validates requests and routes them to the right workers. You add proxies to handle more concurrent connections.
• Coordinator layer. The brains. It assigns tasks, tracks where data lives, balances load across workers, and manages the collection metadata.
• Worker layer. Where the actual work happens. Query nodes load index segments and run searches; data nodes handle incoming writes and build indexes. These are the parts you scale out when traffic or ingest grows.
• Storage layer. The durable foundation: a metadata store, a streaming message log that orders all writes, and object storage holding the vectors and indexes themselves.

The key insight is that worker nodes are mostly stateless caches. Because the real data lives in the storage layer, a query node can be added, removed, or restarted freely — it just loads the segments it needs from storage. That decoupling is exactly what lets Milvus grow to billions of vectors without a single machine becoming the bottleneck.

// A first Milvus example

The fastest way to feel how Milvus works is Milvus Lite, which needs no server at all. The snippet below creates a collection, inserts a few vectors with metadata, and runs a filtered similarity search. The exact same code runs against a full cluster — you only change the connection string.

from pymilvus import MilvusClient

# Milvus Lite: a local file, no server to run.
client = MilvusClient("demo.db")

# 1) Create a collection. dimension must match your embedding model.
client.create_collection(collection_name="docs", dimension=384)

# 2) Insert entities: each has an id, a vector, and scalar metadata.
client.insert(
    collection_name="docs",
    data=[
        {"id": 1, "vector": embed("refunds within 30 days"), "topic": "policy"},
        {"id": 2, "vector": embed("support is open 9 to 6"),  "topic": "hours"},
        {"id": 3, "vector": embed("digital goods are final sale"), "topic": "policy"},
    ],
)

# 3) Search by meaning, filtered to a metadata field.
results = client.search(
    collection_name="docs",
    data=[embed("how long to return an item?")],
    filter="topic == 'policy'",   # combine similarity + hard filter
    limit=2,
    output_fields=["topic"],
)
print(results)

Notice the shape of the workflow: define a collection with a fixed vector dimension, insert entities (vector + metadata together), then search with an optional filter string. To move this to production you would switch MilvusClient("demo.db") to a server URL and explicitly create an index on the vector field — but the mental model stays identical.

// Milvus vs other vector stores

Milvus is one of several good open-source options, and choosing well means matching the tool to your scale and team. Here is how it lines up against the usual alternatives — see the full vector database comparison for the bigger picture.

A simple rule of thumb: if your dataset comfortably fits on one node and you value getting started quickly, a lighter store wins. If you genuinely expect hundreds of millions of vectors, high write throughput, or independent scaling of search and storage, Milvus is purpose-built for that world. The wrong reason to choose Milvus is fear of future scale you don't actually have — premature distribution is a real cost.

// Choosing an index in Milvus

Searching every vector one by one (a 'brute-force' or FLAT scan) gives perfect accuracy but is far too slow at scale. So Milvus builds an approximate nearest neighbor (ANN) index on the vector field — a structure that finds almost the closest vectors much faster. The trick is that there is no single best index; each trades speed, accuracy, and memory differently. Milvus supports several families so you can match the index to the job.

// Going deeper

Once the basics click, a few deeper topics shape how Milvus behaves in production.

Consistency and the streaming log. Every write in Milvus first goes to a streaming message log before it is sealed into a searchable segment. This is why a vector you just inserted may not appear in search results for a brief moment. Milvus exposes tunable consistency levels (from strong to eventually consistent) so you can trade a little freshness for a lot of throughput — a deliberate choice that distributed systems have to make.

Growing vs sealed segments. New data lands in small 'growing' segments held in memory; in the background Milvus compacts and seals them into larger immutable segments and builds the index on those. Understanding this explains real behavior: ingest is fast, but freshly written data is searched slightly differently from sealed data until compaction catches up.

Load before you search. In the distributed model, a collection's segments must be loaded into query nodes' memory before they can be searched. Loading a giant collection takes time and RAM, which is one reason partitions matter — you can load only the partitions you need. This is an operational detail that lighter single-node stores hide from you, and a reminder that Milvus's power comes with knobs to learn.

Managed Milvus (Zilliz Cloud). If you want Milvus's scale without operating the cluster yourself, the original creators offer a managed service. That blurs the line with options like Pinecone: you get the Milvus API and data model, but someone else runs the proxies, coordinators, and storage.

The durable lesson is the same one this article opened with: Milvus is an answer to a scale problem. Its separated-compute design, multiple index types, and cluster components are exactly what billion-scale search needs — and exactly what a small project does not. Pick it when your data is genuinely large or growing fast; pick a lighter store, and revisit Milvus later, when it isn't. To decide deliberately, weigh it against the field in the vector database comparison and make sure your embedding model and index choice fit the same scale.