In plain English
Vector search feels almost free in a demo: a few thousand documents, one laptop, instant results. Then you ship it. Now you have ten million documents, traffic around the clock, and a cloud bill that grows every month for three reasons most teams never separate: making the vectors, keeping the index searchable, and storing everything. Cost optimization is just naming those three buckets and pulling the right lever on each.
Think of it like running a library. First you pay a typist to write a summary card for every book — that's the embedding step, billed per document. Then you keep the most-used card catalog out on fast desks so any librarian can grab a card in a second — that's the index in RAM, the priciest part because memory is expensive. Finally, you keep the original books in a basement warehouse — cheap cold storage. You can cut the bill at any of the three without closing the library; you just have to know which one is actually draining your wallet.
Why it matters
Vector search costs scale with your data and your traffic, and both tend to grow faster than anyone plans for. A pilot that costs a few dollars a month can quietly become a five-figure line item once it holds your whole corpus — and the jump is rarely linear, because the expensive part (the in-memory index) grows with both the number of vectors and their size.
The trap is treating the bill as one number. "Our vector search is too expensive" is not actionable. "Our embedding API spend is 70% of the bill because we re-embed unchanged documents on every deploy" is a fix you can ship this afternoon. Splitting the cost into buckets is the entire game — most teams find one bucket dwarfs the other two, and that bucket is where all your effort should go.
Who should care? Anyone running semantic search or RAG past the prototype stage. The good news: the biggest savings usually cost you almost nothing in quality. Cutting a million unnecessary embedding calls loses zero recall. Quantizing an index might cost a percent or two of recall for a 4x memory cut — a trade most products take happily. The skill is knowing which levers are nearly free and which actually hurt.
How it works: the three cost buckets
Every dollar a vector search system spends falls into one of three buckets. Knowing the shape of each tells you which lever moves your bill.
Bucket 1 — embedding generation
Every document and every query must be turned into a vector before it can be searched. With a hosted embedding API you pay per token; self-hosting an embedding model trades that for GPU time. This is a one-time cost per document plus a small recurring cost per query — which means the danger is re-embedding documents you've already embedded. Re-run your whole corpus on every deploy and you pay the one-time cost over and over.
Bucket 2 — index compute and memory
To search millions of vectors in milliseconds you don't compare the query to every vector — you build an approximate nearest neighbor index (HNSW is the common one). For speed, that index usually lives in RAM, and RAM is the most expensive resource in your stack. This bucket is almost always the largest, and it grows with two things: how many vectors you have, and how big each vector is. Both are levers.
Bucket 3 — storage and egress
The raw vectors, the original text, metadata, and backups all sit on disk. Disk is cheap — often the smallest bucket — but two things bite: cross-region or cross-cloud egress fees when you move data around, and keeping full-precision copies of vectors you also hold compressed in the index. Storage rarely dominates, but it's the easiest to forget until a transfer fee surprises you.
A back-of-envelope cost model
Before optimizing, estimate. The index memory bucket follows a simple formula you can run in your head. A raw vector of D dimensions stored as 32-bit floats takes D × 4 bytes; an HNSW index adds graph links, so a common rule of thumb is to multiply the raw vector size by roughly 1.5 to 2 for total memory.
raw bytes per vector = dimensions x 4 (float32)
index bytes per vector ~ raw bytes x 1.5 to 2 (HNSW graph overhead)
total index RAM ~ vectors x index bytes per vector
Example: 10,000,000 vectors at 1536 dims, float32
raw = 1536 x 4 = 6,144 bytes (~6 KB) per vector
index = 6,144 x 1.75 ~ 10,752 bytes (~10.5 KB) per vector
total = 10,000,000 x 10,752 ~ 107 GB of RAMThat 107 GB number is why the index bucket dominates: RAM at that scale is a real monthly cost, and it's the number quantization and dimension cuts attack directly. Now run the same formula with the levers applied and watch it shrink.
| Configuration | Bytes / vector (raw) | Index RAM (10M vectors) | Typical recall cost |
|---|---|---|---|
| 1536 dims, float32 (baseline) | 6,144 | ~107 GB | — |
| 768 dims, float32 (Matryoshka truncate) | 3,072 | ~54 GB | small |
| 1536 dims, int8 quantized | 1,536 | ~27 GB | 1–3% |
| 1536 dims, binary quantized | 192 | ~3 GB | larger, recover with rescoring |
The levers, by bucket
Here is the practical toolbox. Each lever names the bucket it cuts and the recall it costs, so you can pull the cheap ones first and only reach for the painful ones if you must.
Cut embedding spend (Bucket 1)
- Cache by content hash. Hash each document's text; only re-embed when the hash changes. This alone often eliminates most embedding spend, at zero recall cost — you're just not paying twice for the same text.
- Batch requests. Embedding APIs are cheaper and faster per item when you send many texts per call instead of one. Free savings, no quality change.
- Pick a smaller or cheaper model. A smaller embedding model costs less per token and produces smaller vectors (helping Bucket 2 too). The recall cost depends on your data — measure on your own queries, don't trust a leaderboard.
- Cache query embeddings. Popular queries repeat. Caching the vector for a frequent query skips an API call on every hit.
Shrink the index (Bucket 2 — usually the big win)
- Quantization. Store each vector number in fewer bits — int8 instead of float32 cuts memory ~4x for a 1–3% recall hit; binary quantization (1 bit per number) cuts it ~32x but needs a rescoring pass to recover quality. This is the single highest-leverage move at scale.
- Matryoshka truncation. Some embedding models are trained so you can chop a 1536-dim vector down to 768 or 256 dims and keep most of the meaning. Half the dimensions is roughly half the memory and half the storage, often for a small recall cost — see embedding dimensions.
- Disk-based ANN. Keep the index on fast SSD instead of RAM. Latency rises (milliseconds to tens of milliseconds), but memory cost drops dramatically — ideal for large, latency-tolerant corpora.
- Tune index parameters. HNSW's build and search parameters trade memory and speed against recall. Lowering connectivity shrinks the graph; you don't always need the most accurate setting.
Trim storage and egress (Bucket 3)
- Keep data in one region. Co-locate your app, vector store, and embedding service to avoid cross-region and cross-cloud egress fees — often the sneakiest cost.
- Don't store what the index already holds. If your quantized index is the source of truth for search, you may not need a second full-precision copy of every vector on disk.
- Use tiered storage. Cold or rarely-queried vectors can live on cheaper object storage and be loaded on demand.
A worked example: from $X to a quarter of it
Imagine a documentation-search product: 10 million chunks, 1536-dim float32 vectors, re-embedded on every weekly deploy, all in a RAM-resident HNSW index, served from a different cloud region than the app. Walk the buckets.
- Embedding cache (Bucket 1). The weekly deploy re-embedded all 10M chunks even though almost none changed. Hashing content and embedding only the diff cuts ongoing embedding spend by the share of unchanged docs — typically over 95%. Recall cost: zero.
- int8 quantization (Bucket 2). The ~107 GB index drops to ~27 GB, letting it fit on a much smaller instance. Recall cost: 1–3%, measured and accepted.
- Dimension truncation (Bucket 2). The chosen model supports Matryoshka, so 1536 → 768 dims halves what's left. Recall cost: small, measured on real queries.
- Region co-location (Bucket 3). Moving the vector store into the app's region removes per-gigabyte egress on every backup and sync. Recall cost: zero.
Notice the order: the two zero-recall levers (caching, co-location) come for free and should always be pulled. The recall-costing levers (quantization, truncation) are pulled deliberately, each followed by a recall check. You stop pulling the moment quality drops below your bar — not before, not after.
Going deeper
Once the obvious levers are pulled, a few subtler ideas separate a cost-aware system from a merely cheap one.
Rescoring rescues aggressive compression. Binary quantization looks too lossy to use — until you pair it with a two-pass search: retrieve a generous candidate set fast and cheap from the binary index, then re-rank those few candidates against full-precision (or int8) vectors. You get most of the memory savings with most of the recall back. This retrieve-cheap-then-refine pattern is the standard way to make extreme quantization safe.
The cost of a query is not the cost of a document. Embedding a corpus is a one-time bulk job; embedding queries is forever. At very high query volume, query-side embedding and search compute can overtake the one-time ingestion cost. Profile both — the right lever for an ingest-heavy system (cache documents) differs from a query-heavy one (cache queries, scale read replicas).
Filtering changes the math. If most queries are scoped to a subset (one customer, one language, recent docs), heavy metadata filtering can mean you search a fraction of the index per query — sometimes letting you shard so each node holds less in RAM. The interaction between filtering and ANN indexes is subtle, so test it.
Sometimes the cheapest vector database is your existing one. Before paying for a dedicated service, check whether vector search inside a database you already run (like Postgres with pgvector) covers your scale. The cheapest index is the one you don't add to your bill — see choosing a vector database and the alternatives for when a dedicated store is worth it.
The durable lesson: cost optimization is measurement, not guesswork. Split the bill into the three buckets, find the one that dominates, pull the zero-recall levers everywhere, and pull the recall-costing levers only as far as a real evaluation lets you. A 10x cheaper system that still finds the right answers is almost always sitting one or two well-measured levers away.
FAQ
How do I reduce vector database costs without hurting search quality?
Start with the zero-recall levers: cache embeddings by content hash so you stop re-embedding unchanged documents, batch your embedding calls, and co-locate your data in one region to avoid egress fees. These cut the bill with no effect on recall. Only then reach for int8 quantization or dimension truncation, measuring recall before and after each change.
Where does most vector search spend actually go?
Usually the in-memory index. To search millions of vectors in milliseconds, ANN indexes like HNSW typically live in RAM, the most expensive resource in the stack, and that memory grows with both the number of vectors and their dimensionality. Embedding generation and disk storage are real but usually smaller — split your bill into those three buckets to see which one dominates yours.
Does quantization lower vector storage cost a lot?
Yes — it's often the single biggest lever at scale. Storing each number as int8 instead of float32 cuts memory roughly 4x for a typical 1–3% recall hit, and binary quantization cuts it about 32x if you add a rescoring pass to recover quality. Always measure recall on your own queries, since the exact cost depends on your data.
How can I cut embedding API costs?
Cache results by hashing each document's text and only re-embedding when the text changes — this often removes most ongoing spend because most documents don't change between deploys. Also batch many texts per API call, cache embeddings for frequent queries, and consider a smaller or cheaper embedding model if it holds up on your data.
Is it cheaper to truncate embedding dimensions?
It can be, if your embedding model was trained for it (Matryoshka representation learning). Chopping a 1536-dim vector to 768 roughly halves both index memory and storage, usually for a small recall cost. Don't blindly truncate a model that wasn't trained this way — you'll lose more quality than you save.
Should I keep my vector index in RAM or on disk?
RAM gives the fastest queries but is the priciest resource, so it dominates costs at scale. Disk-based ANN keeps the index on fast SSD, dropping memory cost a lot in exchange for higher latency (tens of milliseconds instead of single-digit). Use disk for large, latency-tolerant corpora and RAM for small, latency-critical ones.