Vector Database and ANN Index Architecture
A vector database stores high-dimensional embeddings and answers nearest-neighbor queries: "which vectors are most similar to this one?" Exact search is O(N) per query, so production systems use approximate nearest neighbor (ANN) indexes that trade a little recall for orders-of-magnitude speed.
The core operation is similarity search over millions to billions of vectors under a distance metric (cosine, dot product, or L2). The architecture is defined by which ANN index it uses, how it handles metadata filtering, and how it scales across machines. Source: compiled from ANN-index literature
ANN index families
| Family | How it works | Strength | Cost |
|---|---|---|---|
| HNSW (graph) | Navigable small-world graph; greedy descent through layers | High recall, fast queries | High memory; slower builds |
| IVF (inverted file) | Cluster vectors, search only nearest clusters (nprobe) |
Tunable speed/recall | Recall drops if clusters miss |
| PQ (product quantization) | Compress vectors into codes | Massive memory savings | Lower precision |
| IVF-PQ | Cluster + compress | Billion-scale on modest RAM | Two sets of knobs to tune |
| Flat | Brute-force exact | Perfect recall | O(N) — small sets only |
HNSW is the common default for quality; IVF-PQ is the common default at extreme scale. Source: Malkov & Yashunin, HNSW, arXiv:1603.09320
System components
- Embedding ingestion — vectors arrive from the RAG ingestion pipeline with attached metadata.
- Index — the ANN structure, held in memory for speed.
- Metadata filter — pre- vs post-filtering. Post-filtering (search then filter) can return too few results; pre-filtering (filter then search) is correct but harder to make fast — a key engineering seam.
- Persistence — vectors + index snapshots on disk for restart; the index is often rebuilt or memory-mapped on load.
- Sharding & replication — partition by vector id for capacity, replicate for QPS/availability (see Sharding and Partitioning Architecture).
Design implications
- Recall is a tunable, not a guarantee.
efSearch(HNSW) andnprobe(IVF) trade latency for recall; measure recall@k on real queries, not synthetic ones. - Memory dominates cost. Float32 vectors are heavy; quantization (PQ, scalar, binary) is how billion-scale stays affordable.
- Filtered search is the hard part. Combining "similar AND metadata-matching" efficiently is where naive setups fall over.
- Updates are not free. Graph indexes degrade with churn; heavy delete/insert workloads need periodic rebuilds.
- Pair with a keyword index. Pure vector search misses exact tokens — hybrid retrieval needs both, fused at query time.
Kevin's qmd is the brain's vector layer in practice: local GGUF embeddings + BM25, fused and reranked on-device, no cloud vector DB.
Architecture Position
| Axis | Value |
|---|---|
| Family | Memory, context, and retrieval |
| Boundary owned | Embedding storage and approximate nearest-neighbor query architecture. |
| Read with | RAG System Architecture, Sevenfold Hybrid Search, Knowledge System Architecture Comparison |
| Use this page when | choosing or evaluating vector indexes |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Memory, context, and retrieval family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Embedding storage and approximate nearest-neighbor query architecture. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured exact-vs-ANN tradeoff, the HNSW/IVF/PQ/IVF-PQ/Flat index families, pre-vs-post metadata filtering, memory/quantization economics, and the qmd grounding. Source: Malkov & Yashunin, HNSW, arXiv:1603.09320