What Is a Vector Database? | Understanding Things

Once you accept that embeddings place text as points in a high-dimensional space, the next question is purely operational: given a query point, how do you find its nearest neighbors among millions or billions of stored points, quickly? That operational problem is what a vector database solves. It is not a different kind of database with different semantics — it’s a database optimized for one specific query shape.

The problem it solves

Naively, finding the nearest neighbors of a query vector means computing its distance to every stored vector and sorting. That’s linear in the number of stored vectors, per query. It’s fine for a few thousand vectors and untenable for tens of millions queried at interactive latency.

A vector database’s job is to make that lookup sub-linear by building an index over the vector space ahead of time — trading a build cost and some accuracy for query speed.

Approximate, not exact

Almost every vector database in production use approximate nearest neighbor (ANN) search, not exact search. Two index families dominate:

Graph-based indexes (HNSW is the common one) build a navigable graph over the points, where each search walks the graph greedily toward the query, converging on a close-but-not-guaranteed-exact set of neighbors.
Partition-based indexes (IVF and its variants) cluster the space ahead of time and restrict search to the clusters nearest the query, trading a chance of missing a true neighbor in an unsearched cluster for a large speedup.

The word “approximate” is doing real work here: these indexes have a tunable recall — the fraction of true nearest neighbors they actually return — and pushing recall higher generally costs latency or memory. This is a genuine engineering trade-off, not an implementation detail to ignore. A retrieval system silently operating at 85% recall because nobody tuned the index will fail in ways that look like model quality problems but are actually infrastructure configuration.

What else it’s actually storing

A vector database is rarely just vectors. Production usage requires:

Metadata alongside each vector — source document ID, timestamp, access control tags, section — because raw geometric proximity alone can’t answer “give me the nearest neighbors that this user is allowed to see and published after date X.”
Filtered search, combining the ANN index with metadata predicates, which is harder than it sounds: naively filtering after ANN search can return too few results if the filter is restrictive, and pushing the filter into the index changes the trade-offs above again.
A write path for updates and deletes, which is nontrivial for graph-based indexes that assume a mostly-static structure — many systems handle this with soft deletes and periodic re-indexing rather than true in-place updates.

Where it fits in a retrieval-augmented system

Query
  ↓
Embed query
  ↓
Vector database (ANN + metadata filter)
  ↓
Candidate documents
  ↓
Reasoning / generation over candidates

The vector database’s entire job ends at “candidate documents.” It has no opinion about whether those candidates actually answer the query, whether they contradict each other, or how they should be synthesized into a response — that’s the reasoning stage’s job, and conflating the two stages is one of the more common architectural mistakes in RAG systems, discussed directly in Why Retrieval and Reasoning Should Be Separate.

Do you need one?

Not always. A vector database earns its complexity when you have enough vectors that brute-force search is genuinely too slow, or when you need the metadata-filtering and operational features (horizontal scaling, managed indexing, hybrid search) that come with it. Below that threshold — a common threshold is somewhere in the low millions of vectors, though it depends heavily on latency requirements — an in-memory brute-force search or a simple index inside an existing datastore is often simpler, cheaper, and just as correct. Reaching for a dedicated vector database before you have that scale is adding a moving part your system doesn’t yet need.