Keyword search asks whether documents contain particular terms. Vector search asks a different question: which items are close to the query in a learned space? That change makes it possible to retrieve “reset my login secret” for a query about changing a password even when the two strings share few words. The same mechanism can match images, products, source code, support tickets, and other objects that an embedding model can represent.

An embedding is only a list of numbers, and a vector index is only a data structure for finding nearby lists. Neither understands meaning in a human sense. The useful behavior comes from a chain of choices: what the model was trained to preserve, how input is prepared, which distance function is used, how approximate search is configured, and how relevance is measured against a real task.

From an object to a searchable representation

An embedding model maps an object a to a fixed-width vector f(a) in R^d. A text model might produce 768 or 1,536 floating-point coordinates; an image model may use another width and geometry. Individual coordinates usually have no stable interpretation such as “coordinate 42 means database knowledge.” Meaning is distributed across the vector, and useful relationships appear through relative position.

The representation must cover the information users expect to retrieve. For a support article, embedding only the title misses details in the body. Embedding an entire manual as one vector blurs unrelated topics. A common document pipeline extracts readable text, preserves useful headings, splits it into semantically coherent chunks with modest overlap, and stores each chunk vector alongside its document ID, section, language, tenant, permissions, and model version. The retrieved chunk can then lead back to the canonical source.

Chunking is part of the retrieval model. Tiny chunks have precise vectors but lack context; huge chunks contain context but combine several intents. Boundaries at headings and paragraphs are generally better than blind character counts. Measure chunk sizes in the tokenizer used by the embedding model, because model limits and cost are expressed in tokens, not JavaScript string length.

Normalization has two different meanings here. Input normalization is domain-specific cleanup before embedding: Unicode normalization, stable whitespace, removal of navigation chrome, or a deliberate prefix such as query: and passage: required by some models. Vector normalization scales a nonzero vector to unit L2 length:

unit⁡(x)=x∥x∥2\operatorname{unit}(x) = \frac{x}{\lVert x \rVert_2}

Unit normalization removes magnitude, making dot product equal cosine similarity. It is helpful only when magnitude is not intended to carry information. Never normalize merely because it sounds hygienic; follow the model contract and apply the same policy during indexing and querying.

Warning

Vectors from different model versions, dimensions, or input-prefix conventions do not share a trustworthy coordinate system. Store the embedding model and preprocessing version with every record, and re-embed into a new index rather than mixing incompatible vectors.

Cosine, dot product, and L2 distance

Let x and y be vectors with d coordinates. Three measures dominate vector retrieval.

Dot product grows when aligned coordinates have large products:

dot⁡(x,y)=∑i=1dxiyi\operatorname{dot}(x,y) = \sum_{i=1}^{d} x_i y_i

Higher is more similar. Dot product uses both direction and magnitude, so a vector with a large norm may outrank a better-aligned smaller vector. Some embedding models intentionally encode confidence or popularity in the norm; others do not.

Cosine similarity divides dot product by both norms:

cosine⁡(x,y)=dot⁡(x,y)∥x∥2∥y∥2\operatorname{cosine}(x,y) = \frac{\operatorname{dot}(x,y)}{\lVert x \rVert_2\lVert y \rVert_2}

For nonzero vectors it ranges from -1 to 1, with larger values indicating closer direction. On unit-normalized vectors, cosine similarity and dot product produce the same ranking.

Euclidean distance, usually called L2 distance, measures straight-line separation:

L2⁡(x,y)=∑i=1d(xi−yi)2\operatorname{L2}(x,y) = \sqrt{\sum_{i=1}^{d}(x_i-y_i)^2}

Lower is more similar. For unit vectors, squared L2 distance and cosine are monotonically related:

L2⁡(x,y)2=2−2cosine⁡(x,y)\operatorname{L2}(x,y)^2 = 2 - 2\operatorname{cosine}(x,y)

Therefore, switching between those measures does not change ranking if every vector is normalized correctly. Without that condition, it can change results dramatically.

Measure Better result Uses magnitude Typical fit Common mistake
Cosine similarity Higher No Text embeddings where direction carries semantics Dividing by a zero norm
Dot product Higher Yes, unless vectors are unit length Models trained for maximum inner product Comparing raw vectors when the model expects normalization
L2 distance Lower Yes Spatial or multimodal models trained with Euclidean geometry Sorting descending as if it were similarity

The index metric must match both the model recommendation and the application code. A database may expose “cosine distance” as 1 - cosine_similarity, so smaller values are better even though the word cosine appears in the API. Verify the exact semantics instead of relying on the operator name.

Exact similarity and search in code

Exact search computes a score against every stored vector, giving O(Nd) query time for N vectors of width d. It is the simplest correct baseline and remains practical for small collections, offline evaluation, or candidate sets already narrowed by metadata. A robust implementation rejects mismatched dimensions and zero vectors instead of returning misleading NaN values.

python
from __future__ import annotations

from math import sqrt
from typing import Iterable, Sequence


def _same_dimension(x: Sequence[float], y: Sequence[float]) -> None:
    if len(x) == 0 or len(x) != len(y):
        raise ValueError("vectors must be non-empty and have equal dimensions")


def dot_product(x: Sequence[float], y: Sequence[float]) -> float:
    _same_dimension(x, y)
    return sum(left * right for left, right in zip(x, y, strict=True))


def l2_distance(x: Sequence[float], y: Sequence[float]) -> float:
    _same_dimension(x, y)
    return sqrt(sum((left - right) ** 2 for left, right in zip(x, y, strict=True)))


def cosine_similarity(x: Sequence[float], y: Sequence[float]) -> float:
    _same_dimension(x, y)
    x_norm = sqrt(sum(value * value for value in x))
    y_norm = sqrt(sum(value * value for value in y))
    if x_norm == 0.0 or y_norm == 0.0:
        raise ValueError("cosine similarity is undefined for a zero vector")
    return dot_product(x, y) / (x_norm * y_norm)


def exact_search(
    query: Sequence[float],
    records: Iterable[tuple[str, Sequence[float]]],
    limit: int = 5,
) -> list[tuple[str, float]]:
    if limit < 0:
        raise ValueError("limit must be non-negative")
    scored = [
        (record_id, cosine_similarity(query, vector))
        for record_id, vector in records
    ]
    return sorted(scored, key=lambda item: (-item[1], item[0]))[:limit]

The deterministic ID tie-break makes tests and pagination stable. For repeated searches, normalize document vectors once during ingestion and normalize each query once, then use dot product in the inner loop. Recomputing every document norm per request wastes CPU.

The same optimized pattern in TypeScript keeps unit vectors in the index and performs exact top-k search with dot product:

ts
type VectorRecord = {
  id: string;
  vector: readonly number[];
};

function normalize(vector: readonly number[]): number[] {
  if (vector.length === 0) throw new Error('vector must not be empty');
  const norm = Math.hypot(...vector);
  if (norm === 0) throw new Error('cannot normalize a zero vector');
  return vector.map((value) => value / norm);
}

function dotProduct(left: readonly number[], right: readonly number[]): number {
  if (left.length === 0 || left.length !== right.length) {
    throw new Error('vectors must be non-empty and have equal dimensions');
  }
  return left.reduce((sum, value, index) => sum + value * right[index]!, 0);
}

function exactSearch(
  query: readonly number[],
  records: readonly VectorRecord[],
  limit = 5,
): Array<{ id: string; score: number }> {
  if (!Number.isInteger(limit) || limit < 0) {
    throw new Error('limit must be a non-negative integer');
  }

  const unitQuery = normalize(query);
  return records
    .map((record) => ({
      id: record.id,
      score: dotProduct(unitQuery, record.vector),
    }))
    .sort((left, right) => right.score - left.score || left.id.localeCompare(right.id))
    .slice(0, limit);
}

function buildIndex(records: readonly VectorRecord[]): VectorRecord[] {
  return records.map((record) => ({
    id: record.id,
    vector: normalize(record.vector),
  }));
}

This code assumes stored vectors were normalized as shown. In numerical workloads, use Float32Array, batched matrix operations, or a vector database rather than spreading very wide arrays through Math.hypot. The baseline is still valuable: compare an approximate index with exact results on the same frozen corpus to calculate recall.

Tip

Keep a brute-force implementation in the evaluation suite even after production moves to ANN. It is the oracle that reveals whether a faster index is dropping relevant neighbors or whether the embedding model itself is the problem.

Why approximate nearest-neighbor indexes exist

Scanning ten million 1,536-dimensional vectors means billions of multiply-add operations per query and substantial memory bandwidth. Approximate nearest-neighbor search, or ANN, skips most vectors and accepts that an exact top result may occasionally be missed. The useful question is not whether ANN is perfectly accurate, but how much recall it retains at a latency and memory cost the service can afford.

For query q, compare the approximate top k with an exact-search oracle on the same frozen corpus:

recall@k⁡(q)=∣ANN⁡k(q)∩Exact⁡k(q)∣k\operatorname{recall@k}(q) = \frac{\left|\operatorname{ANN}_k(q) \cap \operatorname{Exact}_k(q)\right|}{k}

HNSW, short for Hierarchical Navigable Small World, builds a multilayer proximity graph. Upper sparse layers provide long jumps across the space; lower dense layers refine the neighborhood. A query enters near the top, greedily follows promising edges, descends, and explores a candidate frontier at the bottom. Larger construction effort and more graph neighbors usually improve recall but increase build time and memory. A larger query exploration parameter, commonly efSearch, improves recall at the cost of latency.

IVF, or inverted file indexing, first trains coarse centroids and assigns vectors to nearby partitions. At query time it searches only the closest nprobe partitions. More probes increase recall and latency. IVF can combine with product quantization, or PQ, which compresses vector subspaces into short codes. Compression greatly lowers memory and can improve cache behavior, but introduces additional distance error. Training centroids and codebooks on representative data is essential; a distribution shift can leave partitions badly balanced.

Approach Query behavior Memory Update profile Best use
Exact scan Scores every vector Raw vectors only Simple Small corpus and quality oracle
HNSW Traverses a proximity graph High graph overhead Good incremental inserts; deletes need care High-recall, low-latency serving
IVF Probes selected coarse partitions Moderate Often favors batch training and rebuilds Large collections with tunable probing
IVF-PQ Probes compressed partitions Low Training and rebuild complexity Very large, memory-constrained corpora

ANN configuration is workload-specific. Report recall at k, p50/p95/p99 latency, index bytes per vector, build duration, update visibility, and throughput together. Optimizing only average latency can hide tail stalls; optimizing only recall can create an index too large to remain in memory.

Indexing, querying, and metadata filters

A production retrieval system has two related flows. Indexing turns canonical objects into versioned chunks and vectors. Querying applies the same model contract, enforces filters, retrieves candidates, and optionally reranks them with a more expensive model.

flowchart LR subgraph Indexing S[Source objects] --> C[Clean and chunk] C --> E[Embedding model] E --> U[Normalize if required] U --> I[(Vector index)] C --> M[(Metadata store)] end subgraph Querying Q[User query] --> QE[Same embedding model] QE --> QU[Same normalization] F[Tenant and access filters] --> R[ANN retrieval] QU --> R R --> RR[Rerank candidates] RR --> O[Results with source links] end I --> R M --> R

Metadata filters are correctness boundaries, not cosmetic refinements. Tenant, visibility, region, language, object type, and deletion status may determine what a caller is allowed to see. Prefer pre-filtering when the engine can integrate filters into traversal. Post-filtering a top-10 ANN result may return only two authorized items even though many valid items exist deeper in the index. Fetching an oversized candidate set helps but does not guarantee completeness. Highly selective filters may need partitioned indexes, filter-aware graph traversal, or an exact search over the filtered subset.

Updates also need explicit semantics. A source change should write a new content version, generate all vectors successfully, publish them, then retire the old version. Deleting source data must delete chunks, vectors, caches, and backups according to retention policy. Eventual consistency is acceptable only when the product defines how stale results are detected and how quickly revocation becomes effective.

Hybrid retrieval often beats either method alone. Lexical search is excellent for exact identifiers, error codes, names, and rare terms; embeddings capture paraphrase and broad intent. Retrieve from both, fuse ranks with a method such as reciprocal rank fusion, then rerank a bounded candidate set. Do not force vectors to solve exact-match problems they naturally blur.

Evaluation, drift, privacy, and anti-patterns

Start evaluation with a task-specific set of queries and relevance judgments. Include frequent intents, difficult paraphrases, exact identifiers, multiple languages, no-answer cases, and permission boundaries. Useful offline metrics include recall at k, precision at k, mean reciprocal rank, and normalized discounted cumulative gain. For ANN, separately measure index recall against exact nearest neighbors; otherwise model quality and index approximation are conflated.

Online metrics should connect retrieval to user outcomes: successful answer rate, reformulation, click or citation acceptance, time to resolution, and safe abstention. A/B tests need guardrails for latency, empty-result rate, cost, and unauthorized-result incidents. Similarity scores are not calibrated probabilities, and a threshold copied from another model or corpus has no reliable meaning. Choose thresholds from labeled validation data and inspect performance by language, tenant, content type, and query class.

Drift appears when source content, query language, user intent, or the embedding model changes. Monitor vector norms, missing embeddings, chunk lengths, partition sizes, score distributions, filter selectivity, and retrieval metrics over time. Re-embedding is a data migration: build a shadow index, evaluate it, dual-read a sample, switch an alias atomically, and retain rollback until confidence is established.

Embeddings can leak sensitive properties and may preserve more source information than expected. Treat them as derived personal or confidential data when their inputs are sensitive. Encrypt in transit and at rest, isolate tenants, minimize metadata, audit access, define retention, and ensure deletion reaches every derived index. Avoid sending secrets to an external embedding API without an approved data-processing agreement and explicit threat analysis. Do not log raw queries or nearest chunks by default merely because they are useful for debugging.

Common anti-patterns are remarkably consistent:

  • Mixing vectors from different models in one search space.
  • Changing chunking or normalization without versioning and reindexing.
  • Using cosine because it is popular rather than because the model expects it.
  • Treating ANN top-k as deterministic or complete under restrictive filters.
  • Evaluating on a few hand-picked queries with no exact or lexical baseline.
  • Storing vectors without source IDs, permissions, model versions, or deletion paths.
  • Adding a vector database before proving that semantic retrieval improves the task.

Note

The most consequential retrieval bugs often live outside the nearest-neighbor algorithm: stale permissions, malformed chunks, inconsistent preprocessing, missing deletes, and an evaluation set that does not resemble production traffic.

Takeaways

  • An embedding is a model-specific representation; its value comes from relationships in the learned space, not interpretable individual coordinates.
  • Coverage and chunking determine what can be found. Preserve source links, metadata, permissions, and model versions with every vector.
  • Cosine ignores magnitude, dot product can use it, and L2 measures distance. Match the metric and normalization policy to the model contract.
  • Exact search is the correctness baseline. HNSW and IVF trade some recall for lower latency, while quantization trades additional accuracy for memory.
  • Tune recall, tail latency, throughput, memory, freshness, and filter behavior as one system rather than isolated dashboard numbers.
  • Evaluate with labeled task data, monitor drift, migrate model versions through shadow indexes, and keep lexical retrieval for exact terms.
  • Embeddings inherit the sensitivity of their source data. Access control, deletion, retention, and auditability must cover derived vectors too.