diff --git a/docs/docs.json b/docs/docs.json
index f8931c1..2ced6ab 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -18,6 +18,7 @@
"family": "Inter"
},
"styling": {
+ "latex": true,
"codeblocks": {
"theme": {
"light": "vitesse-light",
@@ -108,10 +109,10 @@
"pages": [
"indexing/index",
"indexing/vector-index",
+ "indexing/quantization",
"indexing/fts-index",
"indexing/scalar-index",
"indexing/gpu-indexing",
- "indexing/quantization",
"indexing/reindexing"
]
},
diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx
index fd15d05..7928bdc 100644
--- a/docs/indexing/index.mdx
+++ b/docs/indexing/index.mdx
@@ -1,162 +1,35 @@
---
title: "Indexing Data"
sidebarTitle: "Overview"
-description: "Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search."
+description: "Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more."
icon: "list"
---
-Embeddings for a given dataset are made searchable via an **index**. The index is constructed by using data structures that store the embeddings such that it's very efficient to perform scans and lookups on them.
+An **index** is a data structure that facilitates efficient scans and lookups on an embedded dataset.
+LanceDB provides a comprehensive suite of indexes to optimize performance across different use cases and data types:
-LanceDB provides a comprehensive suite of indexing strategies to optimize query performance across diverse workloads:
-
-- **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors
+- **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings)
- **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases
-- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices)
-
-
-Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
-
-- Vector search (prefilter or post-filter results using metadata)
-- Full-text search (combining keyword matching with structured filters)
-- SQL scans (optimizing WHERE clauses on scalar columns)
-- Key-value lookups (enabling rapid primary key-based retrievals)
-
-
-## Supported Index Types
+- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data
-LanceDB provides a comprehensive suite of indexing strategies for different data types and use cases:
+## Supported Indexes
| Index | Use Case | Description |
| :--------- | :------- | :---------- |
-| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: `l2` `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`|
-| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: `l2` `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`|
+| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
**Quantizations**: `None/Flat` `PQ` `SQ` `RQ`|
+| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs in each partition. Provides improved search quality over pure IVF while maintaining scalability.
**Quantizations**: `None/Flat` `SQ` `PQ`|
| `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. |
| `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. |
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |
-
-TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
-
-
-
-**Operational checks**
-
-For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows.
-
-By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to
-`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar
-columns, not list columns; use `LabelList` for list containment filters.
-
-
-### Quantization Types
-
-Vector indexes can use different quantization methods to compress vectors and improve search performance:
+## Quantization
+LanceDB also supports several [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and reduce storage requirements:
| Quantization | Use Case | Description |
| :----------- | :------- | :---------- |
| `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. |
-| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
| `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
+| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
| `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |
-
-## Understanding the IVF-PQ Index
-
-An ANN (Approximate Nearest Neighbors) index is a data structure that represents data in a way that makes it more efficient to search and retrieve. Using an ANN index is faster, but less accurate than kNN or brute force search because, in essence, the index is a lossy representation of the data.
-
-A key distinguishing feature of LanceDB is it uses a disk-based index: IVF-PQ, which is a variant of the Inverted File Index (IVF) that uses Product Quantization (PQ) to compress the embeddings.
-
-LanceDB is fundamentally different from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy.
-
-## IVF-PQ
-
-IVF-PQ is a composite index that combines inverted file index (IVF) and product quantization (PQ). The implementation in LanceDB provides several parameters to fine-tune the index's size, query throughput, latency and recall, which are described later in this section.
-
-### Product Quantization
-
-Quantization is a compression technique used to reduce the dimensionality of an embedding to speed up search.
-
-Product quantization (PQ) works by dividing a large, high-dimensional vector of size into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector.
-
-
-
-It's important to remember that quantization is a *lossy process*, i.e., the reconstructed vector is not identical to the original vector. This results in a trade-off between the size of the index and the accuracy of the search results.
-
-As an example, consider starting with 128-dimensional vector consisting of 32-bit floats. Quantizing it to an 8-bit integer vector with 4 dimensions as in the image above, we can significantly reduce memory requirements.
-
-
-Original: `128 × 32 = 4096` bits
-Quantized: `4 × 8 = 32` bits
-
-Quantization results in a **128x** reduction in memory requirements for each vector in the index, which is substantial.
-
-
-### Inverted File Index (IVF) Implementation
-
-While PQ helps with reducing the size of the index, IVF primarily addresses search performance. The primary purpose of an inverted file index is to facilitate rapid and effective nearest neighbor search by narrowing down the search space.
-
-In IVF, the PQ vector space is divided into *Voronoi cells*, which are essentially partitions that consist of all the points in the space that are within a threshold distance of the given region's seed point. These seed points are initialized by running K-means over the stored vectors. The centroids of K-means turn into the seed points which then each define a region. These regions are then are used to create an inverted index that correlates each centroid with a list of vectors in the space, allowing a search to be restricted to just a subset of vectors in the index.
-
-
-
-During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query.
-
-
-## HNSW Index Implementation
-
-Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly.
-
-### Types of ANN Search Algorithms
-
-Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly
-
-There are three main types of ANN search algorithms:
-
-* **Tree-based search algorithms**: Use a tree structure to organize and store data points.
-* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice.
-* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex.
-
-HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a k-nearest neighbor (or k-approximate nearest neighbor) graph, which we outline below.
-HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list.
-
-### Understanding k-Nearest Neighbor Graphs
-
-The k-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple:
-
-* Each vector in the dataset is given an associated vertex.
-* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex.
-* For some applications (including nearest-neighbor search), the incoming edges are also added.
-
-Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors:
-
-* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node).
-* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex.
-* Repeat until a local optimum is found.
-
-The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors.
-
-Computing a k-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the k-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster.
-In fact, another data structure is not needed: This can be done "incrementally".
-That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex.
-
-One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index.
-
-### Hierarchical Navigable Small Worlds (HNSW)
-
-HNSW builds on k-ANN in two main ways:
-
-* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant.
-* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list.
-
-This recursive structure can be thought of as separating into layers:
-
-* At the bottom-most layer, a k-ANN graph on the whole dataset is present.
-* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present.
-* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer.
-
-Then the greedy search routine operates as follows:
-
-* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
-* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
-* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx
index c0d5156..9f9c251 100644
--- a/docs/indexing/quantization.mdx
+++ b/docs/indexing/quantization.mdx
@@ -1,79 +1,84 @@
---
title: "Quantization"
sidebarTitle: "Quantization"
-description: "Learn about quantization when creating an index in LanceDB."
+description: "Use quantization to efficiently store your LanceDB vector index."
icon: "compress"
keywords: ["quantization", "quantize", "rabitq"]
---
+import {
+ PyQuantizationCustomParams as QuantizationCustomParams,
+} from '/snippets/indexing.mdx';
-Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
+**Quantization** is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here;
+discussion of LanceDB vector indexes and quantized vector indexes can be found [here](/indexing/vector-index).
-Use quantization when:
+## Quantization Techniques
+LanceDB provides $3$ distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ).
+Recall that all quantizations perform **lossy** compression, in that they irreversibly lose some degree of precision in order to
+compactly store an index.
-- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
-- Index build time and query latency matter
+### Product Quantization (PQ)
+To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has
+ $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on.
-LanceDB currently exposes multiple quantized vector index types, including:
-- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
-- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
-- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
-- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
-- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
+Now, for each $i$, let $S_i$ represent the set containing chunk $i$ of each vector (entries $(i-1)d$ to $di - 1$).
+For each $S_i$ independently, a small set of **centroids** is computed corresponding to an approximate solution to the $k$-means clustering problem.
-Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
+For each original vector in the dataset, every chunk is associated to its nearest centroid; thus, the vector itself is associated with the concatenation of $m$ centroids.
+We then gather all our centroids (from all chunks) into a single lookup table, and for each vector, we store the concatenation of IDs of its corresponding centroids.
+At query time, we need only to compare each chunk of the queried vector with our set of centroids.
-Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
+
+ 
+
-## RaBitQ quantization
+In the above example, the original vector is split into chunks (subvectors), each of which is associated to a centroid. The stored quantization code for this vector
+is the concatenation $2 || 1 || 4 || 3$. The original vector required $128$ dimensions $\times 32$-bit integers $ =4096$ bits total,
+which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of quantized storage.
-RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
+### RaBitQ Quantization (RQ)
-### How RaBitQ works
+RaBitQ is an advanced quantization technique that outperforms PQ in several ways.
+It needs no codebook to train, estimates distances very quickly at query-time,
+and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!**
+In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few thousand bits, while maintaining good recall.
-- Embeddings are grouped around centroids (as in other IVF indexes).
-- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
-- The sign pattern of that vector is stored as bits (1 bit per dimension).
-- Two small corrective factors are stored:
- 1. The distance from the original vector to its centroid
- 2. The dot product between the normalized vector and its quantized version
+The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by
+applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube,
+where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497)
-Compared to `IVF_PQ`, RaBitQ:
-- Avoids training expensive PQ codebooks
-- Builds indexes faster and handles updates more easily
-- Maintains or improves recall at high dimensionality under the same storage budget
+#### Using RaBitQ
+Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`.
+Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`.
-For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
+`num_bits` determines how many bits are used to quantize each dimension.
+`1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute.
-### Using RaBitQ
+
+RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout, and cannot be read by some older LanceDB versions.
+
-You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
+See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation.
-
-When using `IVF_RQ`, vector dimensions must be divisible by `8`.
-
+### Scalar Quantization (SQ)
+Scalar quantization quantizes each entry of a vector independently, by simply replacing it with the closest of a pre-defined set of values.
+In practice, it often uses $8$ bits per dimension of a vector, and supports very fast encoding and decoding.
-`num_bits` controls how many bits per dimension are used:
+For example, suppose we know all vector entires across our dataset lie in the range $[-128 \times 10^5, 127 \times 10^5]$.
+In this case, we could quantize a given value $v$ as an $8$-bit representation of the integer $j$, where $j \times 10^5$ is the closest value to $v$
+among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$.
-1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
-It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
+## Quantization API Reference
-
-Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
-
-## API Reference
-
-The full list of parameters to the algorithm are listed below.
-
-- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"
- The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
-- `num_partitions`: Optional[int], defaults to None
- Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
-- `num_bits`: int, defaults to 1
- Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
-- `max_iterations`: int, defaults to 50
- Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
-- `sample_rate`: int, defaults to 256
- Number of samples per partition during training. Higher values may improve accuracy but increase training time.
-- `target_partition_size`: Optional[int], defaults to None
- Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
+| Parameter | Description |
+| :--- | :--- |
+| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (default `1`, RaBitQ) — not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. |
+
+`max_iterations` and `sample_rate` also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they're documented as general [Build-time Parameters](/indexing/vector-index#build-time-parameters) instead. All three are passed as keyword arguments to `create_index`, alongside `index_type`:
+
+
+
+ {QuantizationCustomParams}
+
+
diff --git a/docs/indexing/scalar-index.mdx b/docs/indexing/scalar-index.mdx
index 7e71459..c5a511d 100644
--- a/docs/indexing/scalar-index.mdx
+++ b/docs/indexing/scalar-index.mdx
@@ -27,6 +27,15 @@ LanceDB supports four types of scalar indexes:
- `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries.
- `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
+
+Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
+
+- Vector search (prefilter results using metadata)
+- Full-text search (combining keyword matching with structured filters)
+- SQL scans (optimizing WHERE clauses on scalar columns)
+- Key-value lookups (enabling rapid primary key-based retrievals)
+
+
## Choosing the Right Index Type
| Data Type | Filter | Index Type |
diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx
index 359a6d2..24deb4f 100644
--- a/docs/indexing/vector-index.mdx
+++ b/docs/indexing/vector-index.mdx
@@ -1,7 +1,7 @@
---
title: "Vector Indexes"
sidebarTitle: "Vector Index"
-description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes."
+description: "Build and manage LanceDB vector indexes."
icon: "arrow-up-right-dots"
---
import {
@@ -24,214 +24,144 @@ import {
PyVectorIndexCustomName as VectorIndexCustomName,
} from '/snippets/indexing.mdx';
-You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two kinds of vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**.
-
-**IVF + HNSW**
-
-In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions.
-
-
-### Manual Indexing
-
-If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process.
-
-### Automatic Indexing
-
- Enterprise-only
-Vector indexing is managed **automatically** in LanceDB Enterprise. As soon as data is updated, the system updates the index and optimizates it. *This is done asynchronously as a background process*.
+Vector indexes are robust tools in facilitating fast searches across large numeric datasets.
+LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with several techniques that provide benefits across a variety of use cases.
-When you create a table in LanceDB Enterprise, LanceDB automatically:
+## Choosing the Right Index
-- Infers the vector columns from the schema
-- Create an optimized `IVF_PQ` index without manual configuration
-- Automatically configure indexing parameters
-
-The default distance is `l2` (Euclidean).
-
-
-You can call `create_index()` with different parameters to create a new index -- this replaces any existing index.
-Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter.
-
-
-Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving.
-
-Rows appended after an index build remain outside that index until optimization refreshes it. Normal
-search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
-fallback and searches only indexed rows.
+LanceDB vector indexes are combined with several [quantization](/indexing/quantization) techniques to admit efficient storage.
+The following table lists provided quantized vector indexes and their common use cases. You can specify index type manually in
+Lance with `index_type`.
-## Choose the Right Index
-Use this table as a quick starting point for choosing the right index type and quantization method for your use case:
+| If your priority is... | Use this index | Why | Approx. compression ratio | Python config class |
+| :--- | :--- | :--- | :--- | :--- |
+| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` |
+| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | `IvfRq` |
+| | `IVF_SQ` | IVF indexing with scalar quantization | Varies | `IvfSq` |
+| | `IVF_HNSW_PQ` | IVF-HNSW indexing with product quantization | Varies | `IvfHnswPq` |
+| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` |
+| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` |
+| | `IVF_FLAT` | IVF indexing with no quantization | `1` | `IvfFlat` |
-| If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors |
-| :--- | :--- | :--- | :--- |
-| Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead |
-| Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size |
-| Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size |
-| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) |
-If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance.
+If your vector search frequently includes metadata filters (`where(...)`), use `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance.
-Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration.
-For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`.
+## Understanding Vector Indexes
-### Index Tuning
+LanceDB offers two vector indexes, which can be created on any numeric Lance dataset:
+**Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**.
-Start with these values, then tune for your workload:
+### IVF
-- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`)
- - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer)
- - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger.
- - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing.
-- `IVF_RQ`
- - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets.
-- `IVF_PQ`
- - `num_partitions`: start at `num_rows // 4096` (rounded to an integer).
- - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes.
- - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance.
+The **Inverted File Index (IVF)** accelerates ANN searches by drastically reducing the search space. The index consists of
+a small set of *centroids* corresponding to an approximate solution to the
+[$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem.
+Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors,
+called its *partition*.
+
+ 
+
-## Example: Construct an IVF Index
+At query time, we can compare the queried vector to the smaller set of *centroids* (as opposed to the entire dataset)
+for a closest match, then run a brute-force comparison against its resulting partition. This technique quickly prunes a large search space,
+giving an approximate ANN result.
-In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search.
+However, observe that a queried vector may lie near the boundary of $2$ or more partitions; thus, the true nearest neighbors are scattered across several different partitions.
+ To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of partitions searched.
+A high `nprobes` parameter will yield more accurate results at slightly higher runtime.
-Make sure you have enough data in your table (at least a few thousand rows) for effective index training.
+
+ 
+
-### Index Configuration
+### HNSW
+**Hierarchical Navigable Small World (HNSW)** constructs a layered graph hierarchy on the vector set, with edges representing distances.
+We can visualize an HNSW index as a vertical stack of graphs, with the top layer having very few edges and
+each other layer having a multiplicative factor more edges than the layer above it.
-Sometimes you need to configure the index beyond default parameters:
+High layers of a HNSW hierarchy represent sparse, higher-distance networks, and lower layers represent finer, lower-distance networks.
+To query a vector $q$, the index proceeds iteratively through layers, first finding $q$'s nearest neighbor in the graph, then proceeding recursively
+through the induced subhierarchy until the lowest layer is reached.
-- Index Types:
- - `IVF_HNSW_FLAT`: highest recall, with no vector quantization
- - `IVF_HNSW_SQ`: best recall/latency trade-off
- - `IVF_RQ`: best compression for large, high-dimensional datasets
- - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance
-- `metrics`: default is `l2`, other available are `cosine` or `dot`
- - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar)
-- `num_partitions`: use index-specific starting points from the section above:
- - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576`
- - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096`
-- `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition
- count from a target number of rows per partition. If you set both `num_partitions` and
- `target_partition_size`, `num_partitions` takes precedence.
-- `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search.
+
+ 
+
-Let's take a look at a sample request for an IVF index:
+To visualize this process, imagine that you must drive your car from San Francisco to a specific house in Boston. Initially, you must first drive thousands of miles
+on the interstate freeway I-90 E. Eventually, you merge onto the Massachusetts Turnpike, the center
+of the greater Boston highway system. From there, you use a series of increasingly smaller, narrower roads within the city (Charles River Bridge, St. Paul St,
+Thatcher St) before finally reaching the house.
+A key observation is that you must initially travel far distances through long-distance road networks
+(the interstate freeway system), before proceeding to finer and finer road networks (greater Boston highway system, central Brookline neighborhood connectors) before finally reaching
+your destination. This iterative series of road networks mimics the layered graph traversals performed by a HNSW query.
+
+**IVF + HNSW**
-
-
- {VectorIndexConfigureIvf}
-
-
-
-### 1. Setup
-
-Connect to LanceDB and open the table you want to index.
-
-
-
- {VectorIndexSetup}
-
-
-
-### 2. Construct an IVF Index
-
-Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target.
-
-
-
- {VectorIndexBuildIvf}
-
-
-
-#### Indexing nested vector fields
-
-If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`:
-
-
-
- {VectorIndexNestedField}
-
-
-
-
-Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes.
-
+In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a substructure which
+further indexes the selected vectors inside each IVF partition.
+This combines the scalability of IVF with the high recall of HNSW.
+LanceDB supports IVF-HNSW-based quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW_SQ`.
+
-### Async API and Config Objects
-With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings:
+## Using Vector Indexes
+Learn how to configure, build, and search LanceDB vector indexes, including build and search time parameters, asynchronous objects, and several examples.
-
-
- {VectorIndexAsyncConfig}
-
-
+### Configuration
-Use these Python config classes for the index types shown on this page:
+#### Build-time Parameters
-| Index type | Python config class |
+| Parameter | Description |
| :--- | :--- |
-| `IVF_FLAT` | `IvfFlat` |
-| `IVF_PQ` | `IvfPq` |
-| `IVF_RQ` | `IvfRq` |
-| `IVF_SQ` | `IvfSq` |
-| `IVF_HNSW_FLAT` | `IvfHnswFlat` |
-| `IVF_HNSW_PQ` | `IvfHnswPq` |
-| `IVF_HNSW_SQ` | `IvfHnswSq` |
-
-### 3. Query the IVF Index
-
-Search using a random 1,536-dimensional embedding.
+| `metric` | Default is `l2`, others available are `cosine` and `dot`.
+| `num_partitions` | The number of IVF partitions constructed (corresponds to the $k$ in $k$-means clustering). Targets roughly `sqrt(num_rows)` by default. |
+| `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size` if both are set. |
+| `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. |
+| `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. |
+| `sample_rate` | Number of k-means training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. |
-
-
- {VectorIndexQueryIvf}
-
-
-
-#### Search Configuration
-
-Core knobs available on a vector search call:
+#### Search-time Parameters
| Parameter | Description |
| :--- | :--- |
-| `limit` | Number of results to return (`k`). |
+| `limit` | Number of results to return (the `k` in `k-ANN`). |
| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. |
-| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. |
-| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. |
-| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. |
-| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. |
+| `minimum_nprobes` | Minimum number of partitions scanned. |
+| `maximum_nprobes` | Maximum number of partitions scanned. Only scans more than `minimum_nprobes` if an initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. |
+| `ef` | HNSW search-time exploration factor. Start around `1.5 * k` and increase up to `10 * k` for higher recall. |
+| `refine_factor` | Reads and reranks additional candidates in memory to recover recall lost to quantization. |
+| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. |
+| `bypass_vector_index()` | Ignore the ANN index entirely and perform an exact (flat) scan. Can be used to measure ANN `recall@k`, or to query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). |
+
+**Recommended `nprobes` behavior by index type:**
+
+| Index type | Guidance |
+| :--- | :--- |
+| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. |
+| `IVF_RQ`, `IVF_PQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. |
+
-**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count.
+**Filtered queries and adaptive `nprobes`.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes`
+partitions and uses a wider scan if too few rows are found.
+Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count.
+Here is an example of a vector search exercising several of the above parameters.
+
{VectorIndexNprobes}
-Recommended `nprobes` behavior by index type:
-| Index type | Guidance |
-| :--- | :--- |
-| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. |
-| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. |
-| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. |
-
-#### Advanced Search Controls
-
-These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints.
-
-| Method | Description |
-| :--- | :--- |
-| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. |
-| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). |
+LanceDB also supports advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints.
**Thresholding with `distance_range`:**
@@ -241,9 +171,9 @@ These controls are useful for thresholded retrieval, recall measurement, and wor
-**Measuring recall with `bypass_vector_index`:**
+**Using `bypass_vector_index`:**
-Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload.
+Use `bypass_vector_index` to compute an exact **kNN** result. Note that exact queries may be prohibitively slow on production scales.
@@ -251,105 +181,102 @@ Compare ANN results against a flat-scan ground truth to compute recall@k. This i
-
-Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries.
-
-
-Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules.
+Multivector indexing currently requires `distance_type="cosine"`. Use `bypass_vector_index()` for non-`cosine` queries on a multivector column. See [Multivector Search](/search/multivector-search) for the full rules.
-## Example: Construct an HNSW Index
+#### Async API and Config Objects
-### Index Configuration
-
-There are four key parameters to set when constructing an HNSW index:
-
-- `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization.
-- `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`.
-- `m`: The number of neighbors to select for each vector in the HNSW graph.
-- `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph.
-
-### 1. Construct an HNSW Index
-
-The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`.
+Create vector indexes asynchronously with `await table.create_index("vector", config=...)`. The `config` object
+admits the same build-time parameters described above — pass an instance of a Python config class from the table in [Choosing the Right Index](#choosing-the-right-index):
- {VectorIndexBuildHnsw}
+ {VectorIndexAsyncConfig}
-### 2. Query the HNSW Index
+### IVF Indexes
+
+This example creates and queries an `IVF_PQ` index for a table of vectors with respect to `cosine` similarity.
+Specify `vector_column_name` if you have multiple vector columns or non-default names.
- {VectorIndexQueryHnsw}
+ {VectorIndexBuildIvf + VectorIndexQueryIvf}
-## Example: Construct a Binary Vector Index
+
+TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization).
+
-Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.
+For a vector field nested inside a struct, pass its full dotted path as `vector_column_name` (e.g. `image.embedding`) — the same path is used at query time and is what `list_indices()` reports under `columns`. See [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full path syntax.
-### Index Configuration
+
+
+ {VectorIndexNestedField}
+
+
-- Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space.
-- Index Type: `IVF_FLAT` is used for indexing binary vectors
-- `metric`: the `hamming` distance is used for similarity search
-- The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16.
+### IVF-HNSW Indexes
-
-**`IVF_FLAT` + `hamming` is the only supported path for binary vectors.**
+Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes:
-- `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns.
-- Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`.
-
+| Parameter | Description |
+| :--- | :--- |
+| `m` | The number of neighbors to select for each vector in the HNSW graph. |
+| `ef_construction` | The number of candidates to evaluate during the construction of the HNSW graph. Start at `150`; increase for better recall, decrease for faster indexing. |
-### 1. Create Table and Schema
+Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above.
+The snippet below builds and queries an `IVF_HNSW_SQ` index.
- {VectorIndexBinarySchema}
+ {VectorIndexBuildHnsw + VectorIndexQueryHnsw}
-### 2. Generate and Add Data
+### Managing Vector Indexes
-
-
- {VectorIndexBinaryAddData}
-
-
+ Enterprise-only
+In LanceDB Enterprise, vector indexes are managed **automatically**. The system asynchronously updates and optimizes indexes as a background process:
+- Automatically manages indexing parameters and storage
+- Infers vector columns from the schema
-### 3. Construct the Binary Index
+ Open-Source
+ LanceDB OSS users can manually create vector indexes by calling `table.create_index()`.
+ See the above sections for guidance on manually tuning index parameters as data changes.
-
-
- {VectorIndexBinaryBuildIndex}
-
-
+
+`create_index()` returns immediately, but the vector index builds asynchronously.
+To wait until all data is indexed, specify the `wait_timeout` parameter, or call `wait_for_index(...)` afterward —
+it waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`.
+
-### 4. Vector Search
+
+Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal
+search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that
+fallback and searches only indexed rows.
+
-
-
- {VectorIndexBinarySearch}
-
-
+
+**Operational checks**
-## Check Index Status
+After appends or other writes, use `optimize()` to fold new rows into existing indexes.
+
+#### Check Index Status
-Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**.
+Vector index creation runs in the background and may take some time to complete.
+While it is ongoing, you can check its status through the API or the **LanceDB Enterprise UI**.
In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes.
-Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built.
-To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table.
+To check status programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built.
Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`):
-| Field | What it tells you |
+| Parameter | Description |
| :--- | :--- |
| `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table |
| `size_bytes` | Total size of the index files on disk |
@@ -358,9 +285,6 @@ Each entry returned by `list_indices()` also carries detailed per-index metadata
| `index_uuid`, `type_url` | Internal identifiers for the index segment |
| `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) |
-
-These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them.
-
@@ -368,12 +292,19 @@ These fields are populated for local and embedded tables. On LanceDB Enterprise
-## Custom Index Names
-The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name.
+
+These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them.
+
+#### Custom Index Names
+
+The `{column}_idx` suffix is the default naming convetion.
+Pass `name=...` to `create_index()` to override it. Once set, the custom name will be reflected in `list_indices()`, `index_stats(name)`, and `wait_for_index([name])`.
{VectorIndexCustomName}
+
+
diff --git a/docs/search/vector-search.mdx b/docs/search/vector-search.mdx
index 06381df..a86276f 100644
--- a/docs/search/vector-search.mdx
+++ b/docs/search/vector-search.mdx
@@ -60,10 +60,10 @@ The right metric improves both search accuracy and query performance. Currently,
| Distance metric | Mathematical form | Notes |
|---|---|---|
-| `l2` | $\|x-y\|_2=\sqrt{\sum_i (x_i-y_i)^2}$ | Measures the straight-line distance between two points in vector space. Calculated as the square root of the sum of squared differences between corresponding vector components. |
-| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Computed as 1 minus cosine similarity (the dot product normalized by both vector magnitudes), so vector length does not affect the score. Use for unnormalized vectors. |
-| `dot` | $x\cdot y=\sum_i x_i y_i$ | Calculates the sum of products of corresponding vector components. Provides raw similarity scores without normalization, sensitive to vector magnitudes. Use for normalized vectors for best performance. |
-| `hamming` | $\sum_i \mathbf{1}[x_i\neq y_i]$ | Counts the number of positions where corresponding bits differ between binary vectors. Only applicable to binary vectors stored as packed uint8 arrays. |
+| `l2` | $$\|x-y\|_2= \sqrt{\sum_i (x_i-y_i)^2}$$ | The $\ell_2$ norm. Measures Euclidean distance between two points in geometric space. |
+| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Insensitive to nonzero scaling of vectors. |
+| `dot` | $x\cdot y= \underset{i}{\sum} x_i y_i$ | The standard dot product of real vectors. Sensitive to nonzero scaling of vectors; use on normalized vectors. |
+| `hamming` | $\underset{i}{\sum} \,\: \mathbf{1}_{\, [x_i\neq y_i]}$ | Counts the number of bitwise-differing bits between binary vectors. |
For indexed search, supported distance metrics vary by index type:
diff --git a/docs/snippets/indexing.mdx b/docs/snippets/indexing.mdx
index 278a097..183150a 100644
--- a/docs/snippets/indexing.mdx
+++ b/docs/snippets/indexing.mdx
@@ -12,6 +12,8 @@ export const PyGpuIndexCuda = "table.create_index(\n num_partitions=256,\n
export const PyGpuIndexMps = "table.create_index(\n num_partitions=256,\n num_sub_vectors=96,\n accelerator=\"mps\",\n)\n";
+export const PyQuantizationCustomParams = "table.create_index(\n index_type=\"IVF_RQ\",\n num_bits=2,\n max_iterations=100,\n sample_rate=512,\n)\n";
+
export const PyReindexingIncremental = "table = db.open_table(\"reindexing_incremental\")\ntable.add([{\"vector\": [3.1, 4.1], \"text\": \"Frodo was a happy puppy\"}])\ntable.optimize()\n";
export const PyScalarIndexBuild = "tbl = db.open_table(\"scalar_index_build\")\ntbl.create_scalar_index(\"book_id\")\ntbl.create_scalar_index(\"publisher\", index_type=\"BITMAP\")\n";
@@ -48,7 +50,7 @@ export const PyVectorIndexBinarySearch = "query = np.random.randint(0, 2, size=n
export const PyVectorIndexBuildHnsw = "table.create_index(index_type=\"IVF_HNSW_SQ\")\n";
-export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n metric=\"cosine\",\n vector_column_name=\"keywords_embeddings\",\n)\n";
+export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n metric=\"cosine\",\n vector_column_name=\"keywords_embeddings\",\n index_type=\"IVF_PQ\",\n)\n";
export const PyVectorIndexBypassRecall = "query = np.random.random(128)\nk = 10\n\n# Ground truth: flat (exhaustive) scan, ignoring the ANN index.\ntruth = set(table.search(query).bypass_vector_index().limit(k).to_pandas()[\"id\"])\n\n# ANN results with the current nprobes setting.\nann = set(table.search(query).nprobes(20).limit(k).to_pandas()[\"id\"])\n\nrecall_at_k = len(truth & ann) / k\n";
diff --git a/docs/static/styles/style.css b/docs/static/styles/style.css
index db71d8b..5a54326 100644
--- a/docs/static/styles/style.css
+++ b/docs/static/styles/style.css
@@ -3,6 +3,20 @@
--banner-right: #e55a2b;
}
+/* Mintlify's KaTeX stylesheet currently serves the AMS font from a broken URL. */
+@font-face {
+ font-family: "LanceDB KaTeX AMS";
+ src: url("https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/fonts/KaTeX_AMS-Regular.woff2") format("woff2");
+ font-display: swap;
+ font-style: normal;
+ font-weight: 400;
+}
+
+.katex .mathbb,
+.katex .textbb {
+ font-family: "LanceDB KaTeX AMS", "KaTeX_AMS", serif !important;
+}
+
/* Mintlify banner gradient */
#banner,
:where(.banner, [data-banner], [class*="Banner_banner"]) {
diff --git a/tests/py/test_indexing.py b/tests/py/test_indexing.py
index 7f63e8f..bfe09b1 100644
--- a/tests/py/test_indexing.py
+++ b/tests/py/test_indexing.py
@@ -63,6 +63,7 @@ def test_vector_index_build_ivf(tmp_db):
table.create_index(
metric="cosine",
vector_column_name="keywords_embeddings",
+ index_type="IVF_PQ",
)
# --8<-- [end:vector_index_build_ivf]
@@ -294,6 +295,25 @@ def test_vector_index_hnsw(tmp_db):
assert len(df) == 2
+def test_quantization_custom_params(tmp_db):
+ table = tmp_db.create_table(
+ "quantization-custom-params",
+ _make_vector_rows(256, 64),
+ mode="overwrite",
+ )
+
+ # --8<-- [start:quantization_custom_params]
+ table.create_index(
+ index_type="IVF_RQ",
+ num_bits=2,
+ max_iterations=100,
+ sample_rate=512,
+ )
+ # --8<-- [end:quantization_custom_params]
+
+ assert table.list_indices()
+
+
def test_vector_index_binary(tmp_db):
table_name = "hamming-index-tbl"
ndim = 256