From d65037493039530d1ff73f587f68b9657774fab8 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 3 Aug 2026 11:54:22 -0700 Subject: [PATCH 01/44] small grammatical, typo, and flow fixes --- docs/indexing/index.mdx | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index fd15d05..7977b8e 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -5,9 +5,7 @@ description: "Optimize search performance in LanceDB using vector indexes, full- icon: "list" --- -Embeddings for a given dataset are made searchable via an **index**. The index is constructed by using data structures that store the embeddings such that it's very efficient to perform scans and lookups on them. - -LanceDB provides a comprehensive suite of indexing strategies to optimize query performance across diverse workloads: +An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads: - **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases @@ -63,15 +61,15 @@ Vector indexes can use different quantization methods to compress vectors and im ## Understanding the IVF-PQ Index -An ANN (Approximate Nearest Neighbors) index is a data structure that represents data in a way that makes it more efficient to search and retrieve. Using an ANN index is faster, but less accurate than kNN or brute force search because, in essence, the index is a lossy representation of the data. - -A key distinguishing feature of LanceDB is it uses a disk-based index: IVF-PQ, which is a variant of the Inverted File Index (IVF) that uses Product Quantization (PQ) to compress the embeddings. +An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-nearest neighbors (kNN)** problem. +It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. -LanceDB is fundamentally different from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. +LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. ## IVF-PQ -IVF-PQ is a composite index that combines inverted file index (IVF) and product quantization (PQ). The implementation in LanceDB provides several parameters to fine-tune the index's size, query throughput, latency and recall, which are described later in this section. +LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. +The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. ### Product Quantization @@ -109,7 +107,8 @@ Approximate Nearest Neighbor (ANN) search is a method for finding data points ne ### Types of ANN Search Algorithms -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly +Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. +For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. There are three main types of ANN search algorithms: From 220338a1cc07648fbad0a54fc11b7f39f0fd81ce Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 3 Aug 2026 13:49:01 -0700 Subject: [PATCH 02/44] light changes --- docs/indexing/index.mdx | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 7977b8e..0ec2c71 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -41,16 +41,16 @@ TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). **Operational checks** -For vector indexes, use the same distance metric when creating the index and searching it. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` if you need to confirm the index has caught up. `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`; it can time out if writes keep adding unindexed rows. +For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. +`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. -By default, automatic vector indexing creates `IVF_PQ`, and scalar index creation defaults to -`BTree` unless you pass another scalar index config. `BTree` and `Bitmap` indexes target scalar -columns, not list columns; use `LabelList` for list containment filters. +Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to +`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. ### Quantization Types -Vector indexes can use different quantization methods to compress vectors and improve search performance: +Vector indexes use different quantization methods to compress vectors and improve search performance: | Quantization | Use Case | Description | | :----------- | :------- | :---------- | @@ -61,7 +61,7 @@ Vector indexes can use different quantization methods to compress vectors and im ## Understanding the IVF-PQ Index -An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-nearest neighbors (kNN)** problem. +An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-Nearest Neighbors (kNN)** problem. It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. @@ -73,9 +73,10 @@ The implementation provides several parameters to fine-tune the index's size, qu ### Product Quantization -Quantization is a compression technique used to reduce the dimensionality of an embedding to speed up search. +Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. -Product quantization (PQ) works by dividing a large, high-dimensional vector of size into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. +Product quantization (PQ) first projects each large, high-dimensional vector into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. +The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. ![](/static/assets/images/indexing/ivfpq_pq_desc.png) @@ -84,10 +85,10 @@ It's important to remember that quantization is a *lossy process*, i.e., the rec As an example, consider starting with 128-dimensional vector consisting of 32-bit floats. Quantizing it to an 8-bit integer vector with 4 dimensions as in the image above, we can significantly reduce memory requirements. -Original: `128 × 32 = 4096` bits -Quantized: `4 × 8 = 32` bits +Original: `128 × 32 = 4096` bits. +Quantized: `4 × 8 = 32` bits. -Quantization results in a **128x** reduction in memory requirements for each vector in the index, which is substantial. +In this example, quantization produces a substantial **128x** reduction in the memory requirement for each indexed vector. ### Inverted File Index (IVF) Implementation From 119b8572520154b0ba43cadd6f2d21ccf0184dca Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 3 Aug 2026 16:09:58 -0700 Subject: [PATCH 03/44] more small changes --- docs/indexing/index.mdx | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 0ec2c71..3fd5df1 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -22,12 +22,12 @@ Scalar indices serve as a foundational optimization layer, accelerating filterin ## Supported Index Types -LanceDB provides a comprehensive suite of indexing strategies for different data types and use cases: +LanceDB provides a comprehensive suite of indexes for different data types and use cases: | Index | Use Case | Description | | :--------- | :------- | :---------- | -| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: `l2` `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| -| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: `l2` `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| +| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: $\ell_2$ `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| +| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: $\ell_2$ `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | @@ -61,7 +61,7 @@ Vector indexes use different quantization methods to compress vectors and improv ## Understanding the IVF-PQ Index -An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **k-Nearest Neighbors (kNN)** problem. +An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. @@ -75,22 +75,22 @@ The implementation provides several parameters to fine-tune the index's size, qu Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. -Product quantization (PQ) first projects each large, high-dimensional vector into equally sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. +Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. ![](/static/assets/images/indexing/ivfpq_pq_desc.png) -It's important to remember that quantization is a *lossy process*, i.e., the reconstructed vector is not identical to the original vector. This results in a trade-off between the size of the index and the accuracy of the search results. - -As an example, consider starting with 128-dimensional vector consisting of 32-bit floats. Quantizing it to an 8-bit integer vector with 4 dimensions as in the image above, we can significantly reduce memory requirements. +As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. -Original: `128 × 32 = 4096` bits. -Quantized: `4 × 8 = 32` bits. +Original storage: `128 × 32 = 4096` bits. +Quantized storage: `4 × 8 = 32` bits. -In this example, quantization produces a substantial **128x** reduction in the memory requirement for each indexed vector. +In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. +It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. + ### Inverted File Index (IVF) Implementation While PQ helps with reducing the size of the index, IVF primarily addresses search performance. The primary purpose of an inverted file index is to facilitate rapid and effective nearest neighbor search by narrowing down the search space. @@ -117,12 +117,12 @@ There are three main types of ANN search algorithms: * **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. * **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. -HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a k-nearest neighbor (or k-approximate nearest neighbor) graph, which we outline below. +HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. -### Understanding k-Nearest Neighbor Graphs +### Understanding $k$-Nearest Neighbor Graphs -The k-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: +The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: * Each vector in the dataset is given an associated vertex. * Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. @@ -136,7 +136,7 @@ Eventually, it was realized that the following greedy search method over such a The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. -Computing a k-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the k-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. +Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. In fact, another data structure is not needed: This can be done "incrementally". That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. From bc4d53a0fe5c1bbef68b944b2ab9e0af2502a276 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 3 Aug 2026 17:07:50 -0700 Subject: [PATCH 04/44] half done with ivf implementation --- docs/indexing/index.mdx | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 3fd5df1..5ef8bfa 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -93,12 +93,22 @@ It's important to remember that quantization is a *lossy process*, i.e., that no ### Inverted File Index (IVF) Implementation -While PQ helps with reducing the size of the index, IVF primarily addresses search performance. The primary purpose of an inverted file index is to facilitate rapid and effective nearest neighbor search by narrowing down the search space. +(note: acknowledge pq in here somewhere) -In IVF, the PQ vector space is divided into *Voronoi cells*, which are essentially partitions that consist of all the points in the space that are within a threshold distance of the given region's seed point. These seed points are initialized by running K-means over the stored vectors. The centroids of K-means turn into the seed points which then each define a region. These regions are then are used to create an inverted index that correlates each centroid with a list of vectors in the space, allowing a search to be restricted to just a subset of vectors in the index. +An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. +Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. +The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. +The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. + +The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. ![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) +To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ +for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. +Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. + +here vv During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. ![](/static/assets/images/indexing/ivfpq_query_vector.webp) From a7b1460f33ca24af71dbf8bf287ee2b57b01ded7 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 10:25:17 -0700 Subject: [PATCH 05/44] quantization light pass --- docs/indexing/index.mdx | 2 +- docs/indexing/quantization.mdx | 19 ++++++++----------- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 5ef8bfa..3b9adf3 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -93,7 +93,7 @@ It's important to remember that quantization is a *lossy process*, i.e., that no ### Inverted File Index (IVF) Implementation -(note: acknowledge pq in here somewhere) +(note: acknowledge pq in here somewhere) test An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index c0d5156..16a8802 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -1,28 +1,25 @@ --- title: "Quantization" sidebarTitle: "Quantization" -description: "Learn about quantization when creating an index in LanceDB." +description: "Learn about quantization in a LanceDB index." icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- -Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality. +Quantization compresses high-dimensional vectors into concise representations that admit efficient storage with only a small compromise in search accuracy. +Quantization is beneficial when your dataset uses high-dimensional vectors ($512, 768, 1024$ or more dimensions), or when index build time and query latency are crucial. -Use quantization when: - -- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+) -- Index build time and query latency matter - -LanceDB currently exposes multiple quantized vector index types, including: +LanceDB currently exposes several quantized vector index types, with a range of options differing in indexing technique (`IVF_*`,`IVF_HNSW_*`) and method of quantization (`PQ`,`RQ`,`SQ`). - `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples. - `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`. - `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details. - `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads. - `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search. -Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool. +Different indexing and quantization techniques may be better suited for certain use cases. For example, `IVF_PQ` works well in many cases, but RaBitQ (`IVF_RQ`) allows for more aggressive compression. +See the ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. -Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. +Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. ## RaBitQ quantization @@ -49,7 +46,7 @@ For a deeper dive into the theory and some benchmark results, see the blog post: You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`. -When using `IVF_RQ`, vector dimensions must be divisible by `8`. +When using `IVF_RQ`, the dimension of vectors must be a multiple of `8`. `num_bits` controls how many bits per dimension are used: From cd7899c666827e3482071ab34897f2cf5bfbc9ad Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 10:51:31 -0700 Subject: [PATCH 06/44] vector index light pass --- docs/indexing/vector-index.mdx | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 359a6d2..6f4f98a 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -24,7 +24,7 @@ import { PyVectorIndexCustomName as VectorIndexCustomName, } from '/snippets/indexing.mdx'; -You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two kinds of vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. +You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. **IVF + HNSW** @@ -39,13 +39,10 @@ If using LanceDB OSS, you will have to create the vector index manually, by call ### Automatic Indexing Enterprise-only -Vector indexing is managed **automatically** in LanceDB Enterprise. As soon as data is updated, the system updates the index and optimizates it. *This is done asynchronously as a background process*. - -When you create a table in LanceDB Enterprise, LanceDB automatically: - -- Infers the vector columns from the schema -- Create an optimized `IVF_PQ` index without manual configuration -- Automatically configure indexing parameters +Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: +- Infers vector columns from the schema +- Optimizes the `IVF_PQ` index without manual configuration +- Automatically manages indexing parameters The default distance is `l2` (Euclidean). @@ -62,7 +59,7 @@ fallback and searches only indexed rows. ## Choose the Right Index -Use this table as a quick starting point for choosing the right index type and quantization method for your use case: +Use this table to choose the right index and quantization type for your use case: | If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors | | :--- | :--- | :--- | :--- | From 86d2e2d344535c6e4c67af97f10988903f3a6ae9 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 11:15:22 -0700 Subject: [PATCH 07/44] whatever --- docs/indexing/index.mdx | 2 +- docs/indexing/quantization.mdx | 2 + out.txt | 376 +++++++++++++++++++++++++++++++++ 3 files changed, 379 insertions(+), 1 deletion(-) create mode 100644 out.txt diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 3b9adf3..5ef8bfa 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -93,7 +93,7 @@ It's important to remember that quantization is a *lossy process*, i.e., that no ### Inverted File Index (IVF) Implementation -(note: acknowledge pq in here somewhere) test +(note: acknowledge pq in here somewhere) An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 16a8802..307b9b2 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -19,6 +19,8 @@ LanceDB currently exposes several quantized vector index types, with a range of Different indexing and quantization techniques may be better suited for certain use cases. For example, `IVF_PQ` works well in many cases, but RaBitQ (`IVF_RQ`) allows for more aggressive compression. See the ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. +todo decide what to do here vv + Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. ## RaBitQ quantization diff --git a/out.txt b/out.txt new file mode 100644 index 0000000..8fad008 --- /dev/null +++ b/out.txt @@ -0,0 +1,376 @@ +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) --- +e7716261 docs/indexing/vector-index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 2) title: "Vector Indexes" +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 3) sidebarTitle: "Vector Index" +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 4) description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes." +e7716261 docs/indexing/vector-index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 5) icon: "arrow-up-right-dots" +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 6) --- +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 7) import { +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 8) PyVectorIndexConfigureIvf as VectorIndexConfigureIvf, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 9) PyVectorIndexSetup as VectorIndexSetup, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 10) PyVectorIndexBuildIvf as VectorIndexBuildIvf, +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 11) PyVectorIndexNestedField as VectorIndexNestedField, +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 12) PyVectorIndexAsyncConfig as VectorIndexAsyncConfig, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 13) PyVectorIndexQueryIvf as VectorIndexQueryIvf, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 14) PyVectorIndexBuildHnsw as VectorIndexBuildHnsw, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 15) PyVectorIndexQueryHnsw as VectorIndexQueryHnsw, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 16) PyVectorIndexBinarySchema as VectorIndexBinarySchema, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 17) PyVectorIndexBinaryAddData as VectorIndexBinaryAddData, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 18) PyVectorIndexBinaryBuildIndex as VectorIndexBinaryBuildIndex, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 19) PyVectorIndexBinarySearch as VectorIndexBinarySearch, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 20) PyVectorIndexCheckStatus as VectorIndexCheckStatus, +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 21) PyVectorIndexNprobes as VectorIndexNprobes, +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 22) PyVectorIndexDistanceRange as VectorIndexDistanceRange, +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 23) PyVectorIndexBypassRecall as VectorIndexBypassRecall, +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 24) PyVectorIndexCustomName as VectorIndexCustomName, +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 25) } from '/snippets/indexing.mdx'; +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 26) +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 27) You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 28) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 29) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 30) **IVF + HNSW** +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 31) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 32) In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 33) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 34) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 35) ### Manual Indexing +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 36) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 37) If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 38) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 39) ### Automatic Indexing +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 40) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 41) Enterprise-only +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 42) Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 43) - Infers vector columns from the schema +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 44) - Optimizes the `IVF_PQ` index without manual configuration +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 45) - Automatically manages indexing parameters +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 46) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 47) The default distance is `l2` (Euclidean). +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 48) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 49) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 50) You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 51) Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 52) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 53) +fa69074f docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 54) Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. +fa69074f docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 55) +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 56) Rows appended after an index build remain outside that index until optimization refreshes it. Normal +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 57) search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 58) fallback and searches only indexed rows. +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 59) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 60) ## Choose the Right Index +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 61) +cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 62) Use this table to choose the right index and quantization type for your use case: +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 63) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 64) | If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors | +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 65) | :--- | :--- | :--- | :--- | +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 66) | Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead | +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 67) | Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size | +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 68) | Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size | +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 69) | Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 70) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 71) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 72) If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 73) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 74) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 75) Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 76) For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 77) +18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 78) ### Index Tuning +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 79) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 80) Start with these values, then tune for your workload: +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 81) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 82) - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 83) - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 84) - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 85) - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 86) - `IVF_RQ` +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 87) - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 88) - `IVF_PQ` +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 89) - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 90) - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 91) - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 92) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 93) ## Example: Construct an IVF Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 94) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 95) In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 96) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 97) Make sure you have enough data in your table (at least a few thousand rows) for effective index training. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 98) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 99) ### Index Configuration +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 100) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 101) Sometimes you need to configure the index beyond default parameters: +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 102) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 103) - Index Types: +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 104) - `IVF_HNSW_FLAT`: highest recall, with no vector quantization +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 105) - `IVF_HNSW_SQ`: best recall/latency trade-off +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 106) - `IVF_RQ`: best compression for large, high-dimensional datasets +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 107) - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 108) - `metrics`: default is `l2`, other available are `cosine` or `dot` +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 109) - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar) +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 110) - `num_partitions`: use index-specific starting points from the section above: +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 111) - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576` +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 112) - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096` +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 113) - `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 114) count from a target number of rows per partition. If you set both `num_partitions` and +25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 115) `target_partition_size`, `num_partitions` takes precedence. +521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 116) - `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 117) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 118) Let's take a look at a sample request for an IVF index: +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 119) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 120) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 121) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 122) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 123) {VectorIndexConfigureIvf} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 124) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 125) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 126) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 127) ### 1. Setup +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 128) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 129) Connect to LanceDB and open the table you want to index. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 130) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 131) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 132) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 133) {VectorIndexSetup} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 134) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 135) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 136) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 137) ### 2. Construct an IVF Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 138) +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 139) Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 140) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 141) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 142) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 143) {VectorIndexBuildIvf} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 144) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 145) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 146) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 147) #### Indexing nested vector fields +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 148) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 149) If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`: +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 150) +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 151) +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 152) +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 153) {VectorIndexNestedField} +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 154) +a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 155) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 156) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 157) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 158) Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 159) +6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 160) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 161) ### Async API and Config Objects +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 162) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 163) With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 164) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 165) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 166) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 167) {VectorIndexAsyncConfig} +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 168) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 169) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 170) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 171) Use these Python config classes for the index types shown on this page: +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 172) +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 173) | Index type | Python config class | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 174) | :--- | :--- | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 175) | `IVF_FLAT` | `IvfFlat` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 176) | `IVF_PQ` | `IvfPq` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 177) | `IVF_RQ` | `IvfRq` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 178) | `IVF_SQ` | `IvfSq` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 179) | `IVF_HNSW_FLAT` | `IvfHnswFlat` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 180) | `IVF_HNSW_PQ` | `IvfHnswPq` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 181) | `IVF_HNSW_SQ` | `IvfHnswSq` | +2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 182) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 183) ### 3. Query the IVF Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 184) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 185) Search using a random 1,536-dimensional embedding. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 186) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 187) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 188) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 189) {VectorIndexQueryIvf} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 190) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 191) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 192) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 193) #### Search Configuration +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 194) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 195) Core knobs available on a vector search call: +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 196) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 197) | Parameter | Description | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 198) | :--- | :--- | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 199) | `limit` | Number of results to return (`k`). | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 200) | `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 201) | `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 202) | `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 203) | `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 204) | `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 205) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 206) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 207) **Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 208) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 209) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 210) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 211) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 212) {VectorIndexNprobes} +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 213) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 214) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 215) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 216) Recommended `nprobes` behavior by index type: +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 217) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 218) | Index type | Guidance | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 219) | :--- | :--- | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 220) | `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 221) | `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 222) | `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 223) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 224) #### Advanced Search Controls +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 225) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 226) These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 227) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 228) | Method | Description | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 229) | :--- | :--- | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 230) | `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 231) | `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 232) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 233) **Thresholding with `distance_range`:** +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 234) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 235) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 236) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 237) {VectorIndexDistanceRange} +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 238) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 239) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 240) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 241) **Measuring recall with `bypass_vector_index`:** +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 242) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 243) Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload. +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 244) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 245) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 246) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 247) {VectorIndexBypassRecall} +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 248) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 249) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 250) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 251) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 252) Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries. +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 253) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 254) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 255) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 256) Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 257) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 258) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 259) ## Example: Construct an HNSW Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 260) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 261) ### Index Configuration +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 262) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 263) There are four key parameters to set when constructing an HNSW index: +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 264) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 265) - `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 266) - `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 267) - `m`: The number of neighbors to select for each vector in the HNSW graph. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 268) - `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 269) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 270) ### 1. Construct an HNSW Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 271) +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 272) The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. +5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 273) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 274) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 275) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 276) {VectorIndexBuildHnsw} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 277) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 278) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 279) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 280) ### 2. Query the HNSW Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 281) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 282) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 283) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 284) {VectorIndexQueryHnsw} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 285) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 286) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 287) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 288) ## Example: Construct a Binary Vector Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 289) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 290) Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 291) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 292) ### Index Configuration +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 293) +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 294) - Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 295) - Index Type: `IVF_FLAT` is used for indexing binary vectors +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 296) - `metric`: the `hamming` distance is used for similarity search +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 297) - The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16. +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 298) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 299) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 300) **`IVF_FLAT` + `hamming` is the only supported path for binary vectors.** +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 301) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 302) - `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns. +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 303) - Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`. +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 304) +6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 305) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 306) ### 1. Create Table and Schema +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 307) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 308) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 309) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 310) {VectorIndexBinarySchema} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 311) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 312) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 313) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 314) ### 2. Generate and Add Data +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 315) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 316) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 317) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 318) {VectorIndexBinaryAddData} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 319) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 320) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 321) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 322) ### 3. Construct the Binary Index +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 323) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 324) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 325) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 326) {VectorIndexBinaryBuildIndex} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 327) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 328) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 329) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 330) ### 4. Vector Search +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 331) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 332) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 333) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 334) {VectorIndexBinarySearch} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 335) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 336) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 337) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 338) ## Check Index Status +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 339) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 340) Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 341) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 342) In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 343) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 344) Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built. +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 345) To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table. +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 346) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 347) Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`): +5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 348) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 349) | Field | What it tells you | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 350) | :--- | :--- | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 351) | `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 352) | `size_bytes` | Total size of the index files on disk | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 353) | `num_segments`, `index_version` | On-disk layout and format version | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 354) | `created_at` | Creation time (ms since the Unix epoch in Node.js) | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 355) | `index_uuid`, `type_url` | Internal identifiers for the index segment | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 356) | `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) | +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 357) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 358) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 359) These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them. +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 360) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 361) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 362) +5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 363) +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 364) {VectorIndexCheckStatus} +250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 365) +9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 366) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 367) +f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 368) ## Custom Index Names +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 369) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 370) The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name. +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 371) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 372) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 373) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 374) {VectorIndexCustomName} +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 375) +7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 376) From 65dc3920d3993eebf55bf5711084117c60e8eb34 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 11:16:42 -0700 Subject: [PATCH 08/44] test --- out.txt | 376 -------------------------------------------------------- 1 file changed, 376 deletions(-) delete mode 100644 out.txt diff --git a/out.txt b/out.txt deleted file mode 100644 index 8fad008..0000000 --- a/out.txt +++ /dev/null @@ -1,376 +0,0 @@ -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) --- -e7716261 docs/indexing/vector-index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 2) title: "Vector Indexes" -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 3) sidebarTitle: "Vector Index" -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 4) description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes." -e7716261 docs/indexing/vector-index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 5) icon: "arrow-up-right-dots" -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 6) --- -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 7) import { -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 8) PyVectorIndexConfigureIvf as VectorIndexConfigureIvf, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 9) PyVectorIndexSetup as VectorIndexSetup, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 10) PyVectorIndexBuildIvf as VectorIndexBuildIvf, -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 11) PyVectorIndexNestedField as VectorIndexNestedField, -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 12) PyVectorIndexAsyncConfig as VectorIndexAsyncConfig, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 13) PyVectorIndexQueryIvf as VectorIndexQueryIvf, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 14) PyVectorIndexBuildHnsw as VectorIndexBuildHnsw, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 15) PyVectorIndexQueryHnsw as VectorIndexQueryHnsw, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 16) PyVectorIndexBinarySchema as VectorIndexBinarySchema, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 17) PyVectorIndexBinaryAddData as VectorIndexBinaryAddData, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 18) PyVectorIndexBinaryBuildIndex as VectorIndexBinaryBuildIndex, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 19) PyVectorIndexBinarySearch as VectorIndexBinarySearch, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 20) PyVectorIndexCheckStatus as VectorIndexCheckStatus, -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 21) PyVectorIndexNprobes as VectorIndexNprobes, -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 22) PyVectorIndexDistanceRange as VectorIndexDistanceRange, -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 23) PyVectorIndexBypassRecall as VectorIndexBypassRecall, -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 24) PyVectorIndexCustomName as VectorIndexCustomName, -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 25) } from '/snippets/indexing.mdx'; -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 26) -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 27) You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 28) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 29) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 30) **IVF + HNSW** -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 31) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 32) In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 33) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 34) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 35) ### Manual Indexing -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 36) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 37) If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 38) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 39) ### Automatic Indexing -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 40) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 41) Enterprise-only -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 42) Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 43) - Infers vector columns from the schema -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 44) - Optimizes the `IVF_PQ` index without manual configuration -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 45) - Automatically manages indexing parameters -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 46) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 47) The default distance is `l2` (Euclidean). -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 48) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 49) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 50) You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 51) Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 52) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 53) -fa69074f docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 54) Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. -fa69074f docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 55) -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 56) Rows appended after an index build remain outside that index until optimization refreshes it. Normal -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 57) search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 58) fallback and searches only indexed rows. -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 59) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 60) ## Choose the Right Index -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 61) -cd7899c6 docs/indexing/vector-index.mdx (jasonz-lance 2026-08-04 10:51:31 -0700 62) Use this table to choose the right index and quantization type for your use case: -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 63) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 64) | If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors | -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 65) | :--- | :--- | :--- | :--- | -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 66) | Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead | -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 67) | Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size | -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 68) | Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size | -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 69) | Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 70) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 71) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 72) If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 73) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 74) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 75) Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 76) For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 77) -18d34126 docs/indexing/vector-index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 78) ### Index Tuning -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 79) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 80) Start with these values, then tune for your workload: -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 81) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 82) - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 83) - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 84) - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 85) - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 86) - `IVF_RQ` -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 87) - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 88) - `IVF_PQ` -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 89) - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 90) - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 91) - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 92) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 93) ## Example: Construct an IVF Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 94) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 95) In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 96) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 97) Make sure you have enough data in your table (at least a few thousand rows) for effective index training. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 98) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 99) ### Index Configuration -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 100) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 101) Sometimes you need to configure the index beyond default parameters: -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 102) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 103) - Index Types: -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 104) - `IVF_HNSW_FLAT`: highest recall, with no vector quantization -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 105) - `IVF_HNSW_SQ`: best recall/latency trade-off -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 106) - `IVF_RQ`: best compression for large, high-dimensional datasets -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 107) - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 108) - `metrics`: default is `l2`, other available are `cosine` or `dot` -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 109) - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar) -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 110) - `num_partitions`: use index-specific starting points from the section above: -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 111) - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576` -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 112) - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096` -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 113) - `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 114) count from a target number of rows per partition. If you set both `num_partitions` and -25d7ccd1 docs/indexing/vector-index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 115) `target_partition_size`, `num_partitions` takes precedence. -521d2fab docs/indexing/vector-index.mdx (BubbleCal 2026-02-14 06:56:35 +0800 116) - `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 117) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 118) Let's take a look at a sample request for an IVF index: -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 119) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 120) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 121) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 122) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 123) {VectorIndexConfigureIvf} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 124) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 125) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 126) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 127) ### 1. Setup -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 128) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 129) Connect to LanceDB and open the table you want to index. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 130) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 131) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 132) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 133) {VectorIndexSetup} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 134) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 135) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 136) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 137) ### 2. Construct an IVF Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 138) -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 139) Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 140) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 141) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 142) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 143) {VectorIndexBuildIvf} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 144) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 145) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 146) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 147) #### Indexing nested vector fields -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 148) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 149) If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`: -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 150) -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 151) -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 152) -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 153) {VectorIndexNestedField} -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 154) -a5448bf4 docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 17:40:19 -0400 155) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 156) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 157) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 158) Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 159) -6b265a8f docs/indexing/vector-index.mdx (mintlify[bot] 2026-05-29 13:49:14 -0400 160) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 161) ### Async API and Config Objects -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 162) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 163) With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 164) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 165) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 166) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 167) {VectorIndexAsyncConfig} -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 168) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 169) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 170) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 171) Use these Python config classes for the index types shown on this page: -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 172) -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 173) | Index type | Python config class | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 174) | :--- | :--- | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 175) | `IVF_FLAT` | `IvfFlat` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 176) | `IVF_PQ` | `IvfPq` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 177) | `IVF_RQ` | `IvfRq` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 178) | `IVF_SQ` | `IvfSq` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 179) | `IVF_HNSW_FLAT` | `IvfHnswFlat` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 180) | `IVF_HNSW_PQ` | `IvfHnswPq` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 181) | `IVF_HNSW_SQ` | `IvfHnswSq` | -2b55c1ee docs/indexing/vector-index.mdx (Prashanth Rao 2026-05-12 11:57:38 -0700 182) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 183) ### 3. Query the IVF Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 184) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 185) Search using a random 1,536-dimensional embedding. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 186) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 187) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 188) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 189) {VectorIndexQueryIvf} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 190) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 191) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 192) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 193) #### Search Configuration -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 194) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 195) Core knobs available on a vector search call: -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 196) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 197) | Parameter | Description | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 198) | :--- | :--- | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 199) | `limit` | Number of results to return (`k`). | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 200) | `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 201) | `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 202) | `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 203) | `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 204) | `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 205) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 206) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 207) **Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 208) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 209) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 210) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 211) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 212) {VectorIndexNprobes} -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 213) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 214) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 215) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 216) Recommended `nprobes` behavior by index type: -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 217) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 218) | Index type | Guidance | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 219) | :--- | :--- | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 220) | `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 221) | `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 222) | `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 223) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 224) #### Advanced Search Controls -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 225) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 226) These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 227) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 228) | Method | Description | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 229) | :--- | :--- | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 230) | `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 231) | `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 232) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 233) **Thresholding with `distance_range`:** -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 234) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 235) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 236) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 237) {VectorIndexDistanceRange} -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 238) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 239) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 240) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 241) **Measuring recall with `bypass_vector_index`:** -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 242) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 243) Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload. -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 244) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 245) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 246) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 247) {VectorIndexBypassRecall} -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 248) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 249) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 250) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 251) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 252) Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries. -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 253) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 254) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 255) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 256) Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 257) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 258) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 259) ## Example: Construct an HNSW Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 260) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 261) ### Index Configuration -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 262) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 263) There are four key parameters to set when constructing an HNSW index: -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 264) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 265) - `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 266) - `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 267) - `m`: The number of neighbors to select for each vector in the HNSW graph. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 268) - `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 269) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 270) ### 1. Construct an HNSW Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 271) -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 272) The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. -5f8fe230 docs/indexing/vector-index.mdx (BubbleCal 2026-04-15 13:45:43 +0800 273) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 274) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 275) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 276) {VectorIndexBuildHnsw} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 277) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 278) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 279) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 280) ### 2. Query the HNSW Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 281) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 282) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 283) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 284) {VectorIndexQueryHnsw} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 285) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 286) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 287) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 288) ## Example: Construct a Binary Vector Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 289) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 290) Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 291) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 292) ### Index Configuration -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 293) -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 294) - Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 295) - Index Type: `IVF_FLAT` is used for indexing binary vectors -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 296) - `metric`: the `hamming` distance is used for similarity search -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 297) - The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16. -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 298) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 299) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 300) **`IVF_FLAT` + `hamming` is the only supported path for binary vectors.** -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 301) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 302) - `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns. -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 303) - Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`. -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 304) -6c0ccc00 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 16:32:32 +0800 305) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 306) ### 1. Create Table and Schema -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 307) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 308) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 309) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 310) {VectorIndexBinarySchema} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 311) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 312) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 313) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 314) ### 2. Generate and Add Data -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 315) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 316) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 317) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 318) {VectorIndexBinaryAddData} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 319) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 320) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 321) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 322) ### 3. Construct the Binary Index -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 323) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 324) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 325) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 326) {VectorIndexBinaryBuildIndex} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 327) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 328) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 329) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 330) ### 4. Vector Search -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 331) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 332) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 333) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 334) {VectorIndexBinarySearch} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 335) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 336) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 337) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 338) ## Check Index Status -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 339) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 340) Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 341) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 342) In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 343) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 344) Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built. -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 345) To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table. -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 346) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 347) Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`): -5918c400 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 348) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 349) | Field | What it tells you | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 350) | :--- | :--- | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 351) | `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 352) | `size_bytes` | Total size of the index files on disk | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 353) | `num_segments`, `index_version` | On-disk layout and format version | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 354) | `created_at` | Creation time (ms since the Unix epoch in Node.js) | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 355) | `index_uuid`, `type_url` | Internal identifiers for the index segment | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 356) | `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) | -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 357) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 358) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 359) These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them. -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 360) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 361) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 362) -5cbe87c7 docs/user-guides/indexing/vector-index.mdx (Prashanth Rao 2025-12-09 22:08:27 -0500 363) -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 364) {VectorIndexCheckStatus} -250d9baa docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-12-05 14:12:10 -0800 365) -9f362963 docs/user-guides/indexing/vector-index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 366) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 367) -f4100554 docs/indexing/vector-index.mdx (mintlify[bot] 2026-07-02 14:37:24 -0400 368) ## Custom Index Names -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 369) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 370) The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name. -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 371) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 372) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 373) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 374) {VectorIndexCustomName} -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 375) -7d261e76 docs/indexing/vector-index.mdx (Prashanth Rao 2026-04-21 15:01:11 +0800 376) From 9a5713516a53b712f2b21c2e564426fd607aa15c Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 14:00:05 -0700 Subject: [PATCH 09/44] reorganized stuff --- docs/indexing/index.mdx | 137 ++------------------- docs/indexing/vector-index.mdx | 219 ++++++++++++++++++++++++++------- out.txt | 172 ++++++++++++++++++++++++++ 3 files changed, 356 insertions(+), 172 deletions(-) create mode 100644 out.txt diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 5ef8bfa..2221404 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -1,15 +1,15 @@ --- title: "Indexing Data" sidebarTitle: "Overview" -description: "Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search." +description: "Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more." icon: "list" --- An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads: -- **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors +- **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings) - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases -- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices) +- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g. timestamps, prices) Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: @@ -20,9 +20,9 @@ Scalar indices serve as a foundational optimization layer, accelerating filterin - Key-value lookups (enabling rapid primary key-based retrievals) -## Supported Index Types +## Supported Indexes -LanceDB provides a comprehensive suite of indexes for different data types and use cases: +LanceDB provides a comprehensive suite of indexes for different use cases and data types: | Index | Use Case | Description | | :--------- | :------- | :---------- | @@ -38,19 +38,10 @@ LanceDB provides a comprehensive suite of indexes for different data types and u TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). - -**Operational checks** +## Quantization +todo vv quantization should be a hyperlink -For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. -`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. - -Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to -`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. - - -### Quantization Types - -Vector indexes use different quantization methods to compress vectors and improve search performance: +LanceDB also supports several different quantization methods, used by vector indexes to compress vectors and improve search performance: | Quantization | Use Case | Description | | :----------- | :------- | :---------- | @@ -58,115 +49,3 @@ Vector indexes use different quantization methods to compress vectors and improv | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | - -## Understanding the IVF-PQ Index - -An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. -It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. - -LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. - -## IVF-PQ - -LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. -The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. - -### Product Quantization - -Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. - -Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. -The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. - -![](/static/assets/images/indexing/ivfpq_pq_desc.png) - -As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. - - -Original storage: `128 × 32 = 4096` bits. -Quantized storage: `4 × 8 = 32` bits. - -In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. - - -It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. - -### Inverted File Index (IVF) Implementation - -(note: acknowledge pq in here somewhere) - -An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. - -Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. -The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. -The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. - -The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. -![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) - -To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ -for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. -Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. - -here vv -During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. -![](/static/assets/images/indexing/ivfpq_query_vector.webp) - -## HNSW Index Implementation - -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. - -### Types of ANN Search Algorithms - -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. -For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. - -There are three main types of ANN search algorithms: - -* **Tree-based search algorithms**: Use a tree structure to organize and store data points. -* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. -* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. - -HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. -HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. - -### Understanding $k$-Nearest Neighbor Graphs - -The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: - -* Each vector in the dataset is given an associated vertex. -* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. -* For some applications (including nearest-neighbor search), the incoming edges are also added. - -Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: - -* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). -* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. -* Repeat until a local optimum is found. - -The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. - -Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. -In fact, another data structure is not needed: This can be done "incrementally". -That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. - -One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. - -### Hierarchical Navigable Small Worlds (HNSW) - -HNSW builds on k-ANN in two main ways: - -* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. -* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. - -This recursive structure can be thought of as separating into layers: - -* At the bottom-most layer, a k-ANN graph on the whole dataset is present. -* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. -* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. - -Then the greedy search routine operates as follows: - -* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. -* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. -* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 6f4f98a..539001c 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -32,32 +32,7 @@ You can create and manage multiple vector indexes on any Lance dataset. LanceDB In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. -### Manual Indexing - -If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process. - -### Automatic Indexing - - Enterprise-only -Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: -- Infers vector columns from the schema -- Optimizes the `IVF_PQ` index without manual configuration -- Automatically manages indexing parameters -The default distance is `l2` (Euclidean). - - -You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. -Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter. - - -Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. - -Rows appended after an index build remain outside that index until optimization refreshes it. Normal -search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that -fallback and searches only indexed rows. - -## Choose the Right Index Use this table to choose the right index and quantization type for your use case: @@ -90,13 +65,171 @@ Start with these values, then tune for your workload: - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. -## Example: Construct an IVF Index + + + +**Operational checks** + +For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. +`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. + +Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to +`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. + + + +## Understanding Vector Indexes + +An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. +It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. + +LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. + +### IVF-PQ + +LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. +The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. + +#### Product Quantization + +Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. + +Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. +The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. + +![](/static/assets/images/indexing/ivfpq_pq_desc.png) + +As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. + + +Original storage: `128 × 32 = 4096` bits. +Quantized storage: `4 × 8 = 32` bits. + +In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. + + +It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. + +#### Inverted File Index (IVF) Implementation + +(note: acknowledge pq in here somewhere) + +An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. + +Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. +The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. +The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. + +The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. +![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) + +To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ +for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. +Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. + +here vv +During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. +![](/static/assets/images/indexing/ivfpq_query_vector.webp) + +### HNSW + +Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. + +#### Types of ANN Search Algorithms + +Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. +For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. + +There are three main types of ANN search algorithms: + +* **Tree-based search algorithms**: Use a tree structure to organize and store data points. +* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. +* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. + +HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. +HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. + +#### Understanding $k$-Nearest Neighbor Graphs + +The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: + +* Each vector in the dataset is given an associated vertex. +* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. +* For some applications (including nearest-neighbor search), the incoming edges are also added. + +Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: + +* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). +* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. +* Repeat until a local optimum is found. + +The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. + +Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. +In fact, another data structure is not needed: This can be done "incrementally". +That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. + +One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. + +#### Hierarchical Navigable Small Worlds (HNSW) + +HNSW builds on k-ANN in two main ways: + +* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. +* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. + +This recursive structure can be thought of as separating into layers: + +* At the bottom-most layer, a k-ANN graph on the whole dataset is present. +* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. +* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. + +Then the greedy search routine operates as follows: + +* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. +* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. +* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). + + + + + + +## Using Vector Indexes + + +### Manual Indexing + +If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process. + +### Automatic Indexing + + Enterprise-only +Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: +- Infers vector columns from the schema +- Optimizes the `IVF_PQ` index without manual configuration +- Automatically manages indexing parameters + +The default distance is `l2` (Euclidean). + + +You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. +Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter. + + +Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. + +Rows appended after an index build remain outside that index until optimization refreshes it. Normal +search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that +fallback and searches only indexed rows. + +### Example: Construct an IVF Index In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. Make sure you have enough data in your table (at least a few thousand rows) for effective index training. -### Index Configuration +#### Index Configuration Sometimes you need to configure the index beyond default parameters: @@ -124,7 +257,7 @@ Let's take a look at a sample request for an IVF index: -### 1. Setup +#### 1. Setup Connect to LanceDB and open the table you want to index. @@ -134,7 +267,7 @@ Connect to LanceDB and open the table you want to index. -### 2. Construct an IVF Index +#### 2. Construct an IVF Index Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. @@ -158,7 +291,7 @@ If your vector column lives inside a struct, pass its full dotted path as `vecto Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. -### Async API and Config Objects +#### Async API and Config Objects With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: @@ -180,7 +313,7 @@ Use these Python config classes for the index types shown on this page: | `IVF_HNSW_PQ` | `IvfHnswPq` | | `IVF_HNSW_SQ` | `IvfHnswSq` | -### 3. Query the IVF Index +#### 3. Query the IVF Index Search using a random 1,536-dimensional embedding. @@ -256,9 +389,9 @@ Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall mea Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. -## Example: Construct an HNSW Index +### Example: Construct an HNSW Index -### Index Configuration +#### Index Configuration There are four key parameters to set when constructing an HNSW index: @@ -267,7 +400,7 @@ There are four key parameters to set when constructing an HNSW index: - `m`: The number of neighbors to select for each vector in the HNSW graph. - `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. -### 1. Construct an HNSW Index +#### 1. Construct an HNSW Index The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. @@ -277,7 +410,7 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang -### 2. Query the HNSW Index +#### 2. Query the HNSW Index @@ -285,11 +418,11 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang -## Example: Construct a Binary Vector Index +### Example: Construct a Binary Vector Index Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits. -### Index Configuration +#### Index Configuration - Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. - Index Type: `IVF_FLAT` is used for indexing binary vectors @@ -303,7 +436,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena - Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`. -### 1. Create Table and Schema +#### 1. Create Table and Schema @@ -311,7 +444,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -### 2. Generate and Add Data +#### 2. Generate and Add Data @@ -319,7 +452,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -### 3. Construct the Binary Index +#### 3. Construct the Binary Index @@ -327,7 +460,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -### 4. Vector Search +#### 4. Vector Search @@ -335,7 +468,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -## Check Index Status +### Check Index Status Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. @@ -365,7 +498,7 @@ These fields are populated for local and embedded tables. On LanceDB Enterprise -## Custom Index Names +### Custom Index Names The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name. diff --git a/out.txt b/out.txt new file mode 100644 index 0000000..f0973e4 --- /dev/null +++ b/out.txt @@ -0,0 +1,172 @@ +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) --- +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 2) title: "Indexing Data" +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 3) sidebarTitle: "Overview" +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 4) description: "Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search." +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 5) icon: "list" +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 6) --- +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 7) +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 8) An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads: +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 9) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 10) - **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 11) - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 12) - **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 13) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 14) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 15) Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 16) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 17) - Vector search (prefilter or post-filter results using metadata) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 18) - Full-text search (combining keyword matching with structured filters) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 19) - SQL scans (optimizing WHERE clauses on scalar columns) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 20) - Key-value lookups (enabling rapid primary key-based retrievals) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 21) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 22) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 23) ## Supported Index Types +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 24) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 25) LanceDB provides a comprehensive suite of indexes for different data types and use cases: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 26) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 27) | Index | Use Case | Description | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 28) | :--------- | :------- | :---------- | +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 29) | `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: $\ell_2$ `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 30) | `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: $\ell_2$ `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| +18d34126 docs/indexing/index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 31) | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | +25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 32) | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 33) | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | +25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 34) | `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | +25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 35) | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 36) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 37) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 38) TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 39) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 40) +fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 41) +fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 42) **Operational checks** +fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 43) +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 44) For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 45) `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. +25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 46) +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 47) Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 48) `BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. +fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 49) +fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 50) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 51) ### Quantization Types +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 52) +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 53) Vector indexes use different quantization methods to compress vectors and improve search performance: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 54) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 55) | Quantization | Use Case | Description | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 56) | :----------- | :------- | :---------- | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 57) | `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 58) | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | +5a45fbe7 docs/indexing/index.mdx (BubbleCal 2026-02-28 07:36:05 +0800 59) | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 60) | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 61) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 62) ## Understanding the IVF-PQ Index +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 63) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 64) An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 65) It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 66) +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 67) LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 68) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 69) ## IVF-PQ +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 70) +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 71) LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 72) The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 73) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 74) ### Product Quantization +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 75) +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 76) Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 77) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 78) Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. +220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 79) The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 80) +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 81) ![](/static/assets/images/indexing/ivfpq_pq_desc.png) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 82) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 83) As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 84) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 85) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 86) Original storage: `128 × 32 = 4096` bits. +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 87) Quantized storage: `4 × 8 = 32` bits. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 88) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 89) In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 90) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 91) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 92) It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 93) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 94) ### Inverted File Index (IVF) Implementation +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 95) +86d2e2d3 docs/indexing/index.mdx (jasonz-lance 2026-08-04 11:15:22 -0700 96) (note: acknowledge pq in here somewhere) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 97) +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 98) An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 99) +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 100) Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 101) The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 102) The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 103) +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 104) The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 105) ![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 106) +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 107) To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 108) for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 109) Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 110) +bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 111) here vv +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 112) During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. +e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 113) ![](/static/assets/images/indexing/ivfpq_query_vector.webp) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 114) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 115) ## HNSW Index Implementation +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 116) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 117) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 118) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 119) ### Types of ANN Search Algorithms +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 120) +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 121) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. +d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 122) For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 123) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 124) There are three main types of ANN search algorithms: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 125) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 126) * **Tree-based search algorithms**: Use a tree structure to organize and store data points. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 127) * **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 128) * **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 129) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 130) HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 131) HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 132) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 133) ### Understanding $k$-Nearest Neighbor Graphs +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 134) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 135) The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 136) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 137) * Each vector in the dataset is given an associated vertex. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 138) * Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 139) * For some applications (including nearest-neighbor search), the incoming edges are also added. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 140) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 141) Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 142) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 143) * Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 144) * Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 145) * Repeat until a local optimum is found. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 146) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 147) The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 148) +119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 149) Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 150) In fact, another data structure is not needed: This can be done "incrementally". +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 151) That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 152) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 153) One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 154) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 155) ### Hierarchical Navigable Small Worlds (HNSW) +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 156) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 157) HNSW builds on k-ANN in two main ways: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 158) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 159) * Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 160) * The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 161) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 162) This recursive structure can be thought of as separating into layers: +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 163) +74a79fe3 docs/indexing/index.mdx (Justin Miller 2026-04-09 12:09:35 -0700 164) * At the bottom-most layer, a k-ANN graph on the whole dataset is present. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 165) * At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 166) * At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 167) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 168) Then the greedy search routine operates as follows: +9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 169) +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 170) * At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 171) * Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. +5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 172) * Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). From 70c9c4cbe7ba2bef77d2b1fc34cee4b373fd4c8c Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 14:25:17 -0700 Subject: [PATCH 10/44] stuff --- docs/indexing/index.mdx | 4 +--- docs/indexing/quantization.mdx | 33 +++++++++++++++++++++++++---- docs/indexing/scalar-index.mdx | 9 ++++++++ docs/indexing/vector-index.mdx | 38 ++++++---------------------------- 4 files changed, 45 insertions(+), 39 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 2221404..42bfab4 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -39,9 +39,7 @@ TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). ## Quantization -todo vv quantization should be a hyperlink - -LanceDB also supports several different quantization methods, used by vector indexes to compress vectors and improve search performance: +LanceDB also supports several different [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and improve search performance: | Quantization | Use Case | Description | | :----------- | :------- | :---------- | diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 307b9b2..b165e7e 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -1,7 +1,7 @@ --- title: "Quantization" sidebarTitle: "Quantization" -description: "Learn about quantization in a LanceDB index." +description: "Use quantization to improve storage requirements and query latency of your LanceDB vector index." icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- @@ -23,11 +23,34 @@ todo decide what to do here vv Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. -## RaBitQ quantization +##Quantization Techniques +#### Product Quantization + +Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. + +Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. +The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. + +![](/static/assets/images/indexing/ivfpq_pq_desc.png) + +As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. + + +Original storage: `128 × 32 = 4096` bits. +Quantized storage: `4 × 8 = 32` bits. + +In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. + + +It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. + + + +### RaBitQ quantization RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall. -### How RaBitQ works +#### How RaBitQ works - Embeddings are grouped around centroids (as in other IVF indexes). - Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere. @@ -43,7 +66,7 @@ Compared to `IVF_PQ`, RaBitQ: For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/). -### Using RaBitQ +#### Using RaBitQ You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`. @@ -60,6 +83,8 @@ It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions. +###SQ (todo?) + ## API Reference The full list of parameters to the algorithm are listed below. diff --git a/docs/indexing/scalar-index.mdx b/docs/indexing/scalar-index.mdx index 7e71459..a417e05 100644 --- a/docs/indexing/scalar-index.mdx +++ b/docs/indexing/scalar-index.mdx @@ -27,6 +27,15 @@ LanceDB supports four types of scalar indexes: - `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries. - `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`. + +Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: + +- Vector search (prefilter or post-filter results using metadata) +- Full-text search (combining keyword matching with structured filters) +- SQL scans (optimizing WHERE clauses on scalar columns) +- Key-value lookups (enabling rapid primary key-based retrievals) + + ## Choosing the Right Index Type | Data Type | Filter | Index Type | diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 539001c..2d8a761 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -85,36 +85,12 @@ It greatly improves upon the runtime of a brute-force kNN search, while admittin LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. -### IVF-PQ +### Inverted File Index (IVF) and IVF-PQ -LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. -The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. +LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently +compress embeddings. We primarily discuss the IVF indexing technique here. -#### Product Quantization - -Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. - -Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. -The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. - -![](/static/assets/images/indexing/ivfpq_pq_desc.png) - -As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. - - -Original storage: `128 × 32 = 4096` bits. -Quantized storage: `4 × 8 = 32` bits. - -In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. - - -It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. - -#### Inverted File Index (IVF) Implementation - -(note: acknowledge pq in here somewhere) - -An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. +An IVF index facilitates rapid nearest neighbor searches by drastically reducing the search space. Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. @@ -131,7 +107,7 @@ here vv During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. ![](/static/assets/images/indexing/ivfpq_query_vector.webp) -### HNSW +### Hierarchical Navigable Small World (HNSW) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. @@ -193,8 +169,6 @@ Then the greedy search routine operates as follows: - - ## Using Vector Indexes @@ -210,7 +184,7 @@ Vector indexing is managed **automatically** in LanceDB Enterprise. When a table - Optimizes the `IVF_PQ` index without manual configuration - Automatically manages indexing parameters -The default distance is `l2` (Euclidean). +The default distance is `l2` (the Euclidean $\ell_2$ norm). You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. From 178949a1b654ac98aa1acb991ecdc93f24ead33e Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 4 Aug 2026 16:00:59 -0700 Subject: [PATCH 11/44] d --- docs/indexing/index.mdx | 9 +-------- docs/indexing/quantization.mdx | 8 ++++---- 2 files changed, 5 insertions(+), 12 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 42bfab4..46e1133 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -11,14 +11,7 @@ An **index** is a data structure that facilitates efficient scans and lookups on - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases - **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g. timestamps, prices) - -Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: - -- Vector search (prefilter or post-filter results using metadata) -- Full-text search (combining keyword matching with structured filters) -- SQL scans (optimizing WHERE clauses on scalar columns) -- Key-value lookups (enabling rapid primary key-based retrievals) - + ## Supported Indexes diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index b165e7e..146b5b6 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -23,8 +23,8 @@ todo decide what to do here vv Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. -##Quantization Techniques -#### Product Quantization +## Quantization Techniques +### Product Quantization Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. @@ -83,14 +83,14 @@ It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions. -###SQ (todo?) +### SQ (todo?) ## API Reference The full list of parameters to the algorithm are listed below. - `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2" - The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product. + The distance metric used in comparison. - `num_partitions`: Optional[int], defaults to None Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count. - `num_bits`: int, defaults to 1 From c19f0c62f9270cc8e868a1bc2a425ab3c34819cc Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 17:10:12 -0700 Subject: [PATCH 12/44] lots of changes to vector-index.mdx, finished ivfpq section --- docs/indexing/index.mdx | 19 +-- docs/indexing/vector-index.mdx | 208 +++++++++++++++++---------------- docs/search/vector-search.mdx | 8 +- 3 files changed, 120 insertions(+), 115 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 46e1133..5feebf8 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -5,17 +5,14 @@ description: "Optimize search performance with LanceDB using vector indexes, ful icon: "list" --- -An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads: - -- **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings) -- **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases -- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g. timestamps, prices) - - +An **index** is a data structure that facilitates efficient scans and lookups on an embedded dataset. ## Supported Indexes +LanceDB provides a comprehensive suite of indexes to optimize performance across different use cases and data types: -LanceDB provides a comprehensive suite of indexes for different use cases and data types: +- **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings) +- **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases +- **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data | Index | Use Case | Description | | :--------- | :------- | :---------- | @@ -27,12 +24,8 @@ LanceDB provides a comprehensive suite of indexes for different use cases and da | `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | - -TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). - - ## Quantization -LanceDB also supports several different [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and improve search performance: +LanceDB also supports several different [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and reduce storage requirements: | Quantization | Use Case | Description | | :----------- | :------- | :---------- | diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 2d8a761..6adb857 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -1,7 +1,7 @@ --- title: "Vector Indexes" sidebarTitle: "Vector Index" -description: "Build and optimize LanceDB vector indexes, including IVF, HNSW and binary quantized indexes." +description: "Build and manage LanceDB vector indexes, including IVF, HNSW and binary quantized indexes." icon: "arrow-up-right-dots" --- import { @@ -26,6 +26,8 @@ import { You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. +(todo intro sentence) (todo up here: introduce/define kNN, ANN) + **IVF + HNSW** @@ -33,6 +35,9 @@ In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's avail +## Understanding Vector Indexes + +### Choosing the Right Index Use this table to choose the right index and quantization type for your use case: @@ -50,82 +55,45 @@ If your vector search frequently includes metadata filters (`where(...)`), prefe Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration. For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`. -### Index Tuning - -Start with these values, then tune for your workload: - -- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) - - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) - - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. - - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. -- `IVF_RQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. -- `IVF_PQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). - - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. - - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. - - - - -**Operational checks** - -For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. -`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. - -Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to -`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. - -## Understanding Vector Indexes - An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. -LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. -### Inverted File Index (IVF) and IVF-PQ +### IVF-PQ -LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently -compress embeddings. We primarily discuss the IVF indexing technique here. +The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently +compress embeddings. We discuss the indexing techniques and algorithms here, deferring discussion of quantization to a later section. For examples of using LanceDB vector indexes, see [Using Vector Indexes](#using-vector-indexes). -An IVF index facilitates rapid nearest neighbor searches by drastically reducing the search space. +An IVF index is a data structure that aims to accelerate nearest neighbor searches by drastically reducing the search space. +Given a large set of stored vectors, the LanceDB indexer first computes a set of *centroids* corresponding to an approximate solution to the [$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. -Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. -The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. - -The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. +Each centroid is then identified with its corresponding set of vectors, called its *cluster* or *partition*. +The following image shows an example geometric space partitioned according to this algorithm. The colored marks denote centroids. ![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) -To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ -for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. -Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. +Once constructed, an IVF index admits efficient ANN queries. Instead of a brute-force comparison of a queried vector to every stored vector, we can now search the smaller set of *centroids* +for a closest match, then execute a brute-force comparison against its cluster. This technique quickly eliminates most clusters from the search space with a single comparison. +Furthermore, since each centroid is close to points in its cluster, we are likely to receive an approximately correct result. -here vv -During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. +As an edge case, observe that a queried vector may lie near the boundary of $2$ or more clusters, as shown in the image below. +In this example, the query's true nearest neighbors may lie scattered across several different clusters; thus, our algorithm, which searches only one such cluster, may return rather +imprecise results. To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched. A query with higher `nprobe` will require more runtime but yield more accurate results. ![](/static/assets/images/indexing/ivfpq_query_vector.webp) -### Hierarchical Navigable Small World (HNSW) +### HNSW -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. - -#### Types of ANN Search Algorithms +The **Hierarchical Navigable Small World (HNSW)** index ... (todo) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. -There are three main types of ANN search algorithms: - -* **Tree-based search algorithms**: Use a tree structure to organize and store data points. -* **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. -* **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. - HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. -#### Understanding $k$-Nearest Neighbor Graphs + Understanding $k$-Nearest Neighbor Graphs The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: @@ -147,7 +115,7 @@ That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. -#### Hierarchical Navigable Small Worlds (HNSW) + Hierarchical Navigable Small Worlds (HNSW) HNSW builds on k-ANN in two main ways: @@ -170,40 +138,15 @@ Then the greedy search routine operates as follows: ## Using Vector Indexes - - -### Manual Indexing - -If using LanceDB OSS, you will have to create the vector index manually, by calling `table.create_index()`, and updating the index as new data arrives and tuning its parameters is also a manual process. - -### Automatic Indexing - - Enterprise-only -Vector indexing is managed **automatically** in LanceDB Enterprise. When a table is created in LanceDB Enterprise, the system asynchronously updates and optimizes the index as a background process: -- Infers vector columns from the schema -- Optimizes the `IVF_PQ` index without manual configuration -- Automatically manages indexing parameters - -The default distance is `l2` (the Euclidean $\ell_2$ norm). - - -You can call `create_index()` with different parameters to create a new index -- this replaces any existing index. -Although the `create_index` API returns immediately, the building of the vector index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter. - - -Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`; it can time out if new writes keep arriving. - -Rows appended after an index build remain outside that index until optimization refreshes it. Normal -search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that -fallback and searches only indexed rows. - + (todo introduce) + ### Example: Construct an IVF Index In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. Make sure you have enough data in your table (at least a few thousand rows) for effective index training. -#### Index Configuration + Index Configuration Sometimes you need to configure the index beyond default parameters: @@ -231,7 +174,7 @@ Let's take a look at a sample request for an IVF index: -#### 1. Setup + 1. Setup Connect to LanceDB and open the table you want to index. @@ -241,7 +184,7 @@ Connect to LanceDB and open the table you want to index. -#### 2. Construct an IVF Index + 2. Construct an IVF Index Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. @@ -251,7 +194,7 @@ Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` -#### Indexing nested vector fields + Indexing nested vector fields If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`: @@ -265,7 +208,7 @@ If your vector column lives inside a struct, pass its full dotted path as `vecto Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. -#### Async API and Config Objects + Async API and Config Objects With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: @@ -287,7 +230,7 @@ Use these Python config classes for the index types shown on this page: | `IVF_HNSW_PQ` | `IvfHnswPq` | | `IVF_HNSW_SQ` | `IvfHnswSq` | -#### 3. Query the IVF Index + 3. Query the IVF Index Search using a random 1,536-dimensional embedding. @@ -297,7 +240,7 @@ Search using a random 1,536-dimensional embedding. -#### Search Configuration + Search Configuration Core knobs available on a vector search call: @@ -328,7 +271,7 @@ Recommended `nprobes` behavior by index type: | `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | | `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | -#### Advanced Search Controls + Advanced Search Controls These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. @@ -365,7 +308,7 @@ Multivector indexing currently requires `distance_type="cosine"` — `l2` is rej ### Example: Construct an HNSW Index -#### Index Configuration + Index Configuration There are four key parameters to set when constructing an HNSW index: @@ -374,7 +317,7 @@ There are four key parameters to set when constructing an HNSW index: - `m`: The number of neighbors to select for each vector in the HNSW graph. - `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. -#### 1. Construct an HNSW Index + 1. Construct an HNSW Index The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. @@ -384,7 +327,7 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang -#### 2. Query the HNSW Index + 2. Query the HNSW Index @@ -396,7 +339,7 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits. -#### Index Configuration + Index Configuration - Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. - Index Type: `IVF_FLAT` is used for indexing binary vectors @@ -410,7 +353,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena - Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`. -#### 1. Create Table and Schema + 1. Create Table and Schema @@ -418,7 +361,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -#### 2. Generate and Add Data + 2. Generate and Add Data @@ -426,7 +369,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -#### 3. Construct the Binary Index + 3. Construct the Binary Index @@ -434,7 +377,7 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena -#### 4. Vector Search + 4. Vector Search @@ -442,6 +385,55 @@ Binary vectors are useful for hash-based retrieval, fingerprinting, or any scena + + +### Manual and Automatic Indexing + + Enterprise-only +In LanceDB Enterprise, vector indexes are managed **automatically**. The system asynchronously updates and optimizes indexes as a background process: +- Automatically manages indexing parameters +- Optimizes `IVF_PQ` storage without manual input +- Infers vector columns from the schema + + Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. + LanceDB also provides an interface to tune parameters manually as data changes. The following default parameters are recommended: + +- `IVF_PQ` + - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). + - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. + - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. +- `IVF_RQ` + - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. +- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) + - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) + - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. + - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. + + +You can call `create_index()` to create a new index (this replaces existing indexes). +`create_index()` returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the `wait_timeout` parameter. + + + +Be careful to use the same distance metric during index creation and search (`l2` by default). + + + +If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. +It can time out if new writes keep arriving. + + + +Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal +search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that +fallback and searches only indexed rows. + + + + + +## Misc(todo choose name) + ### Check Index Status Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. @@ -481,3 +473,23 @@ The `{column}_idx` suffix is a default convention, not the only supported naming {VectorIndexCustomName} + + +**Operational checks** + +For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. +`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. + +Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to +`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. + + + + +## Temp (scrap) + +LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. + + +TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). + \ No newline at end of file diff --git a/docs/search/vector-search.mdx b/docs/search/vector-search.mdx index 06381df..a86276f 100644 --- a/docs/search/vector-search.mdx +++ b/docs/search/vector-search.mdx @@ -60,10 +60,10 @@ The right metric improves both search accuracy and query performance. Currently, | Distance metric | Mathematical form | Notes | |---|---|---| -| `l2` | $\|x-y\|_2=\sqrt{\sum_i (x_i-y_i)^2}$ | Measures the straight-line distance between two points in vector space. Calculated as the square root of the sum of squared differences between corresponding vector components. | -| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Computed as 1 minus cosine similarity (the dot product normalized by both vector magnitudes), so vector length does not affect the score. Use for unnormalized vectors. | -| `dot` | $x\cdot y=\sum_i x_i y_i$ | Calculates the sum of products of corresponding vector components. Provides raw similarity scores without normalization, sensitive to vector magnitudes. Use for normalized vectors for best performance. | -| `hamming` | $\sum_i \mathbf{1}[x_i\neq y_i]$ | Counts the number of positions where corresponding bits differ between binary vectors. Only applicable to binary vectors stored as packed uint8 arrays. | +| `l2` | $$\|x-y\|_2= \sqrt{\sum_i (x_i-y_i)^2}$$ | The $\ell_2$ norm. Measures Euclidean distance between two points in geometric space. | +| `cosine` | $1-\frac{x\cdot y}{\|x\|_2\|y\|_2}$ | Measures directional difference between vectors. Insensitive to nonzero scaling of vectors. | +| `dot` | $x\cdot y= \underset{i}{\sum} x_i y_i$ | The standard dot product of real vectors. Sensitive to nonzero scaling of vectors; use on normalized vectors. | +| `hamming` | $\underset{i}{\sum} \,\: \mathbf{1}_{\, [x_i\neq y_i]}$ | Counts the number of bitwise-differing bits between binary vectors. | For indexed search, supported distance metrics vary by index type: From c43737143fa777b532102a0e9323711e4f34b932 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 20:53:14 -0700 Subject: [PATCH 13/44] tear down hnsw --- docs/indexing/quantization.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 146b5b6..24b2948 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -6,8 +6,8 @@ icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- -Quantization compresses high-dimensional vectors into concise representations that admit efficient storage with only a small compromise in search accuracy. -Quantization is beneficial when your dataset uses high-dimensional vectors ($512, 768, 1024$ or more dimensions), or when index build time and query latency are crucial. +**Quantization** compresses vectors into concise representations that admit efficient storage. +It is beneficial for high ($512, 768, 1024$ or more)-dimensional datasets, or when index build time and query latency are crucial. LanceDB currently exposes several quantized vector index types, with a range of options differing in indexing technique (`IVF_*`,`IVF_HNSW_*`) and method of quantization (`PQ`,`RQ`,`SQ`). - `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples. @@ -17,7 +17,7 @@ LanceDB currently exposes several quantized vector index types, with a range of - `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search. Different indexing and quantization techniques may be better suited for certain use cases. For example, `IVF_PQ` works well in many cases, but RaBitQ (`IVF_RQ`) allows for more aggressive compression. -See the ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. +See the ["Choosing the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. todo decide what to do here vv From 55328b9c4b65252a80a9286480b508024a2452b1 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 20:53:57 -0700 Subject: [PATCH 14/44] tear down hnsw again --- docs/indexing/vector-index.mdx | 36 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 24 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 6adb857..998bfa2 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -85,61 +85,46 @@ imprecise results. To address this, LanceDB exposes the `nprobe` parameter, whic ### HNSW -The **Hierarchical Navigable Small World (HNSW)** index ... (todo) - -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. -For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. - -HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. -HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. - - Understanding $k$-Nearest Neighbor Graphs - -The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: +The **Hierarchical Navigable Small World (HNSW)** index ... (todo. to consider: mention skip list?) +background: kNN graphs: vv * Each vector in the dataset is given an associated vertex. * Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. * For some applications (including nearest-neighbor search), the incoming edges are also added. Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: - * Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). * Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. * Repeat until a local optimum is found. - The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. -Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. -In fact, another data structure is not needed: This can be done "incrementally". -That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. - -One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. - Hierarchical Navigable Small Worlds (HNSW) +computing a kNN graph is slow, so we instead use a k-ANN graph: +In fact, another data structure is not needed: This can be done "incrementally". +(kANN graph is built iteratively on vertices) HNSW builds on k-ANN in two main ways: - * Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. * The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. This recursive structure can be thought of as separating into layers: - * At the bottom-most layer, a k-ANN graph on the whole dataset is present. * At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. * At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. -Then the greedy search routine operates as follows: - +to query: * At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. * Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. * Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). +One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. + ## Using Vector Indexes (todo introduce) - + ### Example: Construct an IVF Index In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. @@ -490,6 +475,9 @@ Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar ind LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. +Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. + + TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). \ No newline at end of file From e0157bb7db98a606581077f72442f78c2d576508 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 21:02:10 -0700 Subject: [PATCH 15/44] gonna rewrite hnsw from scratch --- docs/indexing/vector-index.mdx | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 998bfa2..992fce2 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -100,26 +100,17 @@ The above algorithm also generalizes to e.g. top 10 approximate nearest neighbor computing a kNN graph is slow, so we instead use a k-ANN graph: -In fact, another data structure is not needed: This can be done "incrementally". -(kANN graph is built iteratively on vertices) - -HNSW builds on k-ANN in two main ways: * Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. * The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. This recursive structure can be thought of as separating into layers: -* At the bottom-most layer, a k-ANN graph on the whole dataset is present. -* At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. * At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. to query: -* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. -* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. -* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). - +* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. repeat -One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. +HNSW ends up needing a lot of memory. ## Using Vector Indexes From 151026762354c1ac8e16cc5d58ecccc79c9e6174 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 21:13:44 -0700 Subject: [PATCH 16/44] more changes --- docs/indexing/index.mdx | 4 ++++ docs/indexing/quantization.mdx | 18 +++++++++--------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index 5feebf8..aac426e 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -33,3 +33,7 @@ LanceDB also supports several different [quantization](/indexing/quantization) m | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | + +## temp + +LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 24b2948..e6d10dc 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -89,15 +89,15 @@ Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB The full list of parameters to the algorithm are listed below. -- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2" +- `distance_type`: `Literal["l2", "cosine", "dot"]`. Default: `"l2"` The distance metric used in comparison. -- `num_partitions`: Optional[int], defaults to None - Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count. -- `num_bits`: int, defaults to 1 - Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage. -- `max_iterations`: int, defaults to 50 +- `num_partitions`: `Optional[int]`. Default: `None` + Number of IVF partitions (clusters). High `num_partitions` increases both recall and build time. When unset, LanceDB chooses roughly the square root of the row count. +- `num_bits`: `int`. Default: `1` + Bits per dimension for quantization (`1` corresponds to RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage. +- `max_iterations`: `int`. Default: `50` Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality. -- `sample_rate`: int, defaults to 256 - Number of samples per partition during training. Higher values may improve accuracy but increase training time. -- `target_partition_size`: Optional[int], defaults to None +- `sample_rate`: `int`. Default: `256` + Number of samples per partition during training. Higher values increase both accuracy and training time. +- `target_partition_size`: `Optional[int]`. Default: `None` Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence. From 3bd1b53b1aff8ff74cc1c196ec5d9f2891132656 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 5 Aug 2026 21:44:35 -0700 Subject: [PATCH 17/44] vector-index.mdx introduction part --- docs/indexing/vector-index.mdx | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 992fce2..302eb6a 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -24,22 +24,23 @@ import { PyVectorIndexCustomName as VectorIndexCustomName, } from '/snippets/indexing.mdx'; -You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. -(todo intro sentence) (todo up here: introduce/define kNN, ANN) - - -**IVF + HNSW** - -In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. - +Vector indexes are a robust tool in facilitating fast nearest-neighbor searches across large datasets. +LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two (todo) advanced techniques that, when combined with [quantization](/indexing/quantization), provide large benefits across a variety of use cases. + +This page's discussions assume an ambient dataset of numeric vector types. Vector indexing is not supported for incomparable data types. + ## Understanding Vector Indexes +todo: intro sentence + +You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: +**Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. ### Choosing the Right Index -Use this table to choose the right index and quantization type for your use case: +Use this table to choose the right index and quantization protocol for your use case: | If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors | | :--- | :--- | :--- | :--- | @@ -112,6 +113,15 @@ to query: HNSW ends up needing a lot of memory. + +**IVF + HNSW** + +In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. +What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. +LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. +This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. + + ## Using Vector Indexes (todo introduce) @@ -464,9 +474,7 @@ Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar ind ## Temp (scrap) -LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. -Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. From 0156af654149f9479cbfda54303c168796ee42c2 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Thu, 6 Aug 2026 14:06:30 -0700 Subject: [PATCH 18/44] tear down pq --- docs/indexing/quantization.mdx | 4 ---- docs/indexing/vector-index.mdx | 6 +----- 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index e6d10dc..0cee789 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -26,10 +26,6 @@ Use the same distance metric when training and querying the index. For IVF-based ## Quantization Techniques ### Product Quantization -Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. - -Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. -The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. ![](/static/assets/images/indexing/ivfpq_pq_desc.png) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 302eb6a..41b13ec 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -25,7 +25,7 @@ import { } from '/snippets/indexing.mdx'; -Vector indexes are a robust tool in facilitating fast nearest-neighbor searches across large datasets. +Vector indexes are robust tools in facilitating fast nearest-neighbor searches across large datasets. LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two (todo) advanced techniques that, when combined with [quantization](/indexing/quantization), provide large benefits across a variety of use cases. @@ -58,10 +58,6 @@ For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher co -An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. -It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. - - ### IVF-PQ The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently From 75fead3d6c122945d09b1cc163ef19f43c784a0e Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Thu, 6 Aug 2026 14:58:33 -0700 Subject: [PATCH 19/44] rewrite pq part 1 --- docs/indexing/quantization.mdx | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 0cee789..09b751f 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -26,6 +26,28 @@ Use the same distance metric when training and querying the index. For IVF-based ## Quantization Techniques ### Product Quantization +To visualize PQ quantization, imagine that our dataset consists of a set $S = \{ v_1, \dots, v_n \} \subset \mathbb{R}^d$ of $n$ vectors in $d$-dimensional space. +If each entry of each vector is the same size, for example a `float64`, a naive storage implementation will require $O(nd)$ total bits to store all vectors. +For large $(n,d)$, this is prohibitively expensive. + +Instead, assume we are given a set $C = \{ c_1, \dots, c_k \} \subset \mathbb{R}^d$ of $k$ **centroids**, such that each vector $v_i$ is close to some centroid $c_j$ +(i.e. for some $1 \leq j \leq k$, $\| v_i - c_j \|$ is small). We can then use the following efficient storage scheme: for each $1 \leq i \leq n$, instead of storing $v_i$, +we simply store the integer $j$, where $c_j$ is the centroid associated to $v_i$. To find the distance $\| q - v_i \|$ for some queried vector $q$, +we can recover $c_j$ via a fixed lookup table and simply return $\| q - c_j \|$, which due to the triangle inequality + +$ \| q - v_i \| - \| q - c_j \| \leq \| v_i - c_j \|, $ + +is an approximately correct response. The storage protocol requires $O(\log _2 k)$ bits per vector, instead of $O(d)$ bits in the naive implementation. + +But, how do we obtain the set of $k$ centroids? Let's assume (not unreasonably) that +we would like our set of centroids to have a fixed approximation guarantee. That is, for some $\epsilon > 0$, we would like to guarantee that for every $i$, some centroid $c_j$ satisfies +$\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on a fixed constant $c \in \mathbb{N}$ unique values in each of $d$ dimensions, +spaced far apart. (for example, imagine that our data is the set $\{ (2 \epsilon x_1, \dots, 2 \epsilon x_d) : x_i \in \mathbb{N}, 1 \leq x_i \leq c \} \, \forall \, i$). +In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning each bit would need at least $\log k \geq \Omega(d)$ bits of storage, which is no better than our naive implementation! + + + + ![](/static/assets/images/indexing/ivfpq_pq_desc.png) From 2e67d433f5989ce3b5e67ffc36eecfc9f743a13c Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Fri, 7 Aug 2026 13:19:50 -0700 Subject: [PATCH 20/44] aa --- docs/docs.json | 1 + docs/indexing/quantization.mdx | 31 +++++++++++++++++++------------ docs/static/styles/style.css | 14 ++++++++++++++ 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/docs/docs.json b/docs/docs.json index f8931c1..6e3a637 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -18,6 +18,7 @@ "family": "Inter" }, "styling": { + "latex": true, "codeblocks": { "theme": { "light": "vitesse-light", diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 09b751f..b3947fc 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -26,26 +26,33 @@ Use the same distance metric when training and querying the index. For IVF-based ## Quantization Techniques ### Product Quantization -To visualize PQ quantization, imagine that our dataset consists of a set $S = \{ v_1, \dots, v_n \} \subset \mathbb{R}^d$ of $n$ vectors in $d$-dimensional space. +To visualize PQ quantization, imagine that our dataset consists of a set $S = \{ v_1, \dots, v_n \} \subset ℝ^d$ of $n$ vectors in $d$-dimensional space. If each entry of each vector is the same size, for example a `float64`, a naive storage implementation will require $O(nd)$ total bits to store all vectors. For large $(n,d)$, this is prohibitively expensive. -Instead, assume we are given a set $C = \{ c_1, \dots, c_k \} \subset \mathbb{R}^d$ of $k$ **centroids**, such that each vector $v_i$ is close to some centroid $c_j$ -(i.e. for some $1 \leq j \leq k$, $\| v_i - c_j \|$ is small). We can then use the following efficient storage scheme: for each $1 \leq i \leq n$, instead of storing $v_i$, -we simply store the integer $j$, where $c_j$ is the centroid associated to $v_i$. To find the distance $\| q - v_i \|$ for some queried vector $q$, -we can recover $c_j$ via a fixed lookup table and simply return $\| q - c_j \|$, which due to the triangle inequality + +In this page, we use the notation $\| \cdot \|$ to denote an arbitrary norm. LanceDB indexes use `l2`, the Euclidean $\ell_2$ norm, by default, but users can also specify custom metrics. + -$ \| q - v_i \| - \| q - c_j \| \leq \| v_i - c_j \|, $ +Instead, assume we are given a set $C = \{ c_1, \dots, c_k \} \subset ℝ^d$ of $k$ **centroids**, such that each vector $v_i$ is close to some centroid $c_j$ +(some $\| v_i - c_j \|$ is small). Consider the following improved storage scheme: for each $1 \leq i \leq n$, instead of storing $v_i$, +we simply store the integer $j$, where $c_j$ is the centroid associated to $v_i$. To find a queried distance $\| q - v_i \|$ for some $q$, +we can access $c_j$ via a fixed lookup table and simply return $\| q - c_j \|$, which is a good approximation by the triangle inequality -is an approximately correct response. The storage protocol requires $O(\log _2 k)$ bits per vector, instead of $O(d)$ bits in the naive implementation. +$$ +\| q - v_i \| - \| q - c_j \| \leq \| v_i - c_j \|. +$$ -But, how do we obtain the set of $k$ centroids? Let's assume (not unreasonably) that -we would like our set of centroids to have a fixed approximation guarantee. That is, for some $\epsilon > 0$, we would like to guarantee that for every $i$, some centroid $c_j$ satisfies -$\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on a fixed constant $c \in \mathbb{N}$ unique values in each of $d$ dimensions, -spaced far apart. (for example, imagine that our data is the set $\{ (2 \epsilon x_1, \dots, 2 \epsilon x_d) : x_i \in \mathbb{N}, 1 \leq x_i \leq c \} \, \forall \, i$). -In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning each bit would need at least $\log k \geq \Omega(d)$ bits of storage, which is no better than our naive implementation! +This improved protocol requires $O(\log _2 k)$ bits per vector, instead of the previous $O(d)$ bits. +So, how do we obtain the set of $k$ centroids? Say +we would like our centroid set to have a constant approximation; that is, for some $\epsilon > 0$, we would like every vector $v_i$ to have some centroid $c_j$ +satisfying $\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on a fixed constant $c \in ℕ$-many unique values in each of the $d$ dimensions, +spaced far apart. In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning each bit would need at least $\log k \geq \Omega(d)$ bits of storage, which is no better than our naive implementation! + +To visualize a worst-case scenario for choosing centroids, consider the set $\{ (3 \epsilon x_1, \dots, 3 \epsilon x_d) : x_i \in ℕ, 1 \leq x_i \leq c \quad \forall \, i \}$. + diff --git a/docs/static/styles/style.css b/docs/static/styles/style.css index db71d8b..5a54326 100644 --- a/docs/static/styles/style.css +++ b/docs/static/styles/style.css @@ -3,6 +3,20 @@ --banner-right: #e55a2b; } +/* Mintlify's KaTeX stylesheet currently serves the AMS font from a broken URL. */ +@font-face { + font-family: "LanceDB KaTeX AMS"; + src: url("https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/fonts/KaTeX_AMS-Regular.woff2") format("woff2"); + font-display: swap; + font-style: normal; + font-weight: 400; +} + +.katex .mathbb, +.katex .textbb { + font-family: "LanceDB KaTeX AMS", "KaTeX_AMS", serif !important; +} + /* Mintlify banner gradient */ #banner, :where(.banner, [data-banner], [class*="Banner_banner"]) { From fb4a3391358caff773d39d1af23735b56ef91ab7 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 11 Aug 2026 12:15:56 -0700 Subject: [PATCH 21/44] a --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 41b13ec..97a6b35 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -29,7 +29,7 @@ Vector indexes are robust tools in facilitating fast nearest-neighbor searches a LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two (todo) advanced techniques that, when combined with [quantization](/indexing/quantization), provide large benefits across a variety of use cases. -This page's discussions assume an ambient dataset of numeric vector types. Vector indexing is not supported for incomparable data types. +A dataset of numeric vector types is required for vector indexing. ## Understanding Vector Indexes From f0adbeef3925a1f3e3af00272c657e6016378786 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 11 Aug 2026 17:01:31 -0700 Subject: [PATCH 22/44] almost finished pq --- docs/indexing/quantization.mdx | 37 +++++++++++++++-------------- docs/indexing/vector-index.mdx | 43 +++++++++++++++++++--------------- 2 files changed, 43 insertions(+), 37 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index b3947fc..9a2cac6 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -7,9 +7,8 @@ keywords: ["quantization", "quantize", "rabitq"] --- **Quantization** compresses vectors into concise representations that admit efficient storage. -It is beneficial for high ($512, 768, 1024$ or more)-dimensional datasets, or when index build time and query latency are crucial. - -LanceDB currently exposes several quantized vector index types, with a range of options differing in indexing technique (`IVF_*`,`IVF_HNSW_*`) and method of quantization (`PQ`,`RQ`,`SQ`). +LanceDB currently exposes several quantized vector index types, differing in indexing technique (`IVF_*`,`IVF_HNSW_*`) +and quantization method (`PQ`,`RQ`,`SQ`). - `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples. - `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`. - `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details. @@ -26,18 +25,18 @@ Use the same distance metric when training and querying the index. For IVF-based ## Quantization Techniques ### Product Quantization -To visualize PQ quantization, imagine that our dataset consists of a set $S = \{ v_1, \dots, v_n \} \subset ℝ^d$ of $n$ vectors in $d$-dimensional space. -If each entry of each vector is the same size, for example a `float64`, a naive storage implementation will require $O(nd)$ total bits to store all vectors. +To visualize PQ quantization, imagine a dataset $S = \{ v_1, \dots, v_n \} \subset ℝ^d$ of $n$ vectors in $d$-dimensional space. +If vector entries are fixed-size types, for example a `float64`, a naive storage implementation will require $O(nd)$ total bits to store all vectors. For large $(n,d)$, this is prohibitively expensive. -In this page, we use the notation $\| \cdot \|$ to denote an arbitrary norm. LanceDB indexes use `l2`, the Euclidean $\ell_2$ norm, by default, but users can also specify custom metrics. +In this page, $\| \cdot \|$ denotes an arbitrary norm. LanceDB indexes use `l2`, the Euclidean $\ell_2$ norm, by default. Instead, assume we are given a set $C = \{ c_1, \dots, c_k \} \subset ℝ^d$ of $k$ **centroids**, such that each vector $v_i$ is close to some centroid $c_j$ -(some $\| v_i - c_j \|$ is small). Consider the following improved storage scheme: for each $1 \leq i \leq n$, instead of storing $v_i$, -we simply store the integer $j$, where $c_j$ is the centroid associated to $v_i$. To find a queried distance $\| q - v_i \|$ for some $q$, -we can access $c_j$ via a fixed lookup table and simply return $\| q - c_j \|$, which is a good approximation by the triangle inequality +(some $\| v_i - c_j \|$ is small). For each $1 \leq i \leq n$, instead of storing $v_i$, we can now simply + store the integer $j$, where $c_j$ is the centroid associated to $v_i$. At query time, to find $\| q - v_i \|$ for some $q$, +we can access $c_j$ via a lookup table and simply return $\| q - c_j \|$, which is a good approximation by the triangle inequality $$ \| q - v_i \| - \| q - c_j \| \leq \| v_i - c_j \|. @@ -45,16 +44,21 @@ $$ This improved protocol requires $O(\log _2 k)$ bits per vector, instead of the previous $O(d)$ bits. -So, how do we obtain the set of $k$ centroids? Say -we would like our centroid set to have a constant approximation; that is, for some $\epsilon > 0$, we would like every vector $v_i$ to have some centroid $c_j$ -satisfying $\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on a fixed constant $c \in ℕ$-many unique values in each of the $d$ dimensions, -spaced far apart. In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning each bit would need at least $\log k \geq \Omega(d)$ bits of storage, which is no better than our naive implementation! +So, what about the set of $k$ centroids? Say we would like to have fixed approximation; +that is, for some $\epsilon > 0$, every vector $v_i$ has some centroid $c_j$ +with $\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on constantly many far-apart values in each of the $d$ dimensions. +In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning our lookup table would need +$n \cdot \log k \geq \Omega(nd)$ bits of storage, which is no better than our naive implementation! -To visualize a worst-case scenario for choosing centroids, consider the set $\{ (3 \epsilon x_1, \dots, 3 \epsilon x_d) : x_i \in ℕ, 1 \leq x_i \leq c \quad \forall \, i \}$. +Worst-case scenario: consider the set of points corresponding to the $d$-dimensional grid $\{ (3 \epsilon x_1, \dots, 3 \epsilon x_d) : x_i \in ℕ, 1 \leq x_i \leq c \quad \forall \, i \}$. - +To resolve this problem, product quantization splits each $d$-dimensional vector into $m$ smaller **chunks** of $d / m$ entries each, +yielding $m$ disjoint datasets $\{ S_k \}_{1 \leq k \leq m}$ where $S_k = \{ v_i^{mk \dots m(k+1)}: 1 \leq i \leq n\}$ stores chunk $k$ + of each vector. We then use a separate centroid set $C_k$ for each $S_k$, encoding centroids as before. + We can store all centroid encodings in the same codebook, so our codebook is now size + $O(n \cdot \log (m \cdot c^{d/m})) = O(nd/m + \log m)$ in the worst case, a drastic improvement. ![](/static/assets/images/indexing/ivfpq_pq_desc.png) @@ -67,9 +71,6 @@ Quantized storage: `4 × 8 = 32` bits. In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. -It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. - - ### RaBitQ quantization diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 97a6b35..cee2668 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -61,24 +61,29 @@ For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher co ### IVF-PQ The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently -compress embeddings. We discuss the indexing techniques and algorithms here, deferring discussion of quantization to a later section. For examples of using LanceDB vector indexes, see [Using Vector Indexes](#using-vector-indexes). - -An IVF index is a data structure that aims to accelerate nearest neighbor searches by drastically reducing the search space. -Given a large set of stored vectors, the LanceDB indexer first computes a set of *centroids* corresponding to an approximate solution to the [$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. - -The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. -Each centroid is then identified with its corresponding set of vectors, called its *cluster* or *partition*. -The following image shows an example geometric space partitioned according to this algorithm. The colored marks denote centroids. -![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) - -Once constructed, an IVF index admits efficient ANN queries. Instead of a brute-force comparison of a queried vector to every stored vector, we can now search the smaller set of *centroids* -for a closest match, then execute a brute-force comparison against its cluster. This technique quickly eliminates most clusters from the search space with a single comparison. -Furthermore, since each centroid is close to points in its cluster, we are likely to receive an approximately correct result. - -As an edge case, observe that a queried vector may lie near the boundary of $2$ or more clusters, as shown in the image below. -In this example, the query's true nearest neighbors may lie scattered across several different clusters; thus, our algorithm, which searches only one such cluster, may return rather -imprecise results. To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched. A query with higher `nprobe` will require more runtime but yield more accurate results. -![](/static/assets/images/indexing/ivfpq_query_vector.webp) +compress embeddings. We discuss only the index here. + +An IVF index is a data structure that accelerates ANN searches by drastically reducing the search space. +To construct an IVF index, LanceDB first computes a set of *centroids* corresponding to an approximate solution to the +[$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. +The centroids then partition the dataset as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. +Each centroid remembers its corresponding set of vectors, called its *cluster* or *partition*. + + IVF vector-space partitioning + + +At query time, we can now compare the queried vector to the smaller set of *centroids* +for a closest match, then run a brute-force comparison against the resulting cluster. This technique quickly prunes a large search space, +and since each centroid is close to points in its cluster, we are likely to receive an approximately correct ANN result. + +However, observe that a queried vector may lie near the boundary of $2$ or more clusters. +In the example below, the true nearest neighbors are scattered across several different clusters; thus, our single-cluster brute force search may return +imprecise results. To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched (`default = 1`). +A high `nprobe` parameter will yield more accurate results at slightly higher runtime. + + + IVF vector-space partitioning + ### HNSW @@ -475,4 +480,4 @@ Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar ind TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). - \ No newline at end of file + From b0fdfc0822d55647676a45247c576a99c31b7035 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Thu, 20 Aug 2026 19:20:53 -0700 Subject: [PATCH 23/44] finished pq quantization :) --- docs/indexing/quantization.mdx | 57 +++++++------------------ docs/indexing/vector-index.mdx | 78 ++++++---------------------------- 2 files changed, 30 insertions(+), 105 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 9a2cac6..399cec7 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -15,62 +15,37 @@ and quantization method (`PQ`,`RQ`,`SQ`). - `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads. - `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search. -Different indexing and quantization techniques may be better suited for certain use cases. For example, `IVF_PQ` works well in many cases, but RaBitQ (`IVF_RQ`) allows for more aggressive compression. +Different indexing and quantization techniques may be better suited for certain use cases. See the ["Choosing the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. todo decide what to do here vv Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. -## Quantization Techniques -### Product Quantization - -To visualize PQ quantization, imagine a dataset $S = \{ v_1, \dots, v_n \} \subset ℝ^d$ of $n$ vectors in $d$-dimensional space. -If vector entries are fixed-size types, for example a `float64`, a naive storage implementation will require $O(nd)$ total bits to store all vectors. -For large $(n,d)$, this is prohibitively expensive. - In this page, $\| \cdot \|$ denotes an arbitrary norm. LanceDB indexes use `l2`, the Euclidean $\ell_2$ norm, by default. -Instead, assume we are given a set $C = \{ c_1, \dots, c_k \} \subset ℝ^d$ of $k$ **centroids**, such that each vector $v_i$ is close to some centroid $c_j$ -(some $\| v_i - c_j \|$ is small). For each $1 \leq i \leq n$, instead of storing $v_i$, we can now simply - store the integer $j$, where $c_j$ is the centroid associated to $v_i$. At query time, to find $\| q - v_i \|$ for some $q$, -we can access $c_j$ via a lookup table and simply return $\| q - c_j \|$, which is a good approximation by the triangle inequality - -$$ -\| q - v_i \| - \| q - c_j \| \leq \| v_i - c_j \|. -$$ - -This improved protocol requires $O(\log _2 k)$ bits per vector, instead of the previous $O(d)$ bits. - -So, what about the set of $k$ centroids? Say we would like to have fixed approximation; -that is, for some $\epsilon > 0$, every vector $v_i$ has some centroid $c_j$ -with $\| v_i - c_j \| < \epsilon$. In the worst case, our dataset could take on constantly many far-apart values in each of the $d$ dimensions. -In this case, we may need as many as $k \geq \Omega(c^d)$ centroids, meaning our lookup table would need -$n \cdot \log k \geq \Omega(nd)$ bits of storage, which is no better than our naive implementation! - - -Worst-case scenario: consider the set of points corresponding to the $d$-dimensional grid $\{ (3 \epsilon x_1, \dots, 3 \epsilon x_d) : x_i \in ℕ, 1 \leq x_i \leq c \quad \forall \, i \}$. - - -To resolve this problem, product quantization splits each $d$-dimensional vector into $m$ smaller **chunks** of $d / m$ entries each, -yielding $m$ disjoint datasets $\{ S_k \}_{1 \leq k \leq m}$ where $S_k = \{ v_i^{mk \dots m(k+1)}: 1 \leq i \leq n\}$ stores chunk $k$ - of each vector. We then use a separate centroid set $C_k$ for each $S_k$, encoding centroids as before. - We can store all centroid encodings in the same codebook, so our codebook is now size - $O(n \cdot \log (m \cdot c^{d/m})) = O(nd/m + \log m)$ in the worst case, a drastic improvement. +## Quantization Techniques +### Product Quantization -![](/static/assets/images/indexing/ivfpq_pq_desc.png) +To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has + $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on. -As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. +Now, for each $i$, let $S_i$ represent the set containing chunk $i$ of each vector (entries $(i-1)d$ to $di - 1$). +For each $S_i$ independently, a small set of **centroids** is computed corresponding to an approximate solution to the $k$-means clustering problem. - -Original storage: `128 × 32 = 4096` bits. -Quantized storage: `4 × 8 = 32` bits. +For each original vector in the dataset, every chunk is associated to its nearest centroid; thus, the vector itself is associated with the concatenation of $m$ centroids. +We then gather all our centroids (from all chunks) into a single lookup table, and for each vector, we store the concatenation of IDs of its corresponding centroids. +At query time, we need only to compare each chunk of the queried vector with our set of centroids. -In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. - + + IVF vector-space partitioning + +In the above example, the original vector is split into chunks (subvectors), each of which is associated to a centroid. The stored quantization code for this vector +is the concatenation $2 || 1 || 4 || 3$. The original vector required $128$ dimensions $\times 32$-bit integers $ =4096$ bits total, +which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of quantized storage. ### RaBitQ quantization diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index cee2668..7238ae0 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -25,8 +25,8 @@ import { } from '/snippets/indexing.mdx'; -Vector indexes are robust tools in facilitating fast nearest-neighbor searches across large datasets. -LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two (todo) advanced techniques that, when combined with [quantization](/indexing/quantization), provide large benefits across a variety of use cases. +Vector indexes are robust tools in facilitating fast searches across large datasets. +LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two techniques that, when combined with [quantization](/indexing/quantization), provide benefits across a variety of use cases. A dataset of numeric vector types is required for vector indexing. @@ -63,28 +63,29 @@ For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher co The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently compress embeddings. We discuss only the index here. -An IVF index is a data structure that accelerates ANN searches by drastically reducing the search space. -To construct an IVF index, LanceDB first computes a set of *centroids* corresponding to an approximate solution to the +An IVF index accelerates ANN searches by drastically reducing the search space. The index consists of +a small set of *centroids* corresponding to an approximate solution to the [$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. -The centroids then partition the dataset as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. -Each centroid remembers its corresponding set of vectors, called its *cluster* or *partition*. - +Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, +called its *cluster* or *partition*. + IVF vector-space partitioning -At query time, we can now compare the queried vector to the smaller set of *centroids* -for a closest match, then run a brute-force comparison against the resulting cluster. This technique quickly prunes a large search space, -and since each centroid is close to points in its cluster, we are likely to receive an approximately correct ANN result. +At query time, we can compare the queried vector to the smaller set of *centroids* (as opposed to the entire dataset) +for a closest match, then run a brute-force comparison against its resulting cluster. This technique quickly prunes a large search space, +giving an approximate ANN result. -However, observe that a queried vector may lie near the boundary of $2$ or more clusters. -In the example below, the true nearest neighbors are scattered across several different clusters; thus, our single-cluster brute force search may return -imprecise results. To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched (`default = 1`). +However, observe that a queried vector may lie near the boundary of $2$ or more clusters; thus, the true nearest neighbors are scattered across several different clusters. + To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched (`default = 1`). A high `nprobe` parameter will yield more accurate results at slightly higher runtime. IVF vector-space partitioning +TODO: note default l2 metric + ### HNSW The **Hierarchical Navigable Small World (HNSW)** index ... (todo. to consider: mention skip list?) @@ -322,57 +323,6 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang -### Example: Construct a Binary Vector Index - -Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits. - - Index Configuration - -- Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space. -- Index Type: `IVF_FLAT` is used for indexing binary vectors -- `metric`: the `hamming` distance is used for similarity search -- The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16. - - -**`IVF_FLAT` + `hamming` is the only supported path for binary vectors.** - -- `hamming` distance is only valid on packed binary (uint8) data; it is rejected on float vector columns. -- Quantized index types (`IVF_PQ`, `IVF_RQ`, `IVF_SQ`, `IVF_HNSW_PQ`, `IVF_HNSW_SQ`) do not accept binary inputs — their `distance_type` is restricted to `l2`, `cosine`, or `dot`. - - - 1. Create Table and Schema - - - - {VectorIndexBinarySchema} - - - - 2. Generate and Add Data - - - - {VectorIndexBinaryAddData} - - - - 3. Construct the Binary Index - - - - {VectorIndexBinaryBuildIndex} - - - - 4. Vector Search - - - - {VectorIndexBinarySearch} - - - - ### Manual and Automatic Indexing From d3f7d660cc1b1a4be8e09d688e08ae45d775ff48 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Fri, 21 Aug 2026 15:02:12 -0700 Subject: [PATCH 24/44] fixing examples sections --- docs/docs.json | 2 +- docs/indexing/quantization.mdx | 6 +- docs/indexing/vector-index.mdx | 217 +++++++++++++++------------------ 3 files changed, 102 insertions(+), 123 deletions(-) diff --git a/docs/docs.json b/docs/docs.json index 6e3a637..2ced6ab 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -109,10 +109,10 @@ "pages": [ "indexing/index", "indexing/vector-index", + "indexing/quantization", "indexing/fts-index", "indexing/scalar-index", "indexing/gpu-indexing", - "indexing/quantization", "indexing/reindexing" ] }, diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 399cec7..cfdc207 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -27,7 +27,7 @@ In this page, $\| \cdot \|$ denotes an arbitrary norm. LanceDB indexes use `l2`, ## Quantization Techniques -### Product Quantization +### Product Quantization (PQ) To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on. @@ -47,7 +47,7 @@ In the above example, the original vector is split into chunks (subvectors), eac is the concatenation $2 || 1 || 4 || 3$. The original vector required $128$ dimensions $\times 32$-bit integers $ =4096$ bits total, which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of quantized storage. -### RaBitQ quantization +### RaBitQ Quantization (RQ) RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall. @@ -84,7 +84,7 @@ It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions. -### SQ (todo?) +### Scalar Quantization (SQ) ## API Reference diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 7238ae0..9f1d577 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -1,7 +1,7 @@ --- title: "Vector Indexes" sidebarTitle: "Vector Index" -description: "Build and manage LanceDB vector indexes, including IVF, HNSW and binary quantized indexes." +description: "Build and manage LanceDB vector indexes." icon: "arrow-up-right-dots" --- import { @@ -32,13 +32,17 @@ LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two techn A dataset of numeric vector types is required for vector indexing. -## Understanding Vector Indexes -todo: intro sentence +## Choosing the Right Index + +todo: define all LanceDB supported quantized indexes, exhaustive list. + +-state: all are combinations of an index and a quantization technique. indexes discussed here, quantization in nother tab. + +-then: exposit IVF, HNSW- 2 disclaimers (in overview, description) of HNSW not top-level + +-then: examples of using same IVF, HNSW_quantizeds tuff. -You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: -**Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. -### Choosing the Right Index Use this table to choose the right index and quantization protocol for your use case: @@ -58,6 +62,14 @@ For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher co +## Understanding Vector Indexes +todo: intro sentence + +You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: +**Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. + + + ### IVF-PQ The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently @@ -126,17 +138,8 @@ This combines IVF's scalability with HNSW's higher-recall ANN search within part ## Using Vector Indexes - (todo introduce) - -### Example: Construct an IVF Index - -In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. -Make sure you have enough data in your table (at least a few thousand rows) for effective index training. - - Index Configuration - -Sometimes you need to configure the index beyond default parameters: +### Index Configuration - Index Types: - `IVF_HNSW_FLAT`: highest recall, with no vector quantization @@ -153,28 +156,65 @@ Sometimes you need to configure the index beyond default parameters: `target_partition_size`, `num_partitions` takes precedence. - `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search. -Let's take a look at a sample request for an IVF index: +### Search Configuration + +Core knobs available on a vector search call: + +| Parameter | Description | +| :--- | :--- | +| `limit` | Number of results to return (`k`). | +| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. | +| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. | +| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | +| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | +| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | + +**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. + - {VectorIndexConfigureIvf} + {VectorIndexNprobes} - 1. Setup +Recommended `nprobes` behavior by index type: -Connect to LanceDB and open the table you want to index. +| Index type | Guidance | +| :--- | :--- | +| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | +| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | +| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | + + +### Async API and Config Objects + +With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: - {VectorIndexSetup} + {VectorIndexAsyncConfig} - 2. Construct an IVF Index +Use these Python config classes for the index types shown on this page: + +| Index type | Python config class | +| :--- | :--- | +| `IVF_FLAT` | `IvfFlat` | +| `IVF_PQ` | `IvfPq` | +| `IVF_RQ` | `IvfRq` | +| `IVF_SQ` | `IvfSq` | +| `IVF_HNSW_FLAT` | `IvfHnswFlat` | +| `IVF_HNSW_PQ` | `IvfHnswPq` | +| `IVF_HNSW_SQ` | `IvfHnswSq` | -Create an `IVF_PQ` index with `cosine` similarity. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. + +### Example: Using an IVF Index + +This examples creates an `IVF_PQ` index for a table of vectors under `cosine` similarity. To start, connect to LanceDB and open the table. +Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. @@ -196,68 +236,63 @@ If your vector column lives inside a struct, pass its full dotted path as `vecto Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. - Async API and Config Objects +- `IVF_PQ` + - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). + - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. + - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. +- `IVF_RQ` + - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. -With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: + 3. Query the IVF Index + +Search using a random 1,536-dimensional embedding. - {VectorIndexAsyncConfig} + {VectorIndexQueryIvf} -Use these Python config classes for the index types shown on this page: + +TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). + -| Index type | Python config class | -| :--- | :--- | -| `IVF_FLAT` | `IvfFlat` | -| `IVF_PQ` | `IvfPq` | -| `IVF_RQ` | `IvfRq` | -| `IVF_SQ` | `IvfSq` | -| `IVF_HNSW_FLAT` | `IvfHnswFlat` | -| `IVF_HNSW_PQ` | `IvfHnswPq` | -| `IVF_HNSW_SQ` | `IvfHnswSq` | - 3. Query the IVF Index +### Example: Using an IVF-HNSW Index -Search using a random 1,536-dimensional embedding. + Index Configuration + +There are four key parameters to set when constructing an HNSW index: + +- `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization. +- `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`. +- `m`: The number of neighbors to select for each vector in the HNSW graph. +- `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. + + 1. Construct an HNSW Index + +The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. - {VectorIndexQueryIvf} + {VectorIndexBuildHnsw} - Search Configuration - -Core knobs available on a vector search call: - -| Parameter | Description | -| :--- | :--- | -| `limit` | Number of results to return (`k`). | -| `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. | -| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. | -| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | -| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | -| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | +- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) + - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) + - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. + - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. - -**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. - + 2. Query the HNSW Index - {VectorIndexNprobes} + {VectorIndexQueryHnsw} -Recommended `nprobes` behavior by index type: - -| Index type | Guidance | -| :--- | :--- | -| `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | -| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | -| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | +## Temp Advanced Search Controls @@ -294,36 +329,6 @@ Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall mea Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. -### Example: Construct an HNSW Index - - Index Configuration - -There are four key parameters to set when constructing an HNSW index: - -- `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization. -- `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`. -- `m`: The number of neighbors to select for each vector in the HNSW graph. -- `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. - - 1. Construct an HNSW Index - -The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. - - - - {VectorIndexBuildHnsw} - - - - 2. Query the HNSW Index - - - - {VectorIndexQueryHnsw} - - - - ### Manual and Automatic Indexing Enterprise-only @@ -335,17 +340,6 @@ In LanceDB Enterprise, vector indexes are managed **automatically**. The system Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. LanceDB also provides an interface to tune parameters manually as data changes. The following default parameters are recommended: -- `IVF_PQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). - - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. - - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. -- `IVF_RQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. -- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) - - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) - - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. - - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. - You can call `create_index()` to create a new index (this replaces existing indexes). `create_index()` returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the `wait_timeout` parameter. @@ -367,10 +361,6 @@ fallback and searches only indexed rows. - - -## Misc(todo choose name) - ### Check Index Status Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. @@ -419,15 +409,4 @@ For vector indexes, make sure to use the same distance metric when creating and Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to `BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. - - - - -## Temp (scrap) - - - - - -TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). - + \ No newline at end of file From de1b3d385d6a149716cad9b55687b69da642554d Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 24 Aug 2026 15:40:15 -0700 Subject: [PATCH 25/44] high level organization --- docs/indexing/quantization.mdx | 24 +++----------- docs/indexing/vector-index.mdx | 60 ++++++++++++++-------------------- 2 files changed, 28 insertions(+), 56 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index cfdc207..3df5a76 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -6,29 +6,13 @@ icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- -**Quantization** compresses vectors into concise representations that admit efficient storage. -LanceDB currently exposes several quantized vector index types, differing in indexing technique (`IVF_*`,`IVF_HNSW_*`) -and quantization method (`PQ`,`RQ`,`SQ`). -- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples. -- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`. -- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details. -- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads. -- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search. - -Different indexing and quantization techniques may be better suited for certain use cases. -See the ["Choosing the Right Index"](/indexing/vector-index#choose-the-right-index) table for further discussion. - -todo decide what to do here vv - -Use the same distance metric when training and querying the index. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`. - - -In this page, $\| \cdot \|$ denotes an arbitrary norm. LanceDB indexes use `l2`, the Euclidean $\ell_2$ norm, by default. - +**Quantization** is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here; +discussion of LanceDB vector indexes and quantized vector indexes can be found [here](/indexing/vector-index). ## Quantization Techniques -### Product Quantization (PQ) +LanceDB provides $3$ distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ). +### Product Quantization (PQ) To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on. diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 9f1d577..1f9a5a8 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -26,7 +26,7 @@ import { Vector indexes are robust tools in facilitating fast searches across large datasets. -LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with two techniques that, when combined with [quantization](/indexing/quantization), provide benefits across a variety of use cases. +LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with several techniques that provide benefits across a variety of use cases. A dataset of numeric vector types is required for vector indexing. @@ -34,34 +34,24 @@ A dataset of numeric vector types is required for vector indexing. ## Choosing the Right Index -todo: define all LanceDB supported quantized indexes, exhaustive list. +LanceDB vector indexes are combined with several [quantization](/indexing/quantization) techniques to admit efficient storage. +The following table lists provided quantized vector indexes and their common use cases: --state: all are combinations of an index and a quantization technique. indexes discussed here, quantization in nother tab. - --then: exposit IVF, HNSW- 2 disclaimers (in overview, description) of HNSW not top-level - --then: examples of using same IVF, HNSW_quantizeds tuff. - - - -Use this table to choose the right index and quantization protocol for your use case: - -| If your top priority is... | Use this index | Why | Typical compressed size vs. raw vectors | +| If your top priority is... | Use this index | Why | Approx. compression ratio | | :--- | :--- | :--- | :--- | -| Highest recall / no quantization | `IVF_HNSW_FLAT` | Uses raw vectors inside the IVF+HNSW structure, avoiding quantization loss. | Around raw vector size plus HNSW graph overhead | -| Best recall/latency trade-off | `IVF_HNSW_SQ` | Combines IVF partitioning with HNSW graph search for strong quality at low latency. | Typically a little larger than `1/4` of raw size | -| Maximum compression | `IVF_RQ` | RaBitQ-style quantization with very strong compression. | Around `1/32` of raw size | -| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | On small-dimensional vectors, `IVF_PQ` often provides higher accuracy with similar performance compared to `IVF_RQ`. | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | +| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | +| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | +| | `IVF_SQ` | IVF indexing with scalar quantization | Varies | +| | `IVF_HNSW_PQ` | IVF-HNSW indexing with product quantization | Varies | +| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | +| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | +| | `IVF_FLAT` | IVF indexing with no quantization | Varies | + -If your vector search frequently includes metadata filters (`where(...)`), prefer `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance. +If your vector search frequently includes metadata filters (`where(...)`), use `IVF_RQ` or `IVF_PQ`. In filtered workloads, HNSW-backed IVF indexes such as `IVF_HNSW_FLAT` and `IVF_HNSW_SQ` can show higher latency variance. -Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration. -For small dimensions, choose `IVF_PQ` for accuracy, not for guaranteed higher compression than `IVF_RQ`. - - - ## Understanding Vector Indexes todo: intro sentence @@ -70,12 +60,9 @@ You can create and manage multiple vector indexes on any Lance dataset. LanceDB -### IVF-PQ +### IVF -The **Inverted File-Product Quantization Index (IVF-PQ)** combines the clustering-based **Inverted File Index (IVF)** with [**Product Quantization (PQ)**](/indexing/quantization) to efficiently -compress embeddings. We discuss only the index here. - -An IVF index accelerates ANN searches by drastically reducing the search space. The index consists of +The **Inverted File Index (IVF)** accelerates ANN searches by drastically reducing the search space. The index consists of a small set of *centroids* corresponding to an approximate solution to the [$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, @@ -148,7 +135,7 @@ This combines IVF's scalability with HNSW's higher-recall ANN search within part - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance - `metrics`: default is `l2`, other available are `cosine` or `dot` - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar) -- `num_partitions`: use index-specific starting points from the section above: +- `num_partitions`: use index-specific starting points: - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576` - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096` - `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition @@ -351,7 +338,6 @@ Be careful to use the same distance metric during index creation and search (`l2 If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. -It can time out if new writes keep arriving. @@ -368,7 +354,6 @@ Vector index creation runs in the background and may take some time to complete. In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes. Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built. -To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_index()` or call `wait_for_index()` on the table. Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`): @@ -404,9 +389,12 @@ The `{column}_idx` suffix is a default convention, not the only supported naming **Operational checks** -For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. -`wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. +After appends or other writes, use `optimize()` to fold new rows into existing indexes. +Unless specified otherwise, vector indexing defaults to `IVF_PQ`. + + + +### temp -Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to -`BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. - \ No newline at end of file + +-then: exposit IVF, HNSW- 2 disclaimers (in overview, description) of HNSW not top-level From 8a12b5519e6cb389e00e8404b32f59b4e7e71644 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 24 Aug 2026 17:52:22 -0700 Subject: [PATCH 26/44] examples and stuff --- docs/indexing/vector-index.mdx | 226 ++++++++++++--------------------- 1 file changed, 79 insertions(+), 147 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 1f9a5a8..93128e5 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -25,27 +25,23 @@ import { } from '/snippets/indexing.mdx'; -Vector indexes are robust tools in facilitating fast searches across large datasets. +Vector indexes are robust tools in facilitating fast searches across large numeric datasets. LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with several techniques that provide benefits across a variety of use cases. - -A dataset of numeric vector types is required for vector indexing. - - ## Choosing the Right Index LanceDB vector indexes are combined with several [quantization](/indexing/quantization) techniques to admit efficient storage. The following table lists provided quantized vector indexes and their common use cases: -| If your top priority is... | Use this index | Why | Approx. compression ratio | -| :--- | :--- | :--- | :--- | -| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | -| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | -| | `IVF_SQ` | IVF indexing with scalar quantization | Varies | -| | `IVF_HNSW_PQ` | IVF-HNSW indexing with product quantization | Varies | -| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | -| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | -| | `IVF_FLAT` | IVF indexing with no quantization | Varies | +| If your top priority is... | Use this index | Why | Approx. compression ratio | Python config class | +| :--- | :--- | :--- | :--- | :--- | +| Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` | +| Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | `IvfRq` | +| | `IVF_SQ` | IVF indexing with scalar quantization | Varies | `IvfSq` | +| | `IVF_HNSW_PQ` | IVF-HNSW indexing with product quantization | Varies | `IvfHnswPq` | +| Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` | +| Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` | +| | `IVF_FLAT` | IVF indexing with no quantization | Varies | `IvfFlat` | @@ -86,33 +82,10 @@ A high `nprobe` parameter will yield more accurate results at slightly higher ru TODO: note default l2 metric ### HNSW - +{/* The **Hierarchical Navigable Small World (HNSW)** index ... (todo. to consider: mention skip list?) - -background: kNN graphs: vv -* Each vector in the dataset is given an associated vertex. -* Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. -* For some applications (including nearest-neighbor search), the incoming edges are also added. - -Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: -* Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). -* Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. -* Repeat until a local optimum is found. -The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. - - -computing a kNN graph is slow, so we instead use a k-ANN graph: -* Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. -* The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. - -This recursive structure can be thought of as separating into layers: -* At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. - -to query: -* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. repeat - - -HNSW ends up needing a lot of memory. +*/} +Coming soon! **IVF + HNSW** @@ -125,27 +98,33 @@ This combines IVF's scalability with HNSW's higher-recall ANN search within part ## Using Vector Indexes +This section demonstrates how to configure, build, and search LanceDB vector indexes. -### Index Configuration +### Configuration -- Index Types: - - `IVF_HNSW_FLAT`: highest recall, with no vector quantization - - `IVF_HNSW_SQ`: best recall/latency trade-off - - `IVF_RQ`: best compression for large, high-dimensional datasets - - `IVF_PQ`: often higher accuracy than `IVF_RQ` for small dimensions (`<= 256`) at similar query performance -- `metrics`: default is `l2`, other available are `cosine` or `dot` - - When using `cosine` similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar) -- `num_partitions`: use index-specific starting points: - - HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`): `num_rows // 1,048,576` - - `IVF_RQ` and `IVF_PQ`: `num_rows // 4096` -- `target_partition_size`: alternative IVF sizing knob that asks LanceDB to derive the partition - count from a target number of rows per partition. If you set both `num_partitions` and - `target_partition_size`, `num_partitions` takes precedence. -- `num_sub_vectors`: applies to `IVF_PQ`; start with `dimension // 8`. Larger values often improve recall but can slow search. +#### Build-time Parameters -### Search Configuration +Parameters that apply to every LanceDB quantized index type. Pick an `index_type` using the table in [Choosing the Right Index](#choosing-the-right-index) above. -Core knobs available on a vector search call: +- `metric`: default is `l2`, other available are `cosine` or `dot` + - When using `cosine` distance, values range from $0$ (identical) to $2$ (maximally dissimilar) +- `num_partitions`: when left unset, LanceDB automatically chooses roughly `sqrt(num_rows)`. +- `target_partition_size`: an alternative IVF sizing knob that derives the partition count from a target number of rows per partition, instead of a partition count directly. Defaults to `8192` for IVF-family indexes (`IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ`) and `1,048,576` for IVF-HNSW-family indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) — HNSW graphs are far more memory-intensive to train, so each partition is kept much larger to limit how many separate graphs get built. If both `num_partitions` and `target_partition_size` are set, `num_partitions` takes precedence. +- `num_sub_vectors`: applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if the dimension isn't evenly divisible by 16). Larger values often improve recall but can slow search. + +##### Async API and Config Objects + +With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings — pass an instance of the Python config class from the table in [Choosing the Right Index](#choosing-the-right-index) above: + + + + {VectorIndexAsyncConfig} + + + +#### Search-time Parameters + +Core knobs and controls available on a vector search call: | Parameter | Description | | :--- | :--- | @@ -155,6 +134,8 @@ Core knobs available on a vector search call: | `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | | `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | | `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | +| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | +| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | **Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. @@ -174,33 +155,39 @@ Recommended `nprobes` behavior by index type: | `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | | `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | +##### Thresholding and Recall Measurement -### Async API and Config Objects +These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. -With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: +**Thresholding with `distance_range`:** - {VectorIndexAsyncConfig} + {VectorIndexDistanceRange} -Use these Python config classes for the index types shown on this page: +**Measuring recall with `bypass_vector_index`:** -| Index type | Python config class | -| :--- | :--- | -| `IVF_FLAT` | `IvfFlat` | -| `IVF_PQ` | `IvfPq` | -| `IVF_RQ` | `IvfRq` | -| `IVF_SQ` | `IvfSq` | -| `IVF_HNSW_FLAT` | `IvfHnswFlat` | -| `IVF_HNSW_PQ` | `IvfHnswPq` | -| `IVF_HNSW_SQ` | `IvfHnswSq` | +Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload. + + + {VectorIndexBypassRecall} + + -### Example: Using an IVF Index + +Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries. + -This examples creates an `IVF_PQ` index for a table of vectors under `cosine` similarity. To start, connect to LanceDB and open the table. + +Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. + + +### IVF Indexes + +This example creates an `IVF_PQ` index for a table of vectors under `cosine` similarity. To start, connect to LanceDB and open the table. Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. @@ -209,7 +196,7 @@ Specify `vector_column_name` if you use multiple vector columns or non-default n - Indexing nested vector fields +#### Indexing nested vector fields If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`: @@ -223,14 +210,14 @@ If your vector column lives inside a struct, pass its full dotted path as `vecto Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. -- `IVF_PQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). - - `num_sub_vectors`: start at `dimension // 8`. Increase for better recall, decrease for faster search and smaller indexes. - - For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. -- `IVF_RQ` - - `num_partitions`: start at `num_rows // 4096` (rounded to an integer). This is a strong default for most datasets. +#### Recommended defaults + +Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only IVF-specific knob is: + +- `num_sub_vectors` (`IVF_PQ` only): increase for better recall, decrease for faster search and smaller indexes. +- For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. - 3. Query the IVF Index +#### Querying an IVF Index Search using a random 1,536-dimensional embedding. @@ -244,20 +231,13 @@ Search using a random 1,536-dimensional embedding. TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). +### IVF-HNSW Indexes -### Example: Using an IVF-HNSW Index +Beyond the general build-time parameters above, two additional parameters are specific to constructing an HNSW graph: - Index Configuration - -There are four key parameters to set when constructing an HNSW index: - -- `index_type`: choose `IVF_HNSW_SQ` for a strong recall/latency/size trade-off, or `IVF_HNSW_FLAT` when you want the IVF+HNSW structure without vector quantization. -- `metric`: The default is `l2` euclidean distance metric. Other available are `dot` and `cosine`. - `m`: The number of neighbors to select for each vector in the HNSW graph. - `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. - 1. Construct an HNSW Index - The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. @@ -266,57 +246,21 @@ The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, chang -- HNSW-backed IVF indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) - - `num_partitions`: start at `num_rows // 1,048,576` (rounded to an integer) - - Lower `num_partitions` can reduce search latency, but index build may become slower because partitions are larger. - - `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. - - 2. Query the HNSW Index - - - - {VectorIndexQueryHnsw} - - +#### Recommended defaults -## Temp +Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only HNSW-specific knob is: - Advanced Search Controls +- `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. -These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. - -| Method | Description | -| :--- | :--- | -| `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | -| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | - -**Thresholding with `distance_range`:** +#### Querying an IVF-HNSW Index - {VectorIndexDistanceRange} - - - -**Measuring recall with `bypass_vector_index`:** - -Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload. - - - - {VectorIndexBypassRecall} + {VectorIndexQueryHnsw} - -Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries. - - - -Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. - - -### Manual and Automatic Indexing +### Managing Vector Indexes Enterprise-only In LanceDB Enterprise, vector indexes are managed **automatically**. The system asynchronously updates and optimizes indexes as a background process: @@ -324,30 +268,24 @@ In LanceDB Enterprise, vector indexes are managed **automatically**. The system - Optimizes `IVF_PQ` storage without manual input - Infers vector columns from the schema - Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. - LanceDB also provides an interface to tune parameters manually as data changes. The following default parameters are recommended: + Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. LanceDB also provides an interface to tune parameters manually as data changes — see the recommended defaults in the IVF and IVF-HNSW sections above. You can call `create_index()` to create a new index (this replaces existing indexes). -`create_index()` returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the `wait_timeout` parameter. +`create_index()` returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the `wait_timeout` parameter, or call `wait_for_index(...)` afterward — it waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. Be careful to use the same distance metric during index creation and search (`l2` by default). - -If you need to confirm an async build or refresh is finished, `wait_for_index(...)` waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. - - Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that fallback and searches only indexed rows. - -### Check Index Status +#### Check Index Status Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. @@ -376,7 +314,7 @@ These fields are populated for local and embedded tables. On LanceDB Enterprise -### Custom Index Names +#### Custom Index Names The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name. @@ -392,9 +330,3 @@ The `{column}_idx` suffix is a default convention, not the only supported naming After appends or other writes, use `optimize()` to fold new rows into existing indexes. Unless specified otherwise, vector indexing defaults to `IVF_PQ`. - - -### temp - - --then: exposit IVF, HNSW- 2 disclaimers (in overview, description) of HNSW not top-level From 366659a952ee621fcc9cdc2a28597f5be4da7c7f Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 24 Aug 2026 19:34:44 -0700 Subject: [PATCH 27/44] a --- docs/indexing/vector-index.mdx | 85 +++++++++++----------------------- 1 file changed, 27 insertions(+), 58 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 93128e5..7a46059 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -49,7 +49,6 @@ If your vector search frequently includes metadata filters (`where(...)`), use ` ## Understanding Vector Indexes -todo: intro sentence You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. @@ -79,8 +78,6 @@ A high `nprobe` parameter will yield more accurate results at slightly higher ru IVF vector-space partitioning -TODO: note default l2 metric - ### HNSW {/* The **Hierarchical Navigable Small World (HNSW)** index ... (todo. to consider: mention skip list?) @@ -106,21 +103,12 @@ This section demonstrates how to configure, build, and search LanceDB vector ind Parameters that apply to every LanceDB quantized index type. Pick an `index_type` using the table in [Choosing the Right Index](#choosing-the-right-index) above. -- `metric`: default is `l2`, other available are `cosine` or `dot` - - When using `cosine` distance, values range from $0$ (identical) to $2$ (maximally dissimilar) -- `num_partitions`: when left unset, LanceDB automatically chooses roughly `sqrt(num_rows)`. -- `target_partition_size`: an alternative IVF sizing knob that derives the partition count from a target number of rows per partition, instead of a partition count directly. Defaults to `8192` for IVF-family indexes (`IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ`) and `1,048,576` for IVF-HNSW-family indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) — HNSW graphs are far more memory-intensive to train, so each partition is kept much larger to limit how many separate graphs get built. If both `num_partitions` and `target_partition_size` are set, `num_partitions` takes precedence. -- `num_sub_vectors`: applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if the dimension isn't evenly divisible by 16). Larger values often improve recall but can slow search. - -##### Async API and Config Objects - -With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings — pass an instance of the Python config class from the table in [Choosing the Right Index](#choosing-the-right-index) above: - - - - {VectorIndexAsyncConfig} - - +| Parameter | Description | +| :--- | :--- | +| `metric` | Default is `l2`, other available are `cosine` or `dot`. `cosine` distances range from `0` (identical) to `2` (maximally dissimilar). | +| `num_partitions` | When left unset, LanceDB automatically chooses roughly `sqrt(num_rows)`. | +| `target_partition_size` | An alternative IVF sizing knob that derives the partition count from a target number of rows per partition, instead of a partition count directly. Defaults to `8192` for IVF-family indexes (`IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ`) and `1,048,576` for IVF-HNSW-family indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) — HNSW graphs are far more memory-intensive to train, so each partition is kept much larger to limit how many separate graphs get built. If both `num_partitions` and `target_partition_size` are set, `num_partitions` takes precedence. | +| `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if the dimension isn't evenly divisible by 16). Larger values often improve recall but can slow search. | #### Search-time Parameters @@ -185,78 +173,59 @@ Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall mea Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. -### IVF Indexes +#### Async API and Config Objects -This example creates an `IVF_PQ` index for a table of vectors under `cosine` similarity. To start, connect to LanceDB and open the table. -Specify `vector_column_name` if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. `image.embedding`); see [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full syntax. You can switch `index_type` to `IVF_RQ`, `IVF_HNSW_SQ`, or `IVF_HNSW_FLAT` depending on your recall/latency/compression target. +With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same build-time choices described above, such as distance metric, partition count, and quantization settings — pass an instance of the Python config class from the table in [Choosing the Right Index](#choosing-the-right-index): - {VectorIndexBuildIvf} + {VectorIndexAsyncConfig} -#### Indexing nested vector fields +### IVF Indexes + +This example creates an `IVF_PQ` index for a table of vectors under `cosine` similarity, then queries it with a random 1,536-dimensional embedding. Specify `vector_column_name` if you use multiple vector columns or non-default names. You can also switch `index_type` to `IVF_RQ` here without changing anything else. -If your vector column lives inside a struct, pass its full dotted path as `vector_column_name`. The same path is used at query time and is what `list_indices()` reports under `columns`: +Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only IVF-specific knob is `num_sub_vectors` (`IVF_PQ` only): increase for better recall, decrease for faster search and smaller indexes. - {VectorIndexNestedField} + {VectorIndexBuildIvf + VectorIndexQueryIvf} -Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). The same convention applies to FTS and scalar indexes. +TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). -#### Recommended defaults - -Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only IVF-specific knob is: - -- `num_sub_vectors` (`IVF_PQ` only): increase for better recall, decrease for faster search and smaller indexes. -- For small dimensions (`dimension <= 256`), `IVF_PQ` is often preferred over `IVF_RQ` for better accuracy at similar query performance. - -#### Querying an IVF Index - -Search using a random 1,536-dimensional embedding. +For a vector field nested inside a struct, pass its full dotted path as `vector_column_name` (e.g. `image.embedding`) — the same path is used at query time and is what `list_indices()` reports under `columns`. See [Selecting the vector column](/search/vector-search#selecting-the-vector-column) for the full path syntax. - {VectorIndexQueryIvf} + {VectorIndexNestedField} -TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). +Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). ### IVF-HNSW Indexes -Beyond the general build-time parameters above, two additional parameters are specific to constructing an HNSW graph: - -- `m`: The number of neighbors to select for each vector in the HNSW graph. -- `ef_construction`: The number of candidates to evaluate during the construction of the HNSW graph. +Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes: -The snippet below uses `IVF_HNSW_SQ`. If you want the unquantized variant, change `index_type` to `IVF_HNSW_FLAT`. - - - - {VectorIndexBuildHnsw} - - - -#### Recommended defaults - -Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only HNSW-specific knob is: - -- `ef_construction`: start at `150`; increase for better recall, decrease for faster indexing. +| Parameter | Description | +| :--- | :--- | +| `m` | The number of neighbors to select for each vector in the HNSW graph. | +| `ef_construction` | The number of candidates to evaluate during the construction of the HNSW graph. Start at `150`; increase for better recall, decrease for faster indexing. | -#### Querying an IVF-HNSW Index +Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. +The snippet below builds and queries an `IVF_HNSW_SQ` index. - {VectorIndexQueryHnsw} + {VectorIndexBuildHnsw + VectorIndexQueryHnsw} @@ -268,7 +237,7 @@ In LanceDB Enterprise, vector indexes are managed **automatically**. The system - Optimizes `IVF_PQ` storage without manual input - Infers vector columns from the schema - Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. LanceDB also provides an interface to tune parameters manually as data changes — see the recommended defaults in the IVF and IVF-HNSW sections above. + Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. LanceDB also provides an interface to tune parameters manually as data changes — see the IVF and IVF-HNSW sections above for index-specific tuning guidance. You can call `create_index()` to create a new index (this replaces existing indexes). From 7ea0e9ed90d443c861184720bcd88a5decca4a5f Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 25 Aug 2026 11:13:29 -0700 Subject: [PATCH 28/44] finished rabitq --- docs/indexing/quantization.mdx | 41 ++++++++++++---------------------- 1 file changed, 14 insertions(+), 27 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 3df5a76..d41c599 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -33,41 +33,28 @@ which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of q ### RaBitQ Quantization (RQ) -RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall. +RaBitQ is an advanced quantization technique that outperforms PQ in several ways. +It needs no codebook to train, estimates distances very quickly at query-time, +and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!** +In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few hundred bytes, while maintaining good recall. -#### How RaBitQ works - -- Embeddings are grouped around centroids (as in other IVF indexes). -- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere. -- The sign pattern of that vector is stored as bits (1 bit per dimension). -- Two small corrective factors are stored: - 1. The distance from the original vector to its centroid - 2. The dot product between the normalized vector and its quantized version - -Compared to `IVF_PQ`, RaBitQ: -- Avoids training expensive PQ codebooks -- Builds indexes faster and handles updates more easily -- Maintains or improves recall at high dimensionality under the same storage budget - -For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/). +The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by +applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube, +where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497) #### Using RaBitQ +Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`. +Note that when using `IVF_RQ`, the dimension of the dataset should be a multiple of `8`. -You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`. - - -When using `IVF_RQ`, the dimension of vectors must be a multiple of `8`. - - -`num_bits` controls how many bits per dimension are used: - -1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute. -It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`. +`num_bits` determines how many bits are used to quantize each dimension. +`1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute. -Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions. +RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout, and cannot be read by some older LanceDB versions. +See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation. + ### Scalar Quantization (SQ) ## API Reference From 492c6f583b3d50c3fd3bbfd294c188ba24788438 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Tue, 25 Aug 2026 15:21:47 -0700 Subject: [PATCH 29/44] finished sq quantization --- docs/indexing/quantization.mdx | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index d41c599..e1cbd8f 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -1,7 +1,7 @@ --- title: "Quantization" sidebarTitle: "Quantization" -description: "Use quantization to improve storage requirements and query latency of your LanceDB vector index." +description: "Use quantization efficiently store your LanceDB vector index." icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- @@ -56,7 +56,12 @@ RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layou See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation. ### Scalar Quantization (SQ) +Scalar quantization quantizes each entry of a vector independently, by simply replacing it with the closest of a pre-defined set of values. +In practice, it often uses $8$ bits per dimension of a vector, and supports very fast encoding and decoding. +For example, suppose we know all vector entires across our dataset lie in the range $[-128 \times 10^5, 127 \times 10^5]$. +In this case, we could quantize a given value $v$ as an $8$-bit representation of the integer $j$, where $j \times 10^5$ is the closest value to $v$ +among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$. ## API Reference The full list of parameters to the algorithm are listed below. From 29b2c8676d5fcc3b71b0a40b9b25de0d19aa76a2 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 26 Aug 2026 09:29:42 -0700 Subject: [PATCH 30/44] a --- docs/indexing/index.mdx | 16 ++++----- docs/indexing/vector-index.mdx | 63 ++++++++++++++++------------------ 2 files changed, 36 insertions(+), 43 deletions(-) diff --git a/docs/indexing/index.mdx b/docs/indexing/index.mdx index aac426e..7928bdc 100644 --- a/docs/indexing/index.mdx +++ b/docs/indexing/index.mdx @@ -6,18 +6,18 @@ icon: "list" --- An **index** is a data structure that facilitates efficient scans and lookups on an embedded dataset. - -## Supported Indexes LanceDB provides a comprehensive suite of indexes to optimize performance across different use cases and data types: - **Vector Index**: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings) - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases - **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data +## Supported Indexes + | Index | Use Case | Description | | :--------- | :------- | :---------- | -| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: $\ell_2$ `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| -| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: $\ell_2$ `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| +| `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
**Quantizations**: `None/Flat` `PQ` `SQ` `RQ`| +| `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs in each partition. Provides improved search quality over pure IVF while maintaining scalability.
**Quantizations**: `None/Flat` `SQ` `PQ`| | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | @@ -25,15 +25,11 @@ LanceDB provides a comprehensive suite of indexes to optimize performance across | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | ## Quantization -LanceDB also supports several different [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and reduce storage requirements: +LanceDB also supports several [quantization](/indexing/quantization) methods, used by vector indexes to compress vectors and reduce storage requirements: | Quantization | Use Case | Description | | :----------- | :------- | :---------- | | `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. | -| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | +| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | - -## temp - -LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 7a46059..1674eaf 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -31,9 +31,11 @@ LanceDB implements **ANN (Approximate Nearest-Neighbor)** queries with several t ## Choosing the Right Index LanceDB vector indexes are combined with several [quantization](/indexing/quantization) techniques to admit efficient storage. -The following table lists provided quantized vector indexes and their common use cases: +The following table lists provided quantized vector indexes and their common use cases. You can specify index type manually in +Lance with `index_type`. -| If your top priority is... | Use this index | Why | Approx. compression ratio | Python config class | + +| If your priority is... | Use this index | Why | Approx. compression ratio | Python config class | | :--- | :--- | :--- | :--- | :--- | | Higher accuracy at small dimensions (`dimension <= 256`) | `IVF_PQ` | IVF indexing with product quantization | Usually `1/64` to `1/16` of raw size (depends on `num_sub_vectors`) | `IvfPq` | | Maximum compression | `IVF_RQ` | IVF indexing with RaBitQ quantization | Around `1/32` of raw size | `IvfRq` | @@ -50,11 +52,9 @@ If your vector search frequently includes metadata filters (`where(...)`), use ` ## Understanding Vector Indexes -You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two vector indexing algorithms: +LanceDB offers two vector indexes, which can be created on any numeric Lance dataset: **Inverted File (IVF)** and **Hierarchical Navigable Small World (HNSW)**. - - ### IVF The **Inverted File Index (IVF)** accelerates ANN searches by drastically reducing the search space. The index consists of @@ -87,32 +87,31 @@ Coming soon! **IVF + HNSW** -In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a sub-index inside IVF partitions. -What this means in practice is that vectors are first partitioned by IVF, then each selected partition is searched using an HNSW graph. -LanceDB supports the unquantized variant `IVF_HNSW_FLAT`, along with quantized variants such as `IVF_HNSW_PQ` and `IVF_HNSW_SQ`. -This combines IVF's scalability with HNSW's higher-recall ANN search within partitions. +In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a substructure which +further indexes the selected vectors inside each IVF partition. +This combines the scalability of IVF with the high recall of HNSW. +LanceDB supports quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW_SQ`. ## Using Vector Indexes -This section demonstrates how to configure, build, and search LanceDB vector indexes. +Learn how to configure, build, and search LanceDB vector indexes. ### Configuration #### Build-time Parameters -Parameters that apply to every LanceDB quantized index type. Pick an `index_type` using the table in [Choosing the Right Index](#choosing-the-right-index) above. - +These parameters apply to every LanceDB quantized index type. | Parameter | Description | | :--- | :--- | -| `metric` | Default is `l2`, other available are `cosine` or `dot`. `cosine` distances range from `0` (identical) to `2` (maximally dissimilar). | +| `metric` | Default is `l2`, others available are `cosine` or `dot`. | `num_partitions` | When left unset, LanceDB automatically chooses roughly `sqrt(num_rows)`. | | `target_partition_size` | An alternative IVF sizing knob that derives the partition count from a target number of rows per partition, instead of a partition count directly. Defaults to `8192` for IVF-family indexes (`IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ`) and `1,048,576` for IVF-HNSW-family indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) — HNSW graphs are far more memory-intensive to train, so each partition is kept much larger to limit how many separate graphs get built. If both `num_partitions` and `target_partition_size` are set, `num_partitions` takes precedence. | | `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if the dimension isn't evenly divisible by 16). Larger values often improve recall but can slow search. | #### Search-time Parameters -Core knobs and controls available on a vector search call: +Knobs available on a vector search call: | Parameter | Description | | :--- | :--- | @@ -125,17 +124,7 @@ Core knobs and controls available on a vector search call: | `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | | `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | - -**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB starts by scanning `minimum_nprobes` partitions and only extends toward `maximum_nprobes` if fewer than `limit` rows survive the filter. Setting `minimum_nprobes == maximum_nprobes` (or calling `nprobes(n)`) disables this adaptive behavior and fixes the partition count. - - - - - {VectorIndexNprobes} - - - -Recommended `nprobes` behavior by index type: +We recommend the following `nprobes` behavior by index type: | Index type | Guidance | | :--- | :--- | @@ -143,9 +132,20 @@ Recommended `nprobes` behavior by index type: | `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | | `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | -##### Thresholding and Recall Measurement +An example of a vector search exercising several of the above parameters. + + + + {VectorIndexNprobes} + + -These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. + +**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` +partitions and uses a wider scan if sufficiently few rows are found. +Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count. + +There are also several search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. **Thresholding with `distance_range`:** @@ -155,9 +155,10 @@ These controls are useful for thresholded retrieval, recall measurement, and wor -**Measuring recall with `bypass_vector_index`:** +**Using `bypass_vector_index`:** -Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick `nprobes` for your workload. +Use `bypass_vector_index` to compute an exact **kNN** result. Compare with ANN results to compute **recall@k**. +Recall that exact queries may be prohibitively slow on production scales. @@ -165,10 +166,6 @@ Compare ANN results against a flat-scan ground truth to compute recall@k. This i - -Flat search is $O(n)$ — reserve `bypass_vector_index()` for sampled recall measurements or small tables, not production queries. - - Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. From 7d9e971fe23c9f6d2b364e923b8ab8e172c87047 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 26 Aug 2026 10:24:54 -0700 Subject: [PATCH 31/44] finished using vector indexes/configuration --- docs/indexing/vector-index.mdx | 51 +++++++++++++++++----------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 1674eaf..f002d79 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -95,36 +95,33 @@ LanceDB supports quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW ## Using Vector Indexes -Learn how to configure, build, and search LanceDB vector indexes. +Learn how to configure, build, and search LanceDB vector indexes, including build and search time parameters, asynchronous objects, and several examples. ### Configuration #### Build-time Parameters -These parameters apply to every LanceDB quantized index type. | Parameter | Description | | :--- | :--- | -| `metric` | Default is `l2`, others available are `cosine` or `dot`. -| `num_partitions` | When left unset, LanceDB automatically chooses roughly `sqrt(num_rows)`. | -| `target_partition_size` | An alternative IVF sizing knob that derives the partition count from a target number of rows per partition, instead of a partition count directly. Defaults to `8192` for IVF-family indexes (`IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ`) and `1,048,576` for IVF-HNSW-family indexes (`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`) — HNSW graphs are far more memory-intensive to train, so each partition is kept much larger to limit how many separate graphs get built. If both `num_partitions` and `target_partition_size` are set, `num_partitions` takes precedence. | -| `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if the dimension isn't evenly divisible by 16). Larger values often improve recall but can slow search. | +| `metric` | Default is `l2`, others available are `cosine` and `dot`. +| `num_partitions` | When left unset, targets roughly `sqrt(num_rows)`. | +| `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size`. | +| `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. | #### Search-time Parameters -Knobs available on a vector search call: - | Parameter | Description | | :--- | :--- | -| `limit` | Number of results to return (`k`). | +| `limit` | Number of results to return (the `k` in `k-ANN`). | | `nprobes` | Shorthand that sets both `minimum_nprobes` and `maximum_nprobes` to the same value. LanceDB auto-tunes this by default. | -| `minimum_nprobes` | Partitions that are *always* scanned. Higher values raise recall at the cost of latency. | -| `maximum_nprobes` | Upper bound on partitions scanned. The partitions above `minimum_nprobes` are only searched if the initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | -| `ef` | HNSW search-time exploration factor. Relevant for `IVF_HNSW_FLAT` and `IVF_HNSW_SQ`; start around `1.5 * k` and increase up to `10 * k` for higher recall. | -| `refine_factor` | Reads additional candidates and reranks them in memory to recover recall lost to quantization. | +| `minimum_nprobes` | Minimum number of partitions scanned. | +| `maximum_nprobes` | Maximum number of partitions scanned. Only scans more than `minimum_nprobes` if an initial pass does not return enough results — useful for narrow filters. Set to `0` to remove the cap. | +| `ef` | HNSW search-time exploration factor. Start around `1.5 * k` and increase up to `10 * k` for higher recall. | +| `refine_factor` | Reads and reranks additional candidates in memory to recover recall lost to quantization. | | `distance_range(lower_bound, upper_bound)` | Return only rows whose distance falls within `[lower_bound, upper_bound)`. Either bound is optional. Useful for near-duplicate detection or "close-enough" matching. | -| `bypass_vector_index()` | Skip the ANN index and perform an exhaustive (flat) scan. Primary uses: (1) compute ground-truth results to measure ANN recall@k, and (2) query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | +| `bypass_vector_index()` | Ignore the ANN index entirely and perform an exact (flat) scan. Can be used to measure ANN `recall@k`, or to query with a metric the index was not built for (e.g., a non-cosine query on a multivector column). | -We recommend the following `nprobes` behavior by index type: +**Recommended `nprobes` behavior by index type:** | Index type | Guidance | | :--- | :--- | @@ -134,18 +131,20 @@ We recommend the following `nprobes` behavior by index type: An example of a vector search exercising several of the above parameters. + +**Filtered queries and adaptive `nprobes`.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` +partitions and uses a wider scan if sufficiently few rows are found. +Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count. + + {VectorIndexNprobes} - -**Filtered queries and adaptive nprobes.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` -partitions and uses a wider scan if sufficiently few rows are found. -Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count. - -There are also several search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. + +There are also advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. **Thresholding with `distance_range`:** @@ -157,8 +156,7 @@ There are also several search-time controls for thresholded retrieval, recall me **Using `bypass_vector_index`:** -Use `bypass_vector_index` to compute an exact **kNN** result. Compare with ANN results to compute **recall@k**. -Recall that exact queries may be prohibitively slow on production scales. +Use `bypass_vector_index` to compute an exact **kNN** result. Note that exact queries may be prohibitively slow on production scales. @@ -167,12 +165,13 @@ Recall that exact queries may be prohibitively slow on production scales. -Multivector indexing currently requires `distance_type="cosine"` — `l2` is rejected at index-creation time. That restriction is why `bypass_vector_index()` is the escape hatch for non-cosine queries on a multivector column: the metric you want at query time cannot be served by the index, so you fall back to a flat scan. See [Multivector Search](/search/multivector-search) for the full rules. +Multivector indexing currently requires `distance_type="cosine"`. Use `bypass_vector_index()` for non-`cosine` queries on a multivector column. See [Multivector Search](/search/multivector-search) for the full rules. #### Async API and Config Objects -With asynchronous Python connections, create vector indexes with `await table.create_index("vector", config=...)`. The `config` object carries the same build-time choices described above, such as distance metric, partition count, and quantization settings — pass an instance of the Python config class from the table in [Choosing the Right Index](#choosing-the-right-index): +Create vector indexes asynchronously with `await table.create_index("vector", config=...)`. The `config` object +admits the same build-time parameters described above — pass an instance of a Python config class from the table in [Choosing the Right Index](#choosing-the-right-index): @@ -242,7 +241,7 @@ You can call `create_index()` to create a new index (this replaces existing inde -Be careful to use the same distance metric during index creation and search (`l2` by default). +Be careful to use the same distance metric during index creation and search. From 696618b0b0fb284f86f5217d127150a5c50179fd Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Wed, 26 Aug 2026 11:01:16 -0700 Subject: [PATCH 32/44] almost done --- docs/indexing/quantization.mdx | 34 ++++++++++++----------- docs/indexing/vector-index.mdx | 50 ++++++++++++++++++---------------- docs/snippets/indexing.mdx | 4 ++- 3 files changed, 47 insertions(+), 41 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index e1cbd8f..a86ebec 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -5,6 +5,9 @@ description: "Use quantization efficiently store your LanceDB vector index." icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- +import { + PyQuantizationCustomParams as QuantizationCustomParams, +} from '/snippets/indexing.mdx'; **Quantization** is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here; discussion of LanceDB vector indexes and quantized vector indexes can be found [here](/indexing/vector-index). @@ -62,19 +65,18 @@ In practice, it often uses $8$ bits per dimension of a vector, and supports very For example, suppose we know all vector entires across our dataset lie in the range $[-128 \times 10^5, 127 \times 10^5]$. In this case, we could quantize a given value $v$ as an $8$-bit representation of the integer $j$, where $j \times 10^5$ is the closest value to $v$ among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$. -## API Reference - -The full list of parameters to the algorithm are listed below. - -- `distance_type`: `Literal["l2", "cosine", "dot"]`. Default: `"l2"` - The distance metric used in comparison. -- `num_partitions`: `Optional[int]`. Default: `None` - Number of IVF partitions (clusters). High `num_partitions` increases both recall and build time. When unset, LanceDB chooses roughly the square root of the row count. -- `num_bits`: `int`. Default: `1` - Bits per dimension for quantization (`1` corresponds to RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage. -- `max_iterations`: `int`. Default: `50` - Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality. -- `sample_rate`: `int`. Default: `256` - Number of samples per partition during training. Higher values increase both accuracy and training time. -- `target_partition_size`: `Optional[int]`. Default: `None` - Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence. + +## Quantization API Reference + + +| Parameter | Description | +| :--- | :--- | +| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (default `1`, RaBitQ) — not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. | + +`max_iterations` and `sample_rate` also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they're documented as general [Build-time Parameters](/indexing/vector-index#build-time-parameters) instead. All three are passed as keyword arguments to `create_index`, alongside `index_type`: + + + + {QuantizationCustomParams} + + diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index f002d79..05275eb 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -107,6 +107,8 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | `num_partitions` | When left unset, targets roughly `sqrt(num_rows)`. | | `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size`. | | `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. | +| `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. | +| `sample_rate` | Number of training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. | #### Search-time Parameters @@ -181,9 +183,8 @@ admits the same build-time parameters described above — pass an instance of a ### IVF Indexes -This example creates an `IVF_PQ` index for a table of vectors under `cosine` similarity, then queries it with a random 1,536-dimensional embedding. Specify `vector_column_name` if you use multiple vector columns or non-default names. You can also switch `index_type` to `IVF_RQ` here without changing anything else. - -Partition sizing follows the general `num_partitions`/`target_partition_size` guidance above. The only IVF-specific knob is `num_sub_vectors` (`IVF_PQ` only): increase for better recall, decrease for faster search and smaller indexes. +This example creates and queries an `IVF_PQ` index for a table of vectors with respect to `cosine` similarity. +Specify `vector_column_name` if you use multiple vector columns or non-default names. @@ -229,38 +230,42 @@ The snippet below builds and queries an `IVF_HNSW_SQ` index. Enterprise-only In LanceDB Enterprise, vector indexes are managed **automatically**. The system asynchronously updates and optimizes indexes as a background process: -- Automatically manages indexing parameters -- Optimizes `IVF_PQ` storage without manual input +- Automatically manages indexing parameters and storage - Infers vector columns from the schema - Open-Source LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. LanceDB also provides an interface to tune parameters manually as data changes — see the IVF and IVF-HNSW sections above for index-specific tuning guidance. + Open-Source + LanceDB OSS users can manually create vector indexes by calling `table.create_index()`. + See the above sections for guidance on manually tuning index parameters as data changes. -You can call `create_index()` to create a new index (this replaces existing indexes). -`create_index()` returns immediately, but the vector index builds asynchronously. To wait until all data is indexed, specify the `wait_timeout` parameter, or call `wait_for_index(...)` afterward — it waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. +`create_index()` returns immediately, but the vector index builds asynchronously. +To wait until all data is indexed, specify the `wait_timeout` parameter, or call `wait_for_index(...)` afterward — +it waits for the named index to exist and for `index_stats(...)` to report `num_unindexed_rows == 0`. - -Be careful to use the same distance metric during index creation and search. - - Rows appended after an initial index build remain outside the index until refreshed manually (OSS) or automatically (Enterprise). Normal search still checks those unindexed rows with a slower fallback path; `fast_search()` skips that fallback and searches only indexed rows. + +**Operational checks** + +After appends or other writes, use `optimize()` to fold new rows into existing indexes. + #### Check Index Status -Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the **LanceDB Enterprise UI**. +Vector index creation runs in the background and may take some time to complete. +While it is ongoing, you can check its status through the API or the **LanceDB Enterprise UI**. In the LanceDB Enterprise UI, navigate to your table page - the "Index" column reflects each column's index status: it is blank when no index exists, shows an "in progress" label while the index is being built, and shows the index type once the build completes. -Programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built. +To check status programmatically, use `list_indices()` and `index_stats()`. **By default**, the index name is formed by appending `_idx` to the column name (e.g., a `keywords_embeddings` column produces `keywords_embeddings_idx`). Note that `list_indices()` only returns information after the index is fully built. Each entry returned by `list_indices()` also carries detailed per-index metadata, so you can inspect an index without a follow-up `index_stats()` call. Node.js exposes the same fields in camelCase (`num_indexed_rows` → `numIndexedRows`): -| Field | What it tells you | +| Parameter | Description | | :--- | :--- | | `num_indexed_rows`, `num_unindexed_rows` | Index coverage over the table | | `size_bytes` | Total size of the index files on disk | @@ -269,9 +274,6 @@ Each entry returned by `list_indices()` also carries detailed per-index metadata | `index_uuid`, `type_url` | Internal identifiers for the index segment | | `index_details` | Type-specific details (e.g. IVF partition counts or quantization settings) | - -These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them. - @@ -279,9 +281,14 @@ These fields are populated for local and embedded tables. On LanceDB Enterprise + + +These fields are populated for local and embedded tables. On LanceDB Enterprise remote tables they are returned as `None` / `undefined` until the server response surfaces them. + #### Custom Index Names -The `{column}_idx` suffix is a default convention, not the only supported naming path. Pass `name=...` to `create_index()` to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side `IVF_PQ` and `IVF_HNSW_SQ` builds) or when you script index replacement by name. Once set, `list_indices()`, `index_stats(name)`, and `wait_for_index([name])` all reference the custom name. +The `{column}_idx` suffix is the default naming convetion. +Pass `name=...` to `create_index()` to override it. Once set, the custom name will be reflected in `list_indices()`, `index_stats(name)`, and `wait_for_index([name])`. @@ -289,9 +296,4 @@ The `{column}_idx` suffix is a default convention, not the only supported naming - -**Operational checks** -After appends or other writes, use `optimize()` to fold new rows into existing indexes. -Unless specified otherwise, vector indexing defaults to `IVF_PQ`. - diff --git a/docs/snippets/indexing.mdx b/docs/snippets/indexing.mdx index 278a097..183150a 100644 --- a/docs/snippets/indexing.mdx +++ b/docs/snippets/indexing.mdx @@ -12,6 +12,8 @@ export const PyGpuIndexCuda = "table.create_index(\n num_partitions=256,\n export const PyGpuIndexMps = "table.create_index(\n num_partitions=256,\n num_sub_vectors=96,\n accelerator=\"mps\",\n)\n"; +export const PyQuantizationCustomParams = "table.create_index(\n index_type=\"IVF_RQ\",\n num_bits=2,\n max_iterations=100,\n sample_rate=512,\n)\n"; + export const PyReindexingIncremental = "table = db.open_table(\"reindexing_incremental\")\ntable.add([{\"vector\": [3.1, 4.1], \"text\": \"Frodo was a happy puppy\"}])\ntable.optimize()\n"; export const PyScalarIndexBuild = "tbl = db.open_table(\"scalar_index_build\")\ntbl.create_scalar_index(\"book_id\")\ntbl.create_scalar_index(\"publisher\", index_type=\"BITMAP\")\n"; @@ -48,7 +50,7 @@ export const PyVectorIndexBinarySearch = "query = np.random.randint(0, 2, size=n export const PyVectorIndexBuildHnsw = "table.create_index(index_type=\"IVF_HNSW_SQ\")\n"; -export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n metric=\"cosine\",\n vector_column_name=\"keywords_embeddings\",\n)\n"; +export const PyVectorIndexBuildIvf = "table_name = \"vector-index-build-ivf\"\ntable = db.open_table(table_name)\ntable.create_index(\n metric=\"cosine\",\n vector_column_name=\"keywords_embeddings\",\n index_type=\"IVF_PQ\",\n)\n"; export const PyVectorIndexBypassRecall = "query = np.random.random(128)\nk = 10\n\n# Ground truth: flat (exhaustive) scan, ignoring the ANN index.\ntruth = set(table.search(query).bypass_vector_index().limit(k).to_pandas()[\"id\"])\n\n# ANN results with the current nprobes setting.\nann = set(table.search(query).nprobes(20).limit(k).to_pandas()[\"id\"])\n\nrecall_at_k = len(truth & ann) / k\n"; From 582f63c70b91fcdd1ab1b1ec943b35dd441ba894 Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 10:20:12 -0700 Subject: [PATCH 33/44] Apply batched suggestions from code review Co-authored-by: Weston Pace --- docs/indexing/quantization.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index a86ebec..168c270 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -1,7 +1,7 @@ --- title: "Quantization" sidebarTitle: "Quantization" -description: "Use quantization efficiently store your LanceDB vector index." +description: "Use quantization to efficiently store your LanceDB vector index." icon: "compress" keywords: ["quantization", "quantize", "rabitq"] --- From f1264c93f5624e266203852a6e8442b7f2b87cbf Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 14:40:59 -0700 Subject: [PATCH 34/44] Apply suggestion from @westonpace Co-authored-by: Weston Pace --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 05275eb..426edc6 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -71,7 +71,7 @@ for a closest match, then run a brute-force comparison against its resulting clu giving an approximate ANN result. However, observe that a queried vector may lie near the boundary of $2$ or more clusters; thus, the true nearest neighbors are scattered across several different clusters. - To address this, LanceDB exposes the `nprobe` parameter, which specifies the number of clusters searched (`default = 1`). + To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of clusters searched (`default = 1`). A high `nprobe` parameter will yield more accurate results at slightly higher runtime. From 24082e54be35f737e30cafe4dd6d3b5baffc0976 Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 14:41:14 -0700 Subject: [PATCH 35/44] Apply suggestion from @westonpace Co-authored-by: Weston Pace --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 426edc6..3dea831 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -72,7 +72,7 @@ giving an approximate ANN result. However, observe that a queried vector may lie near the boundary of $2$ or more clusters; thus, the true nearest neighbors are scattered across several different clusters. To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of clusters searched (`default = 1`). -A high `nprobe` parameter will yield more accurate results at slightly higher runtime. +A high `nprobes` parameter will yield more accurate results at slightly higher runtime. IVF vector-space partitioning From 24875599053856e25dfaec5780204be1e96458ee Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 14:42:56 -0700 Subject: [PATCH 36/44] Apply suggestion from @westonpace Co-authored-by: Weston Pace --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 3dea831..ff60cb5 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -108,7 +108,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size`. | | `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. | | `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. | -| `sample_rate` | Number of training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. | +| `sample_rate` | Number of k-means training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. | #### Search-time Parameters From 4de1f8d3ae132d6ec9c96f38f9af1b6b991494d8 Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 14:43:51 -0700 Subject: [PATCH 37/44] Apply suggestion from @westonpace Co-authored-by: Weston Pace --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index ff60cb5..1c4e7dd 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -184,7 +184,7 @@ admits the same build-time parameters described above — pass an instance of a ### IVF Indexes This example creates and queries an `IVF_PQ` index for a table of vectors with respect to `cosine` similarity. -Specify `vector_column_name` if you use multiple vector columns or non-default names. +Specify `vector_column_name` if you have multiple vector columns or non-default names. From acb019fa02704e6a44b16bdf575112ee146c2e43 Mon Sep 17 00:00:00 2001 From: Jason Zheng Date: Fri, 28 Aug 2026 14:44:21 -0700 Subject: [PATCH 38/44] Apply suggestion from @westonpace Co-authored-by: Weston Pace --- docs/indexing/vector-index.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 1c4e7dd..cbb6c50 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -128,8 +128,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | Index type | Guidance | | :--- | :--- | | `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | -| `IVF_RQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | -| `IVF_PQ` | Keep auto-tuned `nprobes`; raise when recall is insufficient. Often preferred over `IVF_RQ` when `dimension <= 256`. | +| `IVF_RQ`, `IVF_PQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | An example of a vector search exercising several of the above parameters. From 7c5e60909a93e132fef113f8fbefeefb6e2ac910 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 11:41:45 -0700 Subject: [PATCH 39/44] a --- out.txt | 172 -------------------------------------- tests/py/test_indexing.py | 20 +++++ 2 files changed, 20 insertions(+), 172 deletions(-) delete mode 100644 out.txt diff --git a/out.txt b/out.txt deleted file mode 100644 index f0973e4..0000000 --- a/out.txt +++ /dev/null @@ -1,172 +0,0 @@ -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 1) --- -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 2) title: "Indexing Data" -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 3) sidebarTitle: "Overview" -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 4) description: "Optimize search performance in LanceDB using vector indexes, full-text search, and scalar indexes. Understand IVF-PQ indexing for efficient vector similarity search." -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 5) icon: "list" -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 6) --- -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 7) -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 8) An **index** is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads: -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 9) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 10) - **Vector Index**: Optimized for searching high-dimensional data (like images, audio, or text embeddings) by efficiently finding the most similar vectors -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 11) - **Full-Text Search Index**: Enables fast keyword-based searches by indexing words and phrases -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 12) - **Scalar Index**: Accelerates filtering and sorting of structured numeric or categorical data (e.g., timestamps, prices) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 13) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 14) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 15) Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 16) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 17) - Vector search (prefilter or post-filter results using metadata) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 18) - Full-text search (combining keyword matching with structured filters) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 19) - SQL scans (optimizing WHERE clauses on scalar columns) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 20) - Key-value lookups (enabling rapid primary key-based retrievals) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 21) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 22) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 23) ## Supported Index Types -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 24) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 25) LanceDB provides a comprehensive suite of indexes for different data types and use cases: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 26) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 27) | Index | Use Case | Description | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 28) | :--------- | :------- | :---------- | -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 29) | `IVF` (Vector) | Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance. | Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search.
Distance metrics: $\ell_2$ `cosine` `dot` `hamming`
Quantizations: `None/Flat` `PQ` `SQ` `RQ`| -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 30) | `IVF_HNSW` (Vector) | Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW. | Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability.
Distance metrics: $\ell_2$ `cosine` `dot`
Quantizations: `None/Flat` `SQ` `PQ`| -18d34126 docs/indexing/index.mdx (Prashanth Rao 2026-02-18 17:38:03 -0500 31) | `FTS` (Full-text search) | String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking. | Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing. | -25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 32) | `BTree` (Scalar) | Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates. | Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads. | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 33) | `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. | -25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 34) | `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_contains_all` or `array_contains_any` filters. | Scalar index for `List` and `LargeList` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. | -25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 35) | `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 36) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 37) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 38) TypeScript currently doesn't support `IvfSq` (IVF with Scalar Quantization). -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 39) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 40) -fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 41) -fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 42) **Operational checks** -fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 43) -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 44) For vector indexes, make sure to use the same distance metric when creating and querying the index. After appends or other writes, use `optimize()` to fold new rows into existing indexes, then check `index_stats(...)` or `wait_for_index(...)` to confirm that the index has caught up. -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 45) `wait_for_index(...)` waits until the named indexes exist and report `num_unindexed_rows == 0`, and can time out if writes keep adding unindexed rows. -25d7ccd1 docs/indexing/index.mdx (Prashanth Rao 2026-07-21 13:47:12 -0400 46) -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 47) Unless specified otherwise, vector indexing defaults to `IVF_PQ`, and scalar index creation defaults to -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 48) `BTree`. `BTree` and `Bitmap` indexes target scalar columns, not list columns; use `LabelList` for list containment filters. -fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 49) -fa69074f docs/indexing/index.mdx (Prashanth Rao 2026-05-23 21:09:50 -0400 50) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 51) ### Quantization Types -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 52) -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 53) Vector indexes use different quantization methods to compress vectors and improve search performance: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 54) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 55) | Quantization | Use Case | Description | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 56) | :----------- | :------- | :---------- | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 57) | `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 58) | `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. | -5a45fbe7 docs/indexing/index.mdx (BubbleCal 2026-02-28 07:36:05 +0800 59) | `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 60) | `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. | -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 61) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 62) ## Understanding the IVF-PQ Index -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 63) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 64) An ANN (Approximate Nearest Neighbors) index is a data structure that quickly produces an approximate solution to the **$k$-Nearest Neighbors (kNN)** problem. -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 65) It greatly improves upon the runtime of a brute-force kNN search, while admitting a slight decrease in accuracy. LanceDB uses the disk-based indexing technique IVF-PQ, discussed below. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 66) -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 67) LanceDB differs from other vector databases in that it is built on top of [Lance](https://github.com/lancedb/lance), an open-source columnar data format designed for performant ML workloads and fast random access. Due to the design of Lance, LanceDB's indexing philosophy adopts a primarily *disk-based* indexing philosophy. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 68) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 69) ## IVF-PQ -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 70) -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 71) LanceDB uses **IVF-PQ** indexing, which combines the clustering-based **Inverted File Index (IVF)** with **Product Quantization (PQ)** to efficiently compress embeddings. -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 72) The implementation provides several parameters to fine-tune the index's size, query throughput, latency, and recall. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 73) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 74) ### Product Quantization -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 75) -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 76) Quantization is a compression technique used to speed up search by reducing the dimensionality of an embedding. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 77) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 78) Product quantization (PQ) first projects each large, high-dimensional vector into equal-sized subvectors. Each subvector is assigned a "reproduction value" that maps to the nearest centroid of points for that subvector. -220338a1 docs/indexing/index.mdx (jasonz-lance 2026-08-03 13:49:01 -0700 79) The reproduction values are then assigned to a codebook using unique IDs, which can be used to reconstruct the original vector. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 80) -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 81) ![](/static/assets/images/indexing/ivfpq_pq_desc.png) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 82) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 83) As an example, consider the above image, which visualizes quantizing a 128-dimensional vector of 32-bit integers into a 4-dimensional vector of 8-bit integers. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 84) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 85) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 86) Original storage: `128 × 32 = 4096` bits. -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 87) Quantized storage: `4 × 8 = 32` bits. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 88) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 89) In this example, quantization achieves a **128x** reduction in the memory requirement of each indexed vector. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 90) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 91) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 92) It's important to remember that quantization is a *lossy process*, i.e., that no operation on the reconstructed vector can exactly recover the original vector. -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 93) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 94) ### Inverted File Index (IVF) Implementation -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 95) -86d2e2d3 docs/indexing/index.mdx (jasonz-lance 2026-08-04 11:15:22 -0700 96) (note: acknowledge pq in here somewhere) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 97) -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 98) An IVF is an index that facilitates rapid nearest neighbor searches by drastically reducing the search space. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 99) -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 100) Given a large set of stored vectors, the algorithm to produce an IVF index first computes a set of *centroids* corresponding to an approximate solution to the $k$-means clustering problem. -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 101) The centroids are then used to partition the set of vectors as follows: each vector is assigned to the centroid nearest to it in the $\ell_2$ (or user-specified) metric. -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 102) The set of vectors assigned to a centroid is called its *cluster*. This data is then recorded as an index which identifies each centroid with its cluster. -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 103) -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 104) The following image shows a $2$-dimensional Euclidean space partitioned according to this algorithm. The colored marks denote centroids. -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 105) ![](/static/assets/images/indexing/ivfpq_ivf_desc.webp) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 106) -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 107) To process a nearest neighbors query, instead of a brute-force comparison of the queried vector to every stored vector, the system can instead search the much-smaller set of *centroids$ -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 108) for a closest match, then execute a brute-force comparison against its associated cluster. This technique quickly eliminates the vast majority of clusters from the search space. -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 109) Furthermore, since each centroid is relatively close to points in its cluster, we are likely to produce an approximately correct result. -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 110) -bc4d53a0 docs/indexing/index.mdx (jasonz-lance 2026-08-03 17:07:50 -0700 111) here vv -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 112) During query time, depending on where the query lands in vector space, it may be close to the border of multiple Voronoi cells, which could make the top-k results ambiguous and span across multiple cells. To address this, the IVF-PQ introduces the `nprobe` parameter, which controls the number of Voronoi cells to search during a query. The higher the `nprobe`, the more accurate the results, but the slower the query. -e7716261 docs/indexing/index.mdx (Prashanth Rao 2025-12-10 22:00:28 -0500 113) ![](/static/assets/images/indexing/ivfpq_query_vector.webp) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 114) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 115) ## HNSW Index Implementation -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 116) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 117) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. HNSW is one of the most accurate and fastest Approximate Nearest Neighbour search algorithms, It's beneficial in high-dimensional spaces where finding the same nearest neighbor would be too slow and costly. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 118) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 119) ### Types of ANN Search Algorithms -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 120) -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 121) Approximate Nearest Neighbor (ANN) search is a method for finding data points near a given point in a dataset, though not always the exact nearest one. -d6503749 docs/indexing/index.mdx (jasonz-lance 2026-08-03 11:54:22 -0700 122) For example, HNSW is an ANN index that performs well in high-dimensional spaces where other techniques prove too slow and costly. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 123) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 124) There are three main types of ANN search algorithms: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 125) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 126) * **Tree-based search algorithms**: Use a tree structure to organize and store data points. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 127) * **Hash-based search algorithms**: Use a specialized geometric hash table to store and manage data points. These algorithms typically focus on theoretical guarantees, and don't usually perform as well as the other approaches in practice. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 128) * **Graph-based search algorithms**: Use a graph structure to store data points, which can be a bit complex. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 129) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 130) HNSW is a graph-based algorithm. All graph-based search algorithms rely on the idea of a $k$-nearest neighbor (or $k$-approximate nearest neighbor) graph, which we outline below. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 131) HNSW also combines this with the ideas behind a classic 1-dimensional search data structure: the skip list. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 132) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 133) ### Understanding $k$-Nearest Neighbor Graphs -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 134) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 135) The $k$-nearest neighbor graph actually predates its use for ANN search. Its construction is quite simple: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 136) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 137) * Each vector in the dataset is given an associated vertex. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 138) * Each vertex has outgoing edges to its k nearest neighbors. That is, the k closest other vertices by Euclidean distance between the two corresponding vectors. This can be thought of as a "friend list" for the vertex. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 139) * For some applications (including nearest-neighbor search), the incoming edges are also added. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 140) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 141) Eventually, it was realized that the following greedy search method over such a graph typically results in good approximate nearest neighbors: -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 142) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 143) * Given a query vector, start at some fixed "entry point" vertex (e.g. the approximate center node). -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 144) * Look at that vertex's neighbors. If any of them are closer to the query vector than the current vertex, then move to that vertex. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 145) * Repeat until a local optimum is found. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 146) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 147) The above algorithm also generalizes to e.g. top 10 approximate nearest neighbors. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 148) -119b8572 docs/indexing/index.mdx (jasonz-lance 2026-08-03 16:09:58 -0700 149) Computing a $k$-nearest neighbor graph is actually quite slow, taking quadratic time in the dataset size. It was quickly realized that near-identical performance can be achieved using a k-approximate nearest neighbor graph. That is, instead of obtaining the $k$-nearest neighbors for each vertex, an approximate nearest neighbor search data structure is used to build much faster. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 150) In fact, another data structure is not needed: This can be done "incrementally". -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 151) That is, if you start with a k-ANN graph for n-1 vertices, you can extend it to a k-ANN graph for n vertices as well by using the graph to obtain the k-ANN for the new vertex. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 152) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 153) One downside of k-NN and k-ANN graphs alone is that one must typically build them with a large value of k to get decent results, resulting in a large index. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 154) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 155) ### Hierarchical Navigable Small Worlds (HNSW) -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 156) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 157) HNSW builds on k-ANN in two main ways: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 158) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 159) * Instead of getting the k-approximate nearest neighbors for a large value of k, it sparsifies the k-ANN graph using a carefully chosen "edge pruning" heuristic, allowing for the number of edges per vertex to be limited to a relatively small constant. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 160) * The "entry point" vertex is chosen dynamically using a recursively constructed data structure on a subset of the data, similarly to a skip list. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 161) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 162) This recursive structure can be thought of as separating into layers: -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 163) -74a79fe3 docs/indexing/index.mdx (Justin Miller 2026-04-09 12:09:35 -0700 164) * At the bottom-most layer, a k-ANN graph on the whole dataset is present. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 165) * At the second layer, a k-ANN graph on a fraction of the dataset (e.g. 10%) is present. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 166) * At the Lth layer, a k-ANN graph is present. It is over a (constant) fraction (e.g. 10%) of the vectors/vertices present in the L-1th layer. -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 167) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 168) Then the greedy search routine operates as follows: -9f362963 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-11-30 21:12:29 -0800 169) -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 170) * At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 171) * Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method. -5918c400 docs/user-guides/indexing/index.mdx (erik-wang-lancedb 2025-12-09 21:55:30 -0800 172) * Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10). diff --git a/tests/py/test_indexing.py b/tests/py/test_indexing.py index 7f63e8f..bfe09b1 100644 --- a/tests/py/test_indexing.py +++ b/tests/py/test_indexing.py @@ -63,6 +63,7 @@ def test_vector_index_build_ivf(tmp_db): table.create_index( metric="cosine", vector_column_name="keywords_embeddings", + index_type="IVF_PQ", ) # --8<-- [end:vector_index_build_ivf] @@ -294,6 +295,25 @@ def test_vector_index_hnsw(tmp_db): assert len(df) == 2 +def test_quantization_custom_params(tmp_db): + table = tmp_db.create_table( + "quantization-custom-params", + _make_vector_rows(256, 64), + mode="overwrite", + ) + + # --8<-- [start:quantization_custom_params] + table.create_index( + index_type="IVF_RQ", + num_bits=2, + max_iterations=100, + sample_rate=512, + ) + # --8<-- [end:quantization_custom_params] + + assert table.list_indices() + + def test_vector_index_binary(tmp_db): table_name = "hamming-index-tbl" ndim = 256 From 40070c64b7cc7fe5621e44661c84fa2afec78a42 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 12:01:29 -0700 Subject: [PATCH 40/44] a --- docs/indexing/quantization.mdx | 6 ++++-- docs/indexing/scalar-index.mdx | 2 +- docs/indexing/vector-index.mdx | 12 ++++++------ 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/indexing/quantization.mdx b/docs/indexing/quantization.mdx index 168c270..9f9c251 100644 --- a/docs/indexing/quantization.mdx +++ b/docs/indexing/quantization.mdx @@ -14,6 +14,8 @@ discussion of LanceDB vector indexes and quantized vector indexes can be found [ ## Quantization Techniques LanceDB provides $3$ distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ). +Recall that all quantizations perform **lossy** compression, in that they irreversibly lose some degree of precision in order to +compactly store an index. ### Product Quantization (PQ) To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has @@ -39,7 +41,7 @@ which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of q RaBitQ is an advanced quantization technique that outperforms PQ in several ways. It needs no codebook to train, estimates distances very quickly at query-time, and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!** -In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few hundred bytes, while maintaining good recall. +In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few thousand bits, while maintaining good recall. The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube, @@ -47,7 +49,7 @@ where $d$ is the dimensionality of the dataset. We defer the details, and an ele #### Using RaBitQ Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`. -Note that when using `IVF_RQ`, the dimension of the dataset should be a multiple of `8`. +Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`. `num_bits` determines how many bits are used to quantize each dimension. `1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute. diff --git a/docs/indexing/scalar-index.mdx b/docs/indexing/scalar-index.mdx index a417e05..c5a511d 100644 --- a/docs/indexing/scalar-index.mdx +++ b/docs/indexing/scalar-index.mdx @@ -30,7 +30,7 @@ LanceDB supports four types of scalar indexes: Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: -- Vector search (prefilter or post-filter results using metadata) +- Vector search (prefilter results using metadata) - Full-text search (combining keyword matching with structured filters) - SQL scans (optimizing WHERE clauses on scalar columns) - Key-value lookups (enabling rapid primary key-based retrievals) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index cbb6c50..9111bef 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -43,7 +43,7 @@ Lance with `index_type`. | | `IVF_HNSW_PQ` | IVF-HNSW indexing with product quantization | Varies | `IvfHnswPq` | | Best recall/latency trade-off | `IVF_HNSW_SQ` | IVF-HNSW indexing with scalar | Typically a little larger than `1/4` of raw size | `IvfHnswSq` | | Highest recall / no quantization | `IVF_HNSW_FLAT` | IVF-HNSW indexing with no quantization | Around raw vector size plus HNSW graph overhead | `IvfHnswFlat` | -| | `IVF_FLAT` | IVF indexing with no quantization | Varies | `IvfFlat` | +| | `IVF_FLAT` | IVF indexing with no quantization | `1` | `IvfFlat` | @@ -61,20 +61,20 @@ The **Inverted File Index (IVF)** accelerates ANN searches by drastically reduci a small set of *centroids* corresponding to an approximate solution to the [$k$-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) problem. Each vector remembers its nearest centroid, and each centroid remembers its associated set of vectors, -called its *cluster* or *partition*. +called its *partition*. IVF vector-space partitioning At query time, we can compare the queried vector to the smaller set of *centroids* (as opposed to the entire dataset) -for a closest match, then run a brute-force comparison against its resulting cluster. This technique quickly prunes a large search space, +for a closest match, then run a brute-force comparison against its resulting partition. This technique quickly prunes a large search space, giving an approximate ANN result. -However, observe that a queried vector may lie near the boundary of $2$ or more clusters; thus, the true nearest neighbors are scattered across several different clusters. - To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of clusters searched (`default = 1`). +However, observe that a queried vector may lie near the boundary of $2$ or more partitions; thus, the true nearest neighbors are scattered across several different partitions. + To address this, LanceDB exposes the `nprobes` parameter, which specifies the number of partitions searched. A high `nprobes` parameter will yield more accurate results at slightly higher runtime. - + IVF vector-space partitioning From c11d2d55d8b11c4f53674d48f3102d5db9b9f6f0 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 12:15:21 -0700 Subject: [PATCH 41/44] a --- docs/indexing/vector-index.mdx | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 9111bef..2ffcd19 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -104,7 +104,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | Parameter | Description | | :--- | :--- | | `metric` | Default is `l2`, others available are `cosine` and `dot`. -| `num_partitions` | When left unset, targets roughly `sqrt(num_rows)`. | +| `num_partitions` | The number of IVF partitions constructed (corresponds to the $k$ in $k$-means clustering). Targets roughly `sqrt(num_rows)` by default. | | `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size`. | | `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. | | `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. | @@ -130,7 +130,6 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | `IVF_HNSW_FLAT`, `IVF_HNSW_SQ` | Keep the auto-tuned `nprobes`, then tune `ef` first. Expect higher latency variance under filtered search. | | `IVF_RQ`, `IVF_PQ` | Keep auto-tuned `nprobes`; raise only when recall is insufficient. | -An example of a vector search exercising several of the above parameters. **Filtered queries and adaptive `nprobes`.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` @@ -138,6 +137,8 @@ partitions and uses a wider scan if sufficiently few rows are found. Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count. +Here is an example of a vector search exercising several of the above parameters. + {VectorIndexNprobes} @@ -145,7 +146,7 @@ Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the -There are also advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. +LanceDB also supports advanced search-time controls for thresholded retrieval, recall measurement, and working around index-level metric constraints. **Thresholding with `distance_range`:** From 1d1ee81c8a47abc8473f82e38237f651d8fc3db3 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 12:22:46 -0700 Subject: [PATCH 42/44] a --- docs/indexing/vector-index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 2ffcd19..4bc6e87 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -133,7 +133,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil **Filtered queries and adaptive `nprobes`.** When a `where(...)` filter is active, LanceDB initially scans `minimum_nprobes` -partitions and uses a wider scan if sufficiently few rows are found. +partitions and uses a wider scan if too few rows are found. Set `minimum_nprobes == maximum_nprobes` or call `nprobes(n)` to instead fix the partition count. From 273c87fe0f392e20095759a1dfa9c9cbe1896a5c Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 13:35:46 -0700 Subject: [PATCH 43/44] a --- docs/indexing/vector-index.mdx | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 4bc6e87..4a18dfc 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -105,7 +105,7 @@ Learn how to configure, build, and search LanceDB vector indexes, including buil | :--- | :--- | | `metric` | Default is `l2`, others available are `cosine` and `dot`. | `num_partitions` | The number of IVF partitions constructed (corresponds to the $k$ in $k$-means clustering). Targets roughly `sqrt(num_rows)` by default. | -| `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size`. | +| `target_partition_size` | An alternative IVF sizing knob that derives the partition count by setting the number of rows per partition. Defaults to `8192 = 2^13` for IVF-family indexes and `1,048,576 = 2^20` for IVF-HNSW-family indexes. `num_partitions` takes precedence over `target_partition_size` if both are set. | | `num_sub_vectors` | Applies to `IVF_PQ`; defaults to `dimension // 16` (or `dimension // 8` if not a multiple of 16). Larger values produce better recall and slower search. | | `max_iterations` | Maximum number of k-means training iterations, for every IVF/HNSW index type. Default `50`. Increase for larger datasets or to improve training quality. | | `sample_rate` | Number of k-means training samples per partition, for every IVF/HNSW index type. Default `256`. Higher values increase both accuracy and training time. | @@ -204,10 +204,6 @@ For a vector field nested inside a struct, pass its full dotted path as `vector_ - -Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `image.thumbnail.embedding`). - - ### IVF-HNSW Indexes Beyond the general build-time parameters above, two additional parameters are specific to IVF-HNSW indexes: From 92b43ba849cc5fc193db55dedd7e26f0c4a045c5 Mon Sep 17 00:00:00 2001 From: jasonz-lance Date: Mon, 31 Aug 2026 14:53:21 -0700 Subject: [PATCH 44/44] hnsw stuff --- docs/indexing/vector-index.mdx | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/indexing/vector-index.mdx b/docs/indexing/vector-index.mdx index 4a18dfc..24deb4f 100644 --- a/docs/indexing/vector-index.mdx +++ b/docs/indexing/vector-index.mdx @@ -79,18 +79,33 @@ A high `nprobes` parameter will yield more accurate results at slightly higher r ### HNSW -{/* -The **Hierarchical Navigable Small World (HNSW)** index ... (todo. to consider: mention skip list?) -*/} -Coming soon! +**Hierarchical Navigable Small World (HNSW)** constructs a layered graph hierarchy on the vector set, with edges representing distances. +We can visualize an HNSW index as a vertical stack of graphs, with the top layer having very few edges and +each other layer having a multiplicative factor more edges than the layer above it. +High layers of a HNSW hierarchy represent sparse, higher-distance networks, and lower layers represent finer, lower-distance networks. +To query a vector $q$, the index proceeds iteratively through layers, first finding $q$'s nearest neighbor in the graph, then proceeding recursively +through the induced subhierarchy until the lowest layer is reached. + + + HNSW layered graph hierarchy + + +To visualize this process, imagine that you must drive your car from San Francisco to a specific house in Boston. Initially, you must first drive thousands of miles +on the interstate freeway I-90 E. Eventually, you merge onto the Massachusetts Turnpike, the center +of the greater Boston highway system. From there, you use a series of increasingly smaller, narrower roads within the city (Charles River Bridge, St. Paul St, +Thatcher St) before finally reaching the house. + +A key observation is that you must initially travel far distances through long-distance road networks +(the interstate freeway system), before proceeding to finer and finer road networks (greater Boston highway system, central Brookline neighborhood connectors) before finally reaching +your destination. This iterative series of road networks mimics the layered graph traversals performed by a HNSW query. **IVF + HNSW** In LanceDB, HNSW is not exposed as a top-level vector index. Instead, it's available as a substructure which further indexes the selected vectors inside each IVF partition. This combines the scalability of IVF with the high recall of HNSW. -LanceDB supports quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW_SQ`. +LanceDB supports IVF-HNSW-based quantized indexes `IVF_HNSW_FLAT`, `IVF_HNSW_PQ`, and `IVF_HNSW_SQ`.