Update indexing and quantization docs

docs/docs.json

@@ -18,6 +18,7 @@
     "family": "Inter"
   },
   "styling": {
+    "latex": true,
     "codeblocks": {
       "theme": {
         "light": "vitesse-light",
@@ -108,10 +109,10 @@
                 "pages": [
                   "indexing/index",
                   "indexing/vector-index",
+                  "indexing/quantization",
                   "indexing/fts-index",
                   "indexing/scalar-index",
                   "indexing/gpu-indexing",
-                  "indexing/quantization",
                   "indexing/reindexing"
                 ]
               },

docs/indexing/quantization.mdx

@@ -1,79 +1,84 @@
 ---
 title: "Quantization"
 sidebarTitle: "Quantization"
-description: "Learn about quantization when creating an index in LanceDB."
+description: "Use quantization to efficiently store your LanceDB vector index."
 icon: "compress"
 keywords: ["quantization", "quantize", "rabitq"]
 ---
+import {
+    PyQuantizationCustomParams as QuantizationCustomParams,
+} from '/snippets/indexing.mdx';
 
-Quantization compresses high-dimensional float vectors into a smaller, approximate representation, where instead of storing every vector as a float32 or float64, it's stored in compressed form, without too much of a compromise in search quality.
+**Quantization** is used in LanceDB to efficiently compress and store vector indexes. We discuss only the quantization techniques here;
+discussion of LanceDB vector indexes and quantized vector indexes can be found [here](/indexing/vector-index).
 
-Use quantization when:
+## Quantization Techniques
+LanceDB provides $3$ distinct quantization techniques: Product Quantization (PQ), RaBitQ Quantization (RQ), and Scalar Quantization (SQ).
+Recall that all quantizations perform **lossy** compression, in that they irreversibly lose some degree of precision in order to 
+compactly store an index.
 
-- You have a large dataset with relatively high-dimensional vectors (512, 768, 1024+)
-- Index build time and query latency matter
+### Product Quantization (PQ)
+To visualize PQ, assume a vector dataset has $d$ dimensions, with a **chunk** of a vector denoting a contiguous block of entries. Imagine that each vector has
+ $m$ disjoint chunks of $d/m$ entries each, where the first chunk represents entries $0$ through $d-1$, the second contains entries $d$ through $2d - 1$, and so on.
 
-LanceDB currently exposes multiple quantized vector index types, including:
-- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
-- `IVF_SQ` -- Inverted File index with Scalar Quantization. This is available in Python and Rust; TypeScript does not currently expose `IvfSq`.
-- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
-- `IVF_HNSW_SQ` -- IVF partitions with an **HNSW graph per partition** plus **Scalar Quantization**. Strong recall/latency/size trade-off for most workloads.
-- `IVF_HNSW_PQ` -- IVF partitions with an **HNSW graph per partition** plus **Product Quantization**. Prefer when PQ-level compression matters and you still want HNSW-style in-partition search.
+Now, for each $i$, let $S_i$ represent the set containing chunk $i$ of each vector (entries $(i-1)d$ to $di - 1$).
+For each $S_i$ independently, a small set of **centroids** is computed corresponding to an approximate solution to the $k$-means clustering problem.
 
-Two axes are being combined here: whether partitions are searched flatly or via an HNSW graph (`IVF_*` vs. `IVF_HNSW_*`), and which quantizer compresses the vectors (`PQ`, `RQ`, or `SQ`). `IVF_PQ` is the default and works well in many cases. For more drastic compression, RaBitQ (`IVF_RQ`) is a reasonable option. For higher recall at low latency, the HNSW-backed variants are usually the right pick. The ["Choose the Right Index"](/indexing/vector-index#choose-the-right-index) table on the vector indexing page is the canonical decision tool.
+For each original vector in the dataset, every chunk is associated to its nearest centroid; thus, the vector itself is associated with the concatenation of $m$ centroids.
+We then gather all our centroids (from all chunks) into a single lookup table, and for each vector, we store the concatenation of IDs of its corresponding centroids. 
+At query time, we need only to compare each chunk of the queried vector with our set of centroids.
 
-Use the same distance metric when training the index and running queries against it. For IVF-based indexes, `num_partitions` controls the number of groups and `sample_rate` controls how many training vectors are sampled per partition, so the training sample is roughly `sample_rate * num_partitions`.
+<Frame caption="">
+  <img src="/static/assets/images/indexing/ivfpq_pq_desc.png" alt="IVF vector-space partitioning" />
+</Frame>
 
-## RaBitQ quantization
+In the above example, the original vector is split into chunks (subvectors), each of which is associated to a centroid. The stored quantization code for this vector 
+is the concatenation $2 || 1 || 4 || 3$. The original vector required $128$ dimensions $\times 32$-bit integers $ =4096$ bits total,
+which has been compressed to $4$ chunks $\times 8$-bit integers $= 32$ bits of quantized storage.
 
-RaBitQ is a binary quantization method that represents each normalized embedding using **1 bit per dimension**, plus a couple of small corrective scalars. In practice, a 1,024-dimensional `float32` vector that would normally take 4 KB can be compressed to roughly a few hundred bytes with RaBitQ, while still maintaining reasonable recall.
+### RaBitQ Quantization (RQ)
 
-### How RaBitQ works
+RaBitQ is an advanced quantization technique that outperforms PQ in several ways.
+It needs no codebook to train, estimates distances very quickly at query-time, 
+and crucially, quantizes each vector in (with some small overhead) just **one bit per dimension!** 
+In practice, RaBitQ compresses a $1024$-dimensional `float32` vector into just a few thousand bits, while maintaining good recall.
 
-- Embeddings are grouped around centroids (as in other IVF indexes).
-- Each residual vector is normalized and mapped to the nearest vertex of a randomly rotated hypercube on the unit sphere.
-- The sign pattern of that vector is stored as bits (1 bit per dimension).
-- Two small corrective factors are stored:
-  1. The distance from the original vector to its centroid
-  2. The dot product between the normalized vector and its quantized version
+The inner workings of RaBitQ quantization are rather mathematically dense. It generates a quantization codebook by 
+applying a uniformly random, approximately distance-preserving orthogonal transformation of the vertices of the $d$-dimensional hypercube, 
+where $d$ is the dimensionality of the dataset. We defer the details, and an elegant theoretical error bound, to [the original paper.](https://arxiv.org/pdf/2405.12497)
 
-Compared to `IVF_PQ`, RaBitQ:
-- Avoids training expensive PQ codebooks
-- Builds indexes faster and handles updates more easily
-- Maintains or improves recall at high dimensionality under the same storage budget
+#### Using RaBitQ
+Use RaBitQ quantization by selecting quantized index types ending in the suffix `RQ`. For example, call `create_index` with `index_type="IVF_RQ"`. 
+Note that when using `IVF_RQ`, the dimension of the dataset must be a multiple of `8`.
 
-For a deeper dive into the theory and some benchmark results, see the blog post: [LanceDB's RaBitQ Quantization for Blazing Fast Vector Search](https://lancedb.com/blog/feature-rabitq-quantization/).
+`num_bits` determines how many bits are used to quantize each dimension.
+`1` is the standard RaBitQ setting. Increase to `2`, `4`, or `8` bits to achieve better recall for additional storage and query-time compute.
 
-### Using RaBitQ
+<Warning title="Reading multi-bit indexes across versions">
+RaBitQ-quantized indexes computed with `num_bits >= 2` use a newer on-disk layout, and cannot be read by some older LanceDB versions. 
+</Warning>
 
-You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`. 
+See this [blog post](https://lancedb.com/blog/feature-rabitq-quantization/) for further discussion and benchmarking of LanceDB's RaBitQ implementation.
 
-<Note title="Dimension requirement">
-When using `IVF_RQ`, vector dimensions must be divisible by `8`.
-</Note>
+### Scalar Quantization (SQ)
+Scalar quantization quantizes each entry of a vector independently, by simply replacing it with the closest of a pre-defined set of values. 
+In practice, it often uses $8$ bits per dimension of a vector, and supports very fast encoding and decoding.
 
-`num_bits` controls how many bits per dimension are used:
+For example, suppose we know all vector entires across our dataset lie in the range $[-128 \times 10^5, 127 \times 10^5]$. 
+In this case, we could quantize a given value $v$ as an $8$-bit representation of the integer $j$, where $j \times 10^5$ is the closest value to $v$
+among all integral multiples $\{j \times 10^5: -128 \leq j \leq 127 \}$.
 
-1 bit is the classic RaBitQ setting. You can set it to 2, 4, or 8 bits to improve fidelity for better precision or recall — the main trade-off is additional storage for the extra bits per dimension, with only a modest increase in query-time compute.
-It's also possible to tune the number of IVF partitions in `IVF_RQ`, similar to how you would do in `IVF_PQ`.
+## Quantization API Reference
 
-<Warning title="Reading multi-bit indexes across versions">
-Indexes built with `num_bits >= 2` use an updated on-disk layout. Older LanceDB versions cannot read them and will fail with a clear missing-column error rather than returning incorrect results. Existing indexes keep working and upgrade automatically when they are rewritten (for example, during compaction, optimize, or remap). `num_bits=1` indexes are unaffected in both directions.
-</Warning>
 
-## API Reference
-
-The full list of parameters to the algorithm are listed below.
-
-- `distance_type`: Literal["l2", "cosine", "dot"], defaults to "l2"  
-  The distance metric to use for similarity comparison. Choose "l2" for Euclidean, "cosine" for cosine similarity, or "dot" for dot product.
-- `num_partitions`: Optional[int], defaults to None  
-  Number of IVF partitions (affects index build time and query accuracy). More partitions can improve recall but may increase build time. When unset, LanceDB chooses roughly the square root of the row count.
-- `num_bits`: int, defaults to 1  
-  Bits per dimension for quantization (1 is standard RaBitQ). Higher values improve fidelity, mainly at the cost of additional storage.
-- `max_iterations`: int, defaults to 50  
-  Maximum number of iterations for training the quantizer. Increase for larger datasets or to improve quantization quality.
-- `sample_rate`: int, defaults to 256  
-  Number of samples per partition during training. Higher values may improve accuracy but increase training time.
-- `target_partition_size`: Optional[int], defaults to None  
-  Target number of vectors per partition. Adjust to control partition granularity and memory usage. If `num_partitions` is also set, `num_partitions` takes precedence.
+| Parameter | Description |
+| :--- | :--- |
+| `num_bits` | Bits per dimension for quantization. Only applies to `IVF_PQ`/`IVF_HNSW_PQ` (default `8`) and `IVF_RQ` (default `1`, RaBitQ) — not used by `IVF_FLAT`/`IVF_SQ`/`IVF_HNSW_FLAT`/`IVF_HNSW_SQ`. Higher values improve accuracy at the cost of additional storage. |
+
+`max_iterations` and `sample_rate` also affect quantizer training, but since they apply to every IVF/HNSW index type (not just quantized ones), they're documented as general [Build-time Parameters](/indexing/vector-index#build-time-parameters) instead. All three are passed as keyword arguments to `create_index`, alongside `index_type`:
+
+<CodeGroup>
+    <CodeBlock filename="Python" language="Python" icon="python">
+    {QuantizationCustomParams}
+    </CodeBlock>
+</CodeGroup>

docs/indexing/scalar-index.mdx

@@ -27,6 +27,15 @@ LanceDB supports four types of scalar indexes:
 - `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries.
 - `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
 
+<Tip title="Working with scalar indices">
+Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with:
+
+- Vector search (prefilter results using metadata)
+- Full-text search (combining keyword matching with structured filters) 
+- SQL scans (optimizing WHERE clauses on scalar columns)
+- Key-value lookups (enabling rapid primary key-based retrievals)
+</Tip>
+
 ## Choosing the Right Index Type
 
 | Data Type                                                       | Filter                                    | Index Type   |