Browse Source

Clarify the vector files utilized for preloading (#116488) (#116622)

Adds clarification for vector preloading, what extension is to what
storage kind, and that quantized vectors are stored in separate files
allowing for individual preload. 

closes: https://github.com/elastic/elasticsearch/issues/116273
Benjamin Trent 11 months ago
parent
commit
f9077a09ef

+ 10 - 3
docs/reference/how-to/knn-search.asciidoc

@@ -95,13 +95,20 @@ and https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector[here] some
 of datasets and configurations that we use for our nightly benchmarks.
 
 [discrete]
+[[dense-vector-preloading]]
 include::search-speed.asciidoc[tag=warm-fs-cache]
 
 The following file extensions are used for the approximate kNN search:
+Each extension is broken down by the quantization types.
 
-* `vec` and `veq` for vector values
-* `vex` for HNSW graph
-* `vem`, `vemf`, and `vemq` for metadata
+* `vex` for the HNSW graph
+* `vec` for all non-quantized vector values. This includes all element types: `float`, `byte`, and `bit`.
+* `veq` for quantized vectors indexed with <<dense-vector-quantization,`quantization`>>: `int4` or `int8`
+* `veb` for binary vectors indexed with <<dense-vector-quantization,`quantization`>>: `bbq`
+* `vem`, `vemf`, `vemq`, and `vemb` for metadata, usually small and not a concern for preloading
+
+Generally, if you are using a quantized index, you should only preload the relevant quantized values and the HNSW graph.
+Preloading the raw vectors is not necessary and might be counterproductive.
 
 [discrete]
 === Reduce the number of index segments

+ 2 - 2
docs/reference/index-modules/store.asciidoc

@@ -143,8 +143,8 @@ terms dictionaries, postings lists and points, which are the most important
 parts of the index for search and aggregations.
 
 For vector search, you use <<approximate-knn, approximate k-nearest neighbor search>>,
-you might want to set the setting to vector search files: `["vec", "vex", "vem"]`
-("vec" is used for vector values, "vex" – for HNSW graph, "vem" – for metadata).
+you might want to set the setting to vector search files. See <<dense-vector-preloading, vector preloading>> for a detailed
+list of the files.
 
 Note that this setting can be dangerous on indices that are larger than the size
 of the main memory of the host, as it would cause the filesystem cache to be