Make int8_hnsw our default index for new dense-vector fields (#106836)

For float32, there is no compelling reason to use all the memory
required by default for HNSW. Using `int8_hnsw` provides a much saner
default when it comes to cost vs relevancy. 

So, on all new indices that use `dense_vector` and want to index them
for fast search, we will default to `int8_hnsw`. 

Users can still customize their parameters, or prefer `hnsw` over
float32 if they so desire.

docs/changelog/106836.yaml

@@ -0,0 +1,5 @@
+pr: 106836
+summary: Make int8_hnsw our default index for new dense-vector fields
+area: Mapping
+type: enhancement
+issues: []

docs/reference/mapping/types/dense-vector.asciidoc

@@ -65,7 +65,7 @@ data structure to support fast kNN retrieval through the <<search-api-knn, `knn`
 Unmapped array fields of float elements with size between 128 and 4096 are dynamically mapped as `dense_vector` with a default similariy of `cosine`.
 You can override the default similarity by explicitly mapping the field as `dense_vector` with the desired similarity.
 
-Indexing is enabled by default for dense vector fields.
+Indexing is enabled by default for dense vector fields and indexed as `int8_hnsw`.
 When indexing is enabled, you can define the vector similarity to use in kNN search:
 
 [source,console]
@@ -116,7 +116,8 @@ that sacrifices result accuracy for improved speed.
 
 The `dense_vector` type supports quantization to reduce the memory footprint required when <<approximate-knn, searching>> `float` vectors.
 Currently the only quantization method supported is `int8` and provided vectors `element_type` must be `float`. To use
-a quantized index, you can set your index type to `int8_hnsw`.
+a quantized index, you can set your index type to `int8_hnsw`. When indexing `float` vectors, the current default
+index type is `int8_hnsw`.
 
 When using the `int8_hnsw` index, each of the `float` vectors' dimensions are quantized to 1-byte integers. This can
 reduce the memory footprint by as much as 75% at the cost of some accuracy. However, the disk usage can increase by
@@ -240,9 +241,10 @@ expense of slower indexing speed.
 The type of kNN algorithm to use. Can be either any of:
 +
 --
-* `hnsw` - The default storage type. This utilizes the https://arxiv.org/abs/1603.09320[HNSW algorithm] for scalable
+* `hnsw` - This utilizes the https://arxiv.org/abs/1603.09320[HNSW algorithm] for scalable
     approximate kNN search. This supports all `element_type` values.
-* `int8_hnsw` - This utilizes the https://arxiv.org/abs/1603.09320[HNSW algorithm] in addition to automatically scalar
+* `int8_hnsw` - The default index type for float vectors.
+This utilizes the https://arxiv.org/abs/1603.09320[HNSW algorithm] in addition to automatically scalar
 quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint
 by 4x at the cost of some accuracy. See <<dense-vector-quantization, Automatically quantize vectors for kNN search>>.
 * `flat` - This utilizes a brute-force search algorithm for exact kNN search. This supports all `element_type` values.

docs/reference/search/search-your-data/knn-search.asciidoc

@@ -272,6 +272,8 @@ If you want to provide `float` vectors, but want the memory savings of `byte` ve
 internally they are indexed as `byte` vectors. Additionally, the original `float` vectors are still retained
 in the index.
 
+NOTE: The default index type for `dense_vector` is `int8_hnsw`.
+
 To use quantization, you can use the index type `int8_hnsw` object in the `dense_vector` mapping.
 
 [source,console]
@@ -652,7 +654,10 @@ PUT passage_vectors
                 "properties": {
                     "vector": {
                         "type": "dense_vector",
-                        "dims": 2
+                        "dims": 2,
+                        "index_options": {
+                            "type": "hnsw"
+                        }
                     },
                     "text": {
                         "type": "text",

qa/rolling-upgrade-legacy/src/test/resources/rest-api-spec/test/old_cluster/30_vector_search.yml

@@ -18,6 +18,10 @@
                 dims: 3
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 100
   - do:
       bulk:
         index: test-float-index

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/100_knn_nested_search.yml

@@ -23,6 +23,10 @@ setup:
                     dims: 5
                     index: true
                     similarity: l2_norm
+                    index_options:
+                      type: hnsw
+                      m: 16
+                      ef_construction: 200
 
   - do:
       index:

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/120_knn_query_multiple_shards.yml

@@ -19,6 +19,10 @@ setup:
                 dims: 4
                 index : true
                 similarity : l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               my_name:
                 type: keyword
                 store: true

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/140_knn_query_with_other_queries.yml

@@ -19,6 +19,10 @@ setup:
                 dims: 4
                 index : true
                 similarity : l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               my_name:
                 type: keyword
                 store: true

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/160_knn_query_missing_params.yml

@@ -15,6 +15,10 @@ setup:
                     dims: 3
                     index: true
                     similarity: l2_norm
+                    index_options:
+                      type: hnsw
+                      ef_construction: 100
+                      m: 16
                   category:
                     type: keyword
                   nested:
@@ -27,6 +31,10 @@ setup:
                         dims: 5
                         index: true
                         similarity: l2_norm
+                        index_options:
+                          type: hnsw
+                          ef_construction: 100
+                          m: 16
 
   - do:
       index:

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/40_knn_search.yml

@@ -15,11 +15,19 @@ setup:
                 dims: 5
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               another_vector:
                 type: dense_vector
                 dims: 5
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
 
   - do:
       index:
@@ -371,6 +379,10 @@ setup:
                 dims: 5
                 index: true
                 similarity: max_inner_product
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
 
   - do:
       index:

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/40_knn_search_cosine.yml

@@ -13,26 +13,46 @@ setup:
                 dims: 5
                 index: true
                 similarity: cosine
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               normalized_vector:
                 type: dense_vector
                 dims: 5
                 index: true
                 similarity: cosine
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               end_normalized:
                 type: dense_vector
                 dims: 5
                 index: true
                 similarity: cosine
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               first_normalized:
                 type: dense_vector
                 dims: 5
                 index: true
                 similarity: cosine
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               middle_normalized:
                 type: dense_vector
                 dims: 5
                 index: true
                 similarity: cosine
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
 
 
   - do:

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/60_knn_search_filter_alias.yml

@@ -17,6 +17,10 @@ setup:
                 dims: 4
                 index : true
                 similarity : l2_norm
+                index_options:
+                  type: hnsw
+                  m: 16
+                  ef_construction: 200
               name:
                 type: keyword
                 store: true

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/80_dense_vector_indexed_by_default.yml

@@ -21,16 +21,10 @@ setup:
       indices.get_mapping:
         index: test
 
-  - match:
-      test:
-        mappings:
-          properties:
-            vector:
-              type: dense_vector
-              dims: 5
-              index: true
-              similarity: cosine
-
+  - match: { test.mappings.properties.vector.type: dense_vector }
+  - match: { test.mappings.properties.vector.dims: 5 }
+  - match: { test.mappings.properties.vector.index: true }
+  - match: { test.mappings.properties.vector.similarity: cosine }
 ---
 "Indexed by default with specified similarity and index options":
   - do:
@@ -127,7 +121,29 @@ setup:
                   type: hnsw
                   m: 32
                   ef_construction: 200
+---
+"Default index options for dense_vector":
+  - skip:
+      version: ' - 8.13.99'
+      reason: 'dense_vector indexed as int8_hnsw by default was added in 8.14'
+  - do:
+      indices.create:
+        index: test_default_index_options
+        body:
+          mappings:
+            properties:
+              vector:
+                type: dense_vector
+                dims: 5
 
+  - match: { acknowledged: true }
 
+  - do:
+      indices.get_mapping:
+        index: test_default_index_options
 
-
+  - match: { test_default_index_options.mappings.properties.vector.type: dense_vector }
+  - match: { test_default_index_options.mappings.properties.vector.dims: 5 }
+  - match: { test_default_index_options.mappings.properties.vector.index: true }
+  - match: { test_default_index_options.mappings.properties.vector.similarity: cosine }
+  - match: { test_default_index_options.mappings.properties.vector.index_options.type: int8_hnsw }

server/src/main/java/org/elasticsearch/index/IndexVersions.java

@@ -103,6 +103,7 @@ public class IndexVersions {
     public static final IndexVersion TIME_SERIES_ID_HASHING = def(8_502_00_1, Version.LUCENE_9_9_2);
     public static final IndexVersion UPGRADE_TO_LUCENE_9_10 = def(8_503_00_0, Version.LUCENE_9_10_0);
     public static final IndexVersion TIME_SERIES_ROUTING_HASH_IN_ID = def(8_504_00_0, Version.LUCENE_9_10_0);
+    public static final IndexVersion DEFAULT_DENSE_VECTOR_TO_INT8_HNSW = def(8_505_00_0, Version.LUCENE_9_10_0);
 
     /*
      * STOP! READ THIS FIRST! No, really,

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

@@ -93,6 +93,7 @@ import java.util.stream.Stream;
 
 import static org.elasticsearch.common.Strings.format;
 import static org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken;
+import static org.elasticsearch.index.IndexVersions.DEFAULT_DENSE_VECTOR_TO_INT8_HNSW;
 
 /**
  * A {@link FieldMapper} for indexing a dense vector of floats.
@@ -108,6 +109,7 @@ public class DenseVectorFieldMapper extends FieldMapper {
     public static final IndexVersion MAGNITUDE_STORED_INDEX_VERSION = IndexVersions.V_7_5_0;
     public static final IndexVersion INDEXED_BY_DEFAULT_INDEX_VERSION = IndexVersions.FIRST_DETACHED_INDEX_VERSION;
     public static final IndexVersion NORMALIZE_COSINE = IndexVersions.NORMALIZED_VECTOR_COSINE;
+    public static final IndexVersion DEFAULT_TO_INT8 = DEFAULT_DENSE_VECTOR_TO_INT8_HNSW;
     public static final IndexVersion LITTLE_ENDIAN_FLOAT_STORED_INDEX_VERSION = IndexVersions.V_8_9_0;
 
     public static final String CONTENT_TYPE = "dense_vector";
@@ -152,15 +154,7 @@ public class DenseVectorFieldMapper extends FieldMapper {
         }, m -> toType(m).fieldType().dims, XContentBuilder::field, Object::toString).setSerializerCheck((id, ic, v) -> v != null)
             .setMergeValidator((previous, current, c) -> previous == null || Objects.equals(previous, current));
         private final Parameter<VectorSimilarity> similarity;
-        private final Parameter<IndexOptions> indexOptions = new Parameter<>(
-            "index_options",
-            false,
-            () -> null,
-            (n, c, o) -> o == null ? null : parseIndexOptions(n, o),
-            m -> toType(m).indexOptions,
-            XContentBuilder::field,
-            Objects::toString
-        ).setSerializerCheck((id, ic, v) -> v != null);
+        private final Parameter<IndexOptions> indexOptions;
         private final Parameter<Boolean> indexed;
         private final Parameter<Map<String, String>> meta = Parameter.metaParam();
 
@@ -170,6 +164,7 @@ public class DenseVectorFieldMapper extends FieldMapper {
             super(name);
             this.indexVersionCreated = indexVersionCreated;
             final boolean indexedByDefault = indexVersionCreated.onOrAfter(INDEXED_BY_DEFAULT_INDEX_VERSION);
+            final boolean defaultInt8Hnsw = indexVersionCreated.onOrAfter(DEFAULT_DENSE_VECTOR_TO_INT8_HNSW);
             this.indexed = Parameter.indexParam(m -> toType(m).fieldType().indexed, indexedByDefault);
             if (indexedByDefault) {
                 // Only serialize on newer index versions to prevent breaking existing indices when upgrading
@@ -182,6 +177,34 @@ public class DenseVectorFieldMapper extends FieldMapper {
                 (Supplier<VectorSimilarity>) () -> indexedByDefault && indexed.getValue() ? VectorSimilarity.COSINE : null,
                 VectorSimilarity.class
             ).acceptsNull().setSerializerCheck((id, ic, v) -> v != null);
+            this.indexOptions = new Parameter<>(
+                "index_options",
+                false,
+                () -> defaultInt8Hnsw && elementType.getValue() != ElementType.BYTE && this.indexed.getValue()
+                    ? new Int8HnswIndexOptions(
+                        Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN,
+                        Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH,
+                        null
+                    )
+                    : null,
+                (n, c, o) -> o == null ? null : parseIndexOptions(n, o),
+                m -> toType(m).indexOptions,
+                (b, n, v) -> {
+                    if (v != null) {
+                        b.field(n, v);
+                    }
+                },
+                Objects::toString
+            ).setSerializerCheck((id, ic, v) -> v != null).addValidator(v -> {
+                if (v != null && v.supportsElementType(elementType.getValue()) == false) {
+                    throw new IllegalArgumentException(
+                        "[element_type] cannot be [" + elementType.getValue().toString() + "] when using index type [" + v.type + "]"
+                    );
+                }
+            }).acceptsNull();
+            if (defaultInt8Hnsw) {
+                this.indexOptions.alwaysSerialize();
+            }
             this.indexed.addValidator(v -> {
                 if (v) {
                     if (similarity.getValue() == null) {
@@ -200,13 +223,6 @@ public class DenseVectorFieldMapper extends FieldMapper {
                     }
                 }
             });
-            this.indexOptions.addValidator(v -> {
-                if (v != null && v.supportsElementType(elementType.getValue()) == false) {
-                    throw new IllegalArgumentException(
-                        "[element_type] cannot be [" + elementType.getValue().toString() + "] when using index type [" + v.type + "]"
-                    );
-                }
-            });
         }
 
         @Override

server/src/test/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapperTests.java

@@ -247,7 +247,15 @@ public class DenseVectorFieldMapperTests extends MapperTestCase {
 
         mapping = mapping(b -> {
             b.startObject("field");
-            b.field("type", "dense_vector").field("dims", 4).field("similarity", "cosine").field("index", true);
+            b.field("type", "dense_vector")
+                .field("dims", 4)
+                .field("similarity", "cosine")
+                .field("index", true)
+                .startObject("index_options")
+                .field("type", "int8_hnsw")
+                .field("m", 16)
+                .field("ef_construction", 100)
+                .endObject();
             b.endObject();
         });
         merge(mapperService, mapping);

x-pack/plugin/rank-rrf/src/internalClusterTest/java/org/elasticsearch/xpack/rank/rrf/RRFRankSingleShardIT.java

@@ -53,6 +53,9 @@ public class RRFRankSingleShardIT extends ESSingleNodeTestCase {
             .field("dims", 1)
             .field("index", true)
             .field("similarity", "l2_norm")
+            .startObject("index_options")
+            .field("type", "hnsw")
+            .endObject()
             .endObject()
             .startObject("text")
             .field("type", "text")
@@ -80,12 +83,18 @@ public class RRFRankSingleShardIT extends ESSingleNodeTestCase {
             .field("dims", 1)
             .field("index", true)
             .field("similarity", "l2_norm")
+            .startObject("index_options")
+            .field("type", "hnsw")
+            .endObject()
             .endObject()
             .startObject("vector_desc")
             .field("type", "dense_vector")
             .field("dims", 1)
             .field("index", true)
             .field("similarity", "l2_norm")
+            .startObject("index_options")
+            .field("type", "hnsw")
+            .endObject()
             .endObject()
             .startObject("int")
             .field("type", "integer")

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/100_rank_rrf.yml

@@ -21,6 +21,10 @@ setup:
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
 
   - do:
       index:

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/200_rank_rrf_script.yml

@@ -18,11 +18,19 @@ setup:
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
               vector_desc:
                 type: dense_vector
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
               int:
                 type: integer
               text:

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/300_rrf_retriever.yml

@@ -21,6 +21,10 @@ setup:
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
 
   - do:
       index:

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/400_rrf_retriever_script.yml

@@ -20,11 +20,19 @@ setup:
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
               vector_desc:
                 type: dense_vector
                 dims: 1
                 index: true
                 similarity: l2_norm
+                index_options:
+                  type: hnsw
+                  ef_construction: 100
+                  m: 16
               int:
                 type: integer
               text: