index-modules.asciidoc 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383
  1. [[index-modules]]
  2. = Index modules
  3. Index Modules are modules created per index and control all aspects related to
  4. an index.
  5. [discrete]
  6. [[index-modules-settings]]
  7. == Index Settings
  8. [[index-modules-settings-description]]
  9. // tag::index-modules-settings-description-tag[]
  10. Index level settings can be set per-index. Settings may be:
  11. _static_::
  12. They can only be set at index creation time or on a
  13. <<indices-open-close,closed index>>.
  14. _dynamic_::
  15. They can be changed on a live index using the
  16. <<indices-update-settings,update-index-settings>> API.
  17. // end::index-modules-settings-description-tag[]
  18. WARNING: Changing static or dynamic index settings on a closed index could
  19. result in incorrect settings that are impossible to rectify without deleting
  20. and recreating the index.
  21. [discrete]
  22. === Static index settings
  23. Below is a list of all _static_ index settings that are not associated with any
  24. specific index module:
  25. [[index-number-of-shards]]
  26. // tag::index-number-of-shards-tag[]
  27. `index.number_of_shards` {ess-icon}::
  28. The number of primary shards that an index should have. Defaults to `1`. This setting can only be set at index creation time. It cannot be changed on a closed index.
  29. +
  30. NOTE: The number of shards are limited to `1024` per index. This limitation is a safety limit to prevent accidental creation of indices that can destabilize a cluster due to resource allocation. The limit can be modified by specifying `export ES_JAVA_OPTS="-Des.index.max_number_of_shards=128"` system property on every node that is part of the cluster.
  31. // end::index-number-of-shards-tag[]
  32. [[index-number-of-routing-shards]]
  33. `index.number_of_routing_shards`::
  34. +
  35. ====
  36. Number of routing shards used to <<indices-split-index,split>> an index.
  37. For example, a 5 shard index with `number_of_routing_shards` set to `30` (`5 x
  38. 2 x 3`) could be split by a factor of `2` or `3`. In other words, it could be
  39. split as follows:
  40. * `5` -> `10` -> `30` (split by 2, then by 3)
  41. * `5` -> `15` -> `30` (split by 3, then by 2)
  42. * `5` -> `30` (split by 6)
  43. This setting's default value depends on the number of primary shards in the
  44. index. The default is designed to allow you to split by factors of 2 up
  45. to a maximum of 1024 shards.
  46. NOTE: In {es} 7.0.0 and later versions, this setting affects how documents are
  47. distributed across shards. When reindexing an older index with custom routing,
  48. you must explicitly set `index.number_of_routing_shards` to maintain the same
  49. document distribution. See the
  50. {ref-70}/breaking-changes-7.0.html#_document_distribution_changes[related
  51. breaking change].
  52. ====
  53. `index.shard.check_on_startup`::
  54. Whether or not shards should be checked for corruption before opening. When
  55. corruption is detected, it will prevent the shard from being opened.
  56. Accepts:
  57. `false`::: (default) Don't check for corruption when opening a shard.
  58. `checksum`::: Check for physical corruption.
  59. `true`::: Check for both physical and logical corruption. This is much more
  60. expensive in terms of CPU and memory usage.
  61. +
  62. WARNING: Expert only. Checking shards may take a lot of time on large
  63. indices.
  64. [[index-codec]] `index.codec`::
  65. The +default+ value compresses stored data with LZ4
  66. compression, but this can be set to +best_compression+
  67. which uses {wikipedia}/DEFLATE[DEFLATE] for a higher
  68. compression ratio, at the expense of slower stored fields performance.
  69. If you are updating the compression type, the new one will be applied
  70. after segments are merged. Segment merging can be forced using
  71. <<indices-forcemerge,force merge>>.
  72. [[routing-partition-size]] `index.routing_partition_size`::
  73. The number of shards a custom <<mapping-routing-field,routing>> value can go to.
  74. Defaults to 1 and can only be set at index creation time. This value must be less
  75. than the `index.number_of_shards` unless the `index.number_of_shards` value is also 1.
  76. See <<routing-index-partition>> for more details about how this setting is used.
  77. [[ccr-index-soft-deletes]]
  78. // tag::ccr-index-soft-deletes-tag[]
  79. `index.soft_deletes.enabled`::
  80. deprecated:[7.6.0, Creating indices with soft-deletes disabled is deprecated and will be removed in future Elasticsearch versions.]
  81. Indicates whether soft deletes are enabled on the index. Soft deletes can only
  82. be configured at index creation and only on indices created on or after
  83. {es} 6.5.0. Defaults to `true`.
  84. // end::ccr-index-soft-deletes-tag[]
  85. [[ccr-index-soft-deletes-retention-period]]
  86. //tag::ccr-index-soft-deletes-retention-tag[]
  87. `index.soft_deletes.retention_lease.period`::
  88. The maximum period to retain a shard history retention lease before it is
  89. considered expired. Shard history retention leases ensure that soft deletes are
  90. retained during merges on the Lucene index. If a soft delete is merged away
  91. before it can be replicated to a follower the following process will fail due
  92. to incomplete history on the leader. Defaults to `12h`.
  93. //end::ccr-index-soft-deletes-retention-tag[]
  94. [[load-fixed-bitset-filters-eagerly]] `index.load_fixed_bitset_filters_eagerly`::
  95. Indicates whether <<query-filter-context, cached filters>> are pre-loaded for
  96. nested queries. Possible values are `true` (default) and `false`.
  97. [[index-hidden]] `index.hidden`::
  98. Indicates whether the index should be hidden by default. Hidden indices are not
  99. returned by default when using a wildcard expression. This behavior is controlled
  100. per request through the use of the `expand_wildcards` parameter. Possible values are
  101. `true` and `false` (default).
  102. [discrete]
  103. [[dynamic-index-settings]]
  104. === Dynamic index settings
  105. Below is a list of all _dynamic_ index settings that are not associated with any
  106. specific index module:
  107. [[dynamic-index-number-of-replicas]]
  108. `index.number_of_replicas`::
  109. The number of replicas each primary shard has. Defaults to 1.
  110. `index.auto_expand_replicas`::
  111. Auto-expand the number of replicas based on the number of data nodes in the
  112. cluster. Set to a dash delimited lower and upper bound (e.g. `0-5`) or use `all`
  113. for the upper bound (e.g. `0-all`). Defaults to `false` (i.e. disabled). Note
  114. that the auto-expanded number of replicas only takes
  115. <<shard-allocation-filtering,allocation filtering>> rules into account, but
  116. ignores other allocation rules such as <<allocation-total-shards,total shards
  117. per node>>, and this can lead to the cluster health becoming `YELLOW` if the
  118. applicable rules prevent all the replicas from being allocated.
  119. +
  120. If the upper bound is `all` then <<shard-allocation-awareness,shard allocation
  121. awareness>> and
  122. <<cluster-routing-allocation-same-shard-host,`cluster.routing.allocation.same_shard.host`>>
  123. are ignored for this index.
  124. `index.search.idle.after`::
  125. How long a shard can not receive a search or get request until it's considered
  126. search idle. (default is `30s`)
  127. [[index-refresh-interval-setting]]
  128. `index.refresh_interval`::
  129. How often to perform a refresh operation, which makes recent changes to the
  130. index visible to search. Defaults to `1s`. Can be set to `-1` to disable
  131. refresh. If this setting is not explicitly set, shards that haven't seen
  132. search traffic for at least `index.search.idle.after` seconds will not receive
  133. background refreshes until they receive a search request. Searches that hit an
  134. idle shard where a refresh is pending will wait for the next background
  135. refresh (within `1s`). This behavior aims to automatically optimize bulk
  136. indexing in the default case when no searches are performed. In order to opt
  137. out of this behavior an explicit value of `1s` should set as the refresh
  138. interval.
  139. [[index-max-result-window]]
  140. `index.max_result_window`::
  141. The maximum value of `from + size` for searches to this index. Defaults to
  142. `10000`. Search requests take heap memory and time proportional to
  143. `from + size` and this limits that memory. See
  144. <<scroll-search-results,Scroll>> or <<search-after,Search After>> for a more efficient alternative
  145. to raising this.
  146. `index.max_inner_result_window`::
  147. The maximum value of `from + size` for inner hits definition and top hits aggregations to this index. Defaults to
  148. `100`. Inner hits and top hits aggregation take heap memory and time proportional to `from + size` and this limits that memory.
  149. `index.max_rescore_window`::
  150. The maximum value of `window_size` for `rescore` requests in searches of this index.
  151. Defaults to `index.max_result_window` which defaults to `10000`. Search
  152. requests take heap memory and time proportional to
  153. `max(window_size, from + size)` and this limits that memory.
  154. `index.max_docvalue_fields_search`::
  155. The maximum number of `docvalue_fields` that are allowed in a query.
  156. Defaults to `100`. Doc-value fields are costly since they might incur
  157. a per-field per-document seek.
  158. `index.max_script_fields`::
  159. The maximum number of `script_fields` that are allowed in a query.
  160. Defaults to `32`.
  161. [[index-max-ngram-diff]]
  162. `index.max_ngram_diff`::
  163. The maximum allowed difference between min_gram and max_gram for NGramTokenizer and NGramTokenFilter.
  164. Defaults to `1`.
  165. [[index-max-shingle-diff]]
  166. `index.max_shingle_diff`::
  167. The maximum allowed difference between max_shingle_size and min_shingle_size
  168. for the <<analysis-shingle-tokenfilter,`shingle` token filter>>. Defaults to
  169. `3`.
  170. `index.max_refresh_listeners`::
  171. Maximum number of refresh listeners available on each shard of the index.
  172. These listeners are used to implement <<docs-refresh,`refresh=wait_for`>>.
  173. `index.analyze.max_token_count`::
  174. The maximum number of tokens that can be produced using _analyze API.
  175. Defaults to `10000`.
  176. [[index-max-analyzed-offset]]
  177. `index.highlight.max_analyzed_offset`::
  178. The maximum number of characters that will be analyzed for a highlight request.
  179. This setting is only applicable when highlighting is requested on a text that was indexed without offsets or term vectors.
  180. Defaults to `1000000`.
  181. [[index-max-terms-count]]
  182. `index.max_terms_count`::
  183. The maximum number of terms that can be used in Terms Query.
  184. Defaults to `65536`.
  185. [[index-max-regex-length]]
  186. `index.max_regex_length`::
  187. The maximum length of regex that can be used in Regexp Query.
  188. Defaults to `1000`.
  189. `index.query.default_field`::
  190. +
  191. --
  192. (string or array of strings)
  193. Wildcard (`*`) patterns matching one or more fields. The following query types
  194. search these matching fields by default:
  195. * <<query-dsl-mlt-query>>
  196. * <<query-dsl-multi-match-query>>
  197. * <<query-dsl-query-string-query>>
  198. * <<query-dsl-simple-query-string-query>>
  199. Defaults to `*`, which matches all fields eligible for
  200. <<term-level-queries,term-level queries>>, excluding metadata fields.
  201. --
  202. `index.routing.allocation.enable`::
  203. Controls shard allocation for this index. It can be set to:
  204. * `all` (default) - Allows shard allocation for all shards.
  205. * `primaries` - Allows shard allocation only for primary shards.
  206. * `new_primaries` - Allows shard allocation only for newly-created primary shards.
  207. * `none` - No shard allocation is allowed.
  208. `index.routing.rebalance.enable`::
  209. Enables shard rebalancing for this index. It can be set to:
  210. * `all` (default) - Allows shard rebalancing for all shards.
  211. * `primaries` - Allows shard rebalancing only for primary shards.
  212. * `replicas` - Allows shard rebalancing only for replica shards.
  213. * `none` - No shard rebalancing is allowed.
  214. `index.gc_deletes`::
  215. The length of time that a <<delete-versioning,deleted document's version number>> remains available for <<index-versioning,further versioned operations>>.
  216. Defaults to `60s`.
  217. [[index-default-pipeline]]
  218. `index.default_pipeline`::
  219. The default <<ingest,ingest node>> pipeline for this index. Index requests will fail
  220. if the default pipeline is set and the pipeline does not exist. The default may be
  221. overridden using the `pipeline` parameter. The special pipeline name `_none` indicates
  222. no ingest pipeline should be run.
  223. [[index-final-pipeline]]
  224. `index.final_pipeline`::
  225. The final <<ingest,ingest node>> pipeline for this index. Index requests
  226. will fail if the final pipeline is set and the pipeline does not exist.
  227. The final pipeline always runs after the request pipeline (if specified) and
  228. the default pipeline (if it exists). The special pipeline name `_none`
  229. indicates no ingest pipeline will run.
  230. [discrete]
  231. === Settings in other index modules
  232. Other index settings are available in index modules:
  233. <<analysis,Analysis>>::
  234. Settings to define analyzers, tokenizers, token filters and character
  235. filters.
  236. <<index-modules-allocation,Index shard allocation>>::
  237. Control over where, when, and how shards are allocated to nodes.
  238. <<index-modules-mapper,Mapping>>::
  239. Enable or disable dynamic mapping for an index.
  240. <<index-modules-merge,Merging>>::
  241. Control over how shards are merged by the background merge process.
  242. <<index-modules-similarity,Similarities>>::
  243. Configure custom similarity settings to customize how search results are
  244. scored.
  245. <<index-modules-slowlog,Slowlog>>::
  246. Control over how slow queries and fetch requests are logged.
  247. <<index-modules-store,Store>>::
  248. Configure the type of filesystem used to access shard data.
  249. <<index-modules-translog,Translog>>::
  250. Control over the transaction log and background flush operations.
  251. <<index-modules-history-retention,History retention>>::
  252. Control over the retention of a history of operations in the index.
  253. <<index-modules-indexing-pressure,Indexing pressure>>::
  254. Configure indexing back pressure limits.
  255. [discrete]
  256. [[x-pack-index-settings]]
  257. === [xpack]#{xpack} index settings#
  258. <<ilm-settings,{ilm-cap}>>::
  259. Specify the lifecycle policy and rollover alias for an index.
  260. include::index-modules/analysis.asciidoc[]
  261. include::index-modules/allocation.asciidoc[]
  262. include::index-modules/blocks.asciidoc[]
  263. include::index-modules/mapper.asciidoc[]
  264. include::index-modules/merge.asciidoc[]
  265. include::index-modules/similarity.asciidoc[]
  266. include::index-modules/slowlog.asciidoc[]
  267. include::index-modules/store.asciidoc[]
  268. include::index-modules/translog.asciidoc[]
  269. include::index-modules/history-retention.asciidoc[]
  270. include::index-modules/index-sorting.asciidoc[]
  271. include::index-modules/indexing-pressure.asciidoc[]