store.asciidoc 4.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118
  1. [[index-modules-store]]
  2. == Store
  3. The store module allows you to control how index data is stored.
  4. The index can either be stored in-memory (no persistence) or on-disk
  5. (the default). In-memory indices provide better performance at the cost
  6. of limiting the index size to the amount of available physical memory.
  7. When using a local gateway (the default), file system storage with *no*
  8. in memory storage is required to maintain index consistency. This is
  9. required since the local gateway constructs its state from the local
  10. index state of each node.
  11. Another important aspect of memory based storage is the fact that
  12. ElasticSearch supports storing the index in memory *outside of the JVM
  13. heap space* using the "Memory" (see below) storage type. It translates
  14. to the fact that there is no need for extra large JVM heaps (with their
  15. own consequences) for storing the index in memory.
  16. [float]
  17. === Store Level Throttling
  18. The way Lucene, the IR library elasticsearch uses under the covers,
  19. works is by creating immutable segments (up to deletes) and constantly
  20. merging them (the merge policy settings allow to control how those
  21. merges happen). The merge process happens in an asynchronous manner
  22. without affecting the indexing / search speed. The problem though,
  23. especially on systems with low IO, is that the merge process can be
  24. expensive and affect search / index operation simply by the fact that
  25. the box is now taxed with more IO happening.
  26. The store module allows to have throttling configured for merges (or
  27. all) either on the node level, or on the index level. The node level
  28. throttling will make sure that out of all the shards allocated on that
  29. node, the merge process won't pass the specific setting bytes per
  30. second. It can be set by setting `indices.store.throttle.type` to
  31. `merge`, and setting `indices.store.throttle.max_bytes_per_sec` to
  32. something like `5mb`. The node level settings can be changed dynamically
  33. using the cluster update settings API. The default is set
  34. to `20mb` with type `merge`.
  35. If specific index level configuration is needed, regardless of the node
  36. level settings, it can be set as well using the
  37. `index.store.throttle.type`, and
  38. `index.store.throttle.max_bytes_per_sec`. The default value for the type
  39. is `node`, meaning it will throttle based on the node level settings and
  40. participate in the global throttling happening. Both settings can be set
  41. using the index update settings API dynamically.
  42. The following sections lists all the different storage types supported.
  43. [float]
  44. === File System
  45. File system based storage is the default storage used. There are
  46. different implementations or storage types. The best one for the
  47. operating environment will be automatically chosen: `mmapfs` on
  48. Solaris/Windows 64bit, `simplefs` on Windows 32bit, and `niofs` for the
  49. rest.
  50. The following are the different file system based storage types:
  51. [float]
  52. ==== Simple FS
  53. The `simplefs` type is a straightforward implementation of file system
  54. storage (maps to Lucene `SimpleFsDirectory`) using a random access file.
  55. This implementation has poor concurrent performance (multiple threads
  56. will bottleneck). It is usually better to use the `niofs` when you need
  57. index persistence.
  58. [float]
  59. ==== NIO FS
  60. The `niofs` type stores the shard index on the file system (maps to
  61. Lucene `NIOFSDirectory`) using NIO. It allows multiple threads to read
  62. from the same file concurrently. It is not recommended on Windows
  63. because of a bug in the SUN Java implementation.
  64. [float]
  65. ==== MMap FS
  66. The `mmapfs` type stores the shard index on the file system (maps to
  67. Lucene `MMapDirectory`) by mapping a file into memory (mmap). Memory
  68. mapping uses up a portion of the virtual memory address space in your
  69. process equal to the size of the file being mapped. Before using this
  70. class, be sure your have plenty of virtual address space.
  71. [float]
  72. === Memory
  73. The `memory` type stores the index in main memory with the following
  74. configuration options:
  75. There are also *node* level settings that control the caching of buffers
  76. (important when using direct buffers):
  77. [cols="<,<",options="header",]
  78. |=======================================================================
  79. |Setting |Description
  80. |`cache.memory.direct` |Should the memory be allocated outside of the
  81. JVM heap. Defaults to `true`.
  82. |`cache.memory.small_buffer_size` |The small buffer size, defaults to
  83. `1kb`.
  84. |`cache.memory.large_buffer_size` |The large buffer size, defaults to
  85. `1mb`.
  86. |`cache.memory.small_cache_size` |The small buffer cache size, defaults
  87. to `10mb`.
  88. |`cache.memory.large_cache_size` |The large buffer cache size, defaults
  89. to `500mb`.
  90. |=======================================================================