configuration.asciidoc 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378
  1. [[setup-configuration]]
  2. == Configuration
  3. [float]
  4. === Environment Variables
  5. Within the scripts, Elasticsearch comes with built in `JAVA_OPTS` passed
  6. to the JVM started. The most important setting for that is the `-Xmx` to
  7. control the maximum allowed memory for the process, and `-Xms` to
  8. control the minimum allocated memory for the process (_in general, the
  9. more memory allocated to the process, the better_).
  10. Most times it is better to leave the default `JAVA_OPTS` as they are,
  11. and use the `ES_JAVA_OPTS` environment variable in order to set / change
  12. JVM settings or arguments.
  13. The `ES_HEAP_SIZE` environment variable allows to set the heap memory
  14. that will be allocated to elasticsearch java process. It will allocate
  15. the same value to both min and max values, though those can be set
  16. explicitly (not recommended) by setting `ES_MIN_MEM` (defaults to
  17. `256m`), and `ES_MAX_MEM` (defaults to `1g`).
  18. It is recommended to set the min and max memory to the same value, and
  19. enable <<setup-configuration-memory,`mlockall`>>.
  20. [float]
  21. [[system]]
  22. === System Configuration
  23. [float]
  24. [[file-descriptors]]
  25. ==== File Descriptors
  26. Make sure to increase the number of open files descriptors on the
  27. machine (or for the user running elasticsearch). Setting it to 32k or
  28. even 64k is recommended.
  29. You can retrieve the `max_file_descriptors` for each node
  30. using the <<cluster-nodes-info>> API, with:
  31. [source,js]
  32. --------------------------------------------------
  33. curl localhost:9200/_nodes/stats/process?pretty
  34. --------------------------------------------------
  35. [float]
  36. [[vm-max-map-count]]
  37. ==== Virtual memory
  38. Elasticsearch uses a <<default_fs,`hybrid mmapfs / niofs`>> directory by default to store its indices. The default
  39. operating system limits on mmap counts is likely to be too low, which may
  40. result in out of memory exceptions. On Linux, you can increase the limits by
  41. running the following command as `root`:
  42. [source,sh]
  43. -------------------------------------
  44. sysctl -w vm.max_map_count=262144
  45. -------------------------------------
  46. To set this value permanently, update the `vm.max_map_count` setting in
  47. `/etc/sysctl.conf`.
  48. NOTE: If you installed Elasticsearch using a package (.deb, .rpm) this setting will be changed automatically. To verify, run `sysctl vm.max_map_count`.
  49. [float]
  50. [[setup-configuration-memory]]
  51. ==== Memory Settings
  52. Most operating systems try to use as much memory as possible for file system
  53. caches and eagerly swap out unused application memory, possibly resulting
  54. in the elasticsearch process being swapped. Swapping is very bad for
  55. performance and for node stability, so it should be avoided at all costs.
  56. There are three options:
  57. * **Disable swap**
  58. +
  59. --
  60. The simplest option is to completely disable swap. Usually Elasticsearch
  61. is the only service running on a box, and its memory usage is controlled
  62. by the `ES_HEAP_SIZE` environment variable. There should be no need
  63. to have swap enabled.
  64. On Linux systems, you can disable swap temporarily
  65. by running: `sudo swapoff -a`. To disable it permanently, you will need
  66. to edit the `/etc/fstab` file and comment out any lines that contain the
  67. word `swap`.
  68. On Windows, the equivalent can be achieved by disabling the paging file entirely
  69. via `System Properties → Advanced → Performance → Advanced → Virtual memory`.
  70. --
  71. * **Configure `swappiness`**
  72. +
  73. --
  74. The second option is to ensure that the sysctl value `vm.swappiness` is set
  75. to `0`. This reduces the kernel's tendency to swap and should not lead to
  76. swapping under normal circumstances, while still allowing the whole system
  77. to swap in emergency conditions.
  78. NOTE: From kernel version 3.5-rc1 and above, a `swappiness` of `0` will
  79. cause the OOM killer to kill the process instead of allowing swapping.
  80. You will need to set `swappiness` to `1` to still allow swapping in
  81. emergencies.
  82. --
  83. * **`mlockall`**
  84. +
  85. --
  86. The third option is to use
  87. http://opengroup.org/onlinepubs/007908799/xsh/mlockall.html[mlockall] on Linux/Unix systems, or https://msdn.microsoft.com/en-us/library/windows/desktop/aa366895%28v=vs.85%29.aspx[VirtualLock] on Windows, to
  88. try to lock the process address space into RAM, preventing any Elasticsearch
  89. memory from being swapped out. This can be done, by adding this line
  90. to the `config/elasticsearch.yml` file:
  91. [source,yaml]
  92. --------------
  93. bootstrap.mlockall: true
  94. --------------
  95. After starting Elasticsearch, you can see whether this setting was applied
  96. successfully by checking the value of `mlockall` in the output from this
  97. request:
  98. [source,sh]
  99. --------------
  100. curl http://localhost:9200/_nodes/process?pretty
  101. --------------
  102. If you see that `mlockall` is `false`, then it means that the `mlockall`
  103. request has failed. The most probable reason, on Linux/Unix systems, is that
  104. the user running Elasticsearch doesn't have permission to lock memory. This can
  105. be granted by running `ulimit -l unlimited` as `root` before starting Elasticsearch.
  106. Another possible reason why `mlockall` can fail is that the temporary directory
  107. (usually `/tmp`) is mounted with the `noexec` option. This can be solved by
  108. specifying a new temp directory, by starting Elasticsearch with:
  109. [source,sh]
  110. --------------
  111. ./bin/elasticsearch -Djna.tmpdir=/path/to/new/dir
  112. --------------
  113. WARNING: `mlockall` might cause the JVM or shell session to exit if it tries
  114. to allocate more memory than is available!
  115. --
  116. [float]
  117. [[settings]]
  118. === Elasticsearch Settings
  119. *elasticsearch* configuration files can be found under `ES_HOME/config`
  120. folder. The folder comes with two files, the `elasticsearch.yml` for
  121. configuring Elasticsearch different
  122. <<modules,modules>>, and `logging.yml` for
  123. configuring the Elasticsearch logging.
  124. The configuration format is http://www.yaml.org/[YAML]. Here is an
  125. example of changing the address all network based modules will use to
  126. bind and publish to:
  127. [source,yaml]
  128. --------------------------------------------------
  129. network :
  130. host : 10.0.0.4
  131. --------------------------------------------------
  132. [float]
  133. [[paths]]
  134. ==== Paths
  135. In production use, you will almost certainly want to change paths for
  136. data and log files:
  137. [source,yaml]
  138. --------------------------------------------------
  139. path:
  140. logs: /var/log/elasticsearch
  141. data: /var/data/elasticsearch
  142. --------------------------------------------------
  143. [float]
  144. [[cluster-name]]
  145. ==== Cluster name
  146. Also, don't forget to give your production cluster a name, which is used
  147. to discover and auto-join other nodes:
  148. [source,yaml]
  149. --------------------------------------------------
  150. cluster:
  151. name: <NAME OF YOUR CLUSTER>
  152. --------------------------------------------------
  153. Make sure that you don't reuse the same cluster names in different
  154. environments, otherwise you might end up with nodes joining the wrong cluster.
  155. For instance you could use `logging-dev`, `logging-stage`, and `logging-prod`
  156. for the development, staging, and production clusters.
  157. [float]
  158. [[node-name]]
  159. ==== Node name
  160. You may also want to change the default node name for each node to
  161. something like the display hostname. By default Elasticsearch will
  162. randomly pick a Marvel character name from a list of around 3000 names
  163. when your node starts up.
  164. [source,yaml]
  165. --------------------------------------------------
  166. node:
  167. name: <NAME OF YOUR NODE>
  168. --------------------------------------------------
  169. The hostname of the machine is provided in the environment
  170. variable `HOSTNAME`. If on your machine you only run a
  171. single elasticsearch node for that cluster, you can set
  172. the node name to the hostname using the `${...}` notation:
  173. [source,yaml]
  174. --------------------------------------------------
  175. node:
  176. name: ${HOSTNAME}
  177. --------------------------------------------------
  178. Internally, all settings are collapsed into "namespaced" settings. For
  179. example, the above gets collapsed into `node.name`. This means that
  180. its easy to support other configuration formats, for example,
  181. http://www.json.org[JSON]. If JSON is a preferred configuration format,
  182. simply rename the `elasticsearch.yml` file to `elasticsearch.json` and
  183. add:
  184. [float]
  185. [[styles]]
  186. ==== Configuration styles
  187. [source,yaml]
  188. --------------------------------------------------
  189. {
  190. "network" : {
  191. "host" : "10.0.0.4"
  192. }
  193. }
  194. --------------------------------------------------
  195. It also means that its easy to provide the settings externally either
  196. using the `ES_JAVA_OPTS` or as parameters to the `elasticsearch`
  197. command, for example:
  198. [source,sh]
  199. --------------------------------------------------
  200. $ elasticsearch -Des.network.host=10.0.0.4
  201. --------------------------------------------------
  202. Another option is to set `es.default.` prefix instead of `es.` prefix,
  203. which means the default setting will be used only if not explicitly set
  204. in the configuration file.
  205. Another option is to use the `${...}` notation within the configuration
  206. file which will resolve to an environment setting, for example:
  207. [source,js]
  208. --------------------------------------------------
  209. {
  210. "network" : {
  211. "host" : "${ES_NET_HOST}"
  212. }
  213. }
  214. --------------------------------------------------
  215. Additionally, for settings that you do not wish to store in the configuration
  216. file, you can use the value `${prompt.text}` or `${prompt.secret}` and start
  217. Elasticsearch in the foreground. `${prompt.secret}` has echoing disabled so
  218. that the value entered will not be shown in your terminal; `${prompt.text}`
  219. will allow you to see the value as you type it in. For example:
  220. [source,yaml]
  221. --------------------------------------------------
  222. node:
  223. name: ${prompt.text}
  224. --------------------------------------------------
  225. On execution of the `elasticsearch` command, you will be prompted to enter
  226. the actual value like so:
  227. [source,sh]
  228. --------------------------------------------------
  229. Enter value for [node.name]:
  230. --------------------------------------------------
  231. NOTE: Elasticsearch will not start if `${prompt.text}` or `${prompt.secret}`
  232. is used in the settings and the process is run as a service or in the background.
  233. [float]
  234. [[configuration-index-settings]]
  235. === Index Settings
  236. Indices created within the cluster can provide their own settings. For
  237. example, the following creates an index with a refresh interval of 5
  238. seconds instead of the default refresh interval (the format can be either
  239. YAML or JSON):
  240. [source,sh]
  241. --------------------------------------------------
  242. $ curl -XPUT http://localhost:9200/kimchy/ -d \
  243. '
  244. index:
  245. refresh_interval: 5s
  246. '
  247. --------------------------------------------------
  248. Index level settings can be set on the node level as well, for example,
  249. within the `elasticsearch.yml` file, the following can be set:
  250. [source,yaml]
  251. --------------------------------------------------
  252. index :
  253. refresh_interval: 5s
  254. --------------------------------------------------
  255. This means that every index that gets created on the specific node
  256. started with the mentioned configuration will use a refresh interval of
  257. 5 seconds *unless the index explicitly sets it*. In other words, any
  258. index level settings override what is set in the node configuration. Of
  259. course, the above can also be set as a "collapsed" setting, for example:
  260. [source,sh]
  261. --------------------------------------------------
  262. $ elasticsearch -Des.index.refresh_interval=5s
  263. --------------------------------------------------
  264. All of the index level configuration can be found within each
  265. <<index-modules,index module>>.
  266. [float]
  267. [[logging]]
  268. === Logging
  269. Elasticsearch uses an internal logging abstraction and comes, out of the
  270. box, with http://logging.apache.org/log4j/1.2/[log4j]. It tries to simplify
  271. log4j configuration by using http://www.yaml.org/[YAML] to configure it,
  272. and the logging configuration file is `config/logging.yml`. The
  273. http://en.wikipedia.org/wiki/JSON[JSON] and
  274. http://en.wikipedia.org/wiki/.properties[properties] formats are also
  275. supported. Multiple configuration files can be loaded, in which case they will
  276. get merged, as long as they start with the `logging.` prefix and end with one
  277. of the supported suffixes (either `.yml`, `.yaml`, `.json` or `.properties`).
  278. The logger section contains the java packages and their corresponding log
  279. level, where it is possible to omit the `org.elasticsearch` prefix. The
  280. appender section contains the destinations for the logs. Extensive information
  281. on how to customize logging and all the supported appenders can be found on
  282. the http://logging.apache.org/log4j/1.2/manual.html[log4j documentation].
  283. Additional Appenders and other logging classes provided by
  284. http://logging.apache.org/log4j/extras/[log4j-extras] are also available,
  285. out of the box.
  286. [float]
  287. [[deprecation-logging]]
  288. ==== Deprecation logging
  289. In addition to regular logging, Elasticsearch allows you to enable logging
  290. of deprecated actions. For example this allows you to determine early, if
  291. you need to migrate certain functionality in the future. By default,
  292. deprecation logging is disabled. You can enable it in the `config/logging.yml`
  293. file by setting the deprecation log level to `DEBUG`.
  294. [source,yaml]
  295. --------------------------------------------------
  296. deprecation: DEBUG, deprecation_log_file
  297. --------------------------------------------------
  298. This will create a daily rolling deprecation log file in your log directory.
  299. Check this file regularly, especially when you intend to upgrade to a new
  300. major version.