policy-definitions.asciidoc 22 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[ilm-policy-definition]]
  4. == Policy phases and actions
  5. There are four stages in the index lifecycle, in the order
  6. they are executed.
  7. [options="header"]
  8. |======
  9. | Name | Description
  10. | `hot` | The index is actively being written to
  11. | `warm` | The index is generally not being written to, but is still queried
  12. | `cold` | The index is no longer being updated and is seldom queried. The
  13. information still needs to be searchable, but it's okay if those queries are
  14. slower.
  15. | `delete` | The index is no longer needed and can safely be deleted
  16. |======
  17. Each of these stages is called a "phase". A policy does not need to configure
  18. each phase for an index. For example, one policy may define only the hot
  19. phase and the delete phase, while another may define all four phases.
  20. === Timing
  21. Indices enter phases based on a phase's `min_age` parameter.
  22. The index will not enter the phase until the index's age is older than that
  23. of the `min_age`. The parameter is configured using a time
  24. duration format (see <<time-units, Time Units>>).
  25. `min_age` defaults to zero seconds `0s` for each phase if not specified.
  26. [source,console]
  27. --------------------------------------------------
  28. PUT _ilm/policy/my_policy
  29. {
  30. "policy": {
  31. "phases": {
  32. "warm": {
  33. "min_age": "1d",
  34. "actions": {
  35. "allocate": {
  36. "number_of_replicas": 1
  37. }
  38. }
  39. },
  40. "delete": {
  41. "min_age": "30d",
  42. "actions": {
  43. "delete": {}
  44. }
  45. }
  46. }
  47. }
  48. }
  49. --------------------------------------------------
  50. The Above example configures a policy that moves the index into the warm
  51. phase after one day. Until then, the index is in a waiting state. After
  52. moving into the warm phase, it will wait until 30 days have elapsed before
  53. moving to the delete phase and deleting the index.
  54. `min_age` is usually the time elapsed from the time the index is created, unless
  55. the `index.lifecycle.origination_date` index setting is configured, in which
  56. case the `min_age` will be the time elapsed since that specified date. If the
  57. index is rolled over, then `min_age` is the time elapsed from the time the
  58. index is rolled over. The intention here is to execute following phases and
  59. actions relative to when data was written last to a rolled over index.
  60. The previous phase's actions must complete before {ilm} will check `min_age` and
  61. transition into the next phase. By default, {ilm} checks for indices that meet
  62. policy criteria, like `min_age`, every 10 minutes. You can use the
  63. `indices.lifecycle.poll_interval` cluster setting to control how often this
  64. check occurs.
  65. === Phase Execution
  66. The current phase definition, of an index's policy being executed, is stored
  67. in the index's metadata. The phase and its actions are compiled into a series
  68. of discrete steps that are executed sequentially. Since some {ilm-init} actions
  69. are more complex and involve multiple operations against an index, each of these
  70. operations are done in isolation in a unit called a "step". The
  71. <<ilm-explain-lifecycle,Explain Lifecycle API>> exposes this information to us
  72. to see which step our index is either to execute next, or is currently
  73. executing.
  74. === Actions
  75. The below list shows the actions which are available in each phase.
  76. NOTE: The order that configured actions are performed in within each phase is
  77. determined automatically by {ilm-init}, and cannot be changed by changing the
  78. policy definition.
  79. * Hot
  80. - <<ilm-set-priority-action,Set Priority>>
  81. - <<ilm-unfollow-action,Unfollow>>
  82. - <<ilm-rollover-action,Rollover>>
  83. * Warm
  84. - <<ilm-set-priority-action,Set Priority>>
  85. - <<ilm-unfollow-action,Unfollow>>
  86. - <<ilm-readonly-action,Read-Only>>
  87. - <<ilm-allocate-action,Allocate>>
  88. - <<ilm-shrink-action,Shrink>>
  89. - <<ilm-forcemerge-action,Force Merge>>
  90. * Cold
  91. - <<ilm-set-priority-action,Set Priority>>
  92. - <<ilm-unfollow-action,Unfollow>>
  93. - <<ilm-allocate-action,Allocate>>
  94. - <<ilm-freeze-action,Freeze>>
  95. * Delete
  96. - <<ilm-delete-action,Wait For Snapshot>>
  97. - <<ilm-delete-action,Delete>>
  98. [[ilm-allocate-action]]
  99. ==== Allocate
  100. Phases allowed: warm, cold.
  101. The Allocate action allows you to specify which nodes are allowed to host the
  102. shards of the index and set the number of replicas.
  103. Behind the scenes, it is modifying the index settings
  104. for shard filtering and/or replica counts. When updating the number of replicas,
  105. configuring allocation rules is optional. When configuring allocation rules,
  106. setting number of replicas is optional. Although this action can be treated as
  107. two separate index settings updates, both can be configured at once.
  108. For more information about how {es} uses replicas for scaling, see
  109. <<scalability>>. See <<shard-allocation-filtering>> for more information about
  110. controlling where Elasticsearch allocates shards of a particular index.
  111. [[ilm-allocate-options]]
  112. .Allocate Options
  113. [options="header"]
  114. |======
  115. | Name | Required | Default | Description
  116. | `number_of_replicas` | no | - | The number of replicas to
  117. assign to the index
  118. | `include` | no | - | assigns an index to nodes
  119. having at least _one_ of the attributes
  120. | `exclude` | no | - | assigns an index to nodes having
  121. _none_ of the attributes
  122. | `require` | no | - | assigns an index to nodes having
  123. _all_ of the attributes
  124. |======
  125. If `number_of_replicas` is not configured, then at least one of `include`,
  126. `exclude`, and `require` is required. An empty Allocate Action with no configuration
  127. is invalid.
  128. ===== Example: Change number of replicas
  129. In this example, the index's number of replicas is changed to `2`, while allocation
  130. rules are unchanged.
  131. [source,console]
  132. --------------------------------------------------
  133. PUT _ilm/policy/my_policy
  134. {
  135. "policy": {
  136. "phases": {
  137. "warm": {
  138. "actions": {
  139. "allocate" : {
  140. "number_of_replicas" : 2
  141. }
  142. }
  143. }
  144. }
  145. }
  146. }
  147. --------------------------------------------------
  148. ===== Example: Assign index to node with specific "box_type" attribute
  149. This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
  150. [source,console]
  151. --------------------------------------------------
  152. PUT _ilm/policy/my_policy
  153. {
  154. "policy": {
  155. "phases": {
  156. "warm": {
  157. "actions": {
  158. "allocate" : {
  159. "include" : {
  160. "box_type": "hot,warm"
  161. }
  162. }
  163. }
  164. }
  165. }
  166. }
  167. }
  168. --------------------------------------------------
  169. ===== Example: Assign index to a specific node and update replica settings
  170. This example updates the index to have one replica per shard and be allocated
  171. to nodes with a `box_type` attribute of "cold".
  172. [source,console]
  173. --------------------------------------------------
  174. PUT _ilm/policy/my_policy
  175. {
  176. "policy": {
  177. "phases": {
  178. "warm": {
  179. "actions": {
  180. "allocate" : {
  181. "number_of_replicas": 1,
  182. "require" : {
  183. "box_type": "cold"
  184. }
  185. }
  186. }
  187. }
  188. }
  189. }
  190. }
  191. --------------------------------------------------
  192. [[ilm-wait-for-snapshot-action]]
  193. ==== Wait For Snapshot
  194. Phases allowed: delete.
  195. The Wait For Snapshot Action waits for defined SLM policy to be executed to ensure that snapshot of index exists before
  196. deletion.
  197. [[ilm-wait-for-snapshot-options]]
  198. .Wait For Snapshot
  199. [options="header"]
  200. |======
  201. | Name | Required | Default | Description
  202. | `policy` | yes | - | SLM policy name that this action should wait for
  203. |======
  204. [source,console]
  205. --------------------------------------------------
  206. PUT _ilm/policy/my_policy
  207. {
  208. "policy": {
  209. "phases": {
  210. "delete": {
  211. "actions": {
  212. "wait_for_snapshot" : {
  213. "policy": "slm-policy-name"
  214. }
  215. }
  216. }
  217. }
  218. }
  219. }
  220. --------------------------------------------------
  221. [[ilm-delete-action]]
  222. ==== Delete
  223. Phases allowed: delete.
  224. The Delete Action does just that, it deletes the index.
  225. This action does not have any options associated with it.
  226. [source,console]
  227. --------------------------------------------------
  228. PUT _ilm/policy/my_policy
  229. {
  230. "policy": {
  231. "phases": {
  232. "delete": {
  233. "actions": {
  234. "delete" : { }
  235. }
  236. }
  237. }
  238. }
  239. }
  240. --------------------------------------------------
  241. [[ilm-forcemerge-action]]
  242. ==== Force Merge
  243. Phases allowed: warm.
  244. NOTE: Index will be be made read-only when this action is run
  245. (see: <<dynamic-index-settings,index.blocks.write>>)
  246. The Force Merge Action <<indices-forcemerge,force merges>> the index into at
  247. most a specific number of <<indices-segments,segments>>.
  248. [[ilm-forcemerge-options]]
  249. .Force Merge Options
  250. [options="header"]
  251. |======
  252. | Name | Required | Default | Description
  253. | `max_num_segments` | yes | - | The number of
  254. segments to merge to.
  255. To fully merge the
  256. index, set it to `1`
  257. |======
  258. [source,console]
  259. --------------------------------------------------
  260. PUT _ilm/policy/my_policy
  261. {
  262. "policy": {
  263. "phases": {
  264. "warm": {
  265. "actions": {
  266. "forcemerge" : {
  267. "max_num_segments": 1
  268. }
  269. }
  270. }
  271. }
  272. }
  273. }
  274. --------------------------------------------------
  275. [[ilm-freeze-action]]
  276. ==== Freeze
  277. Phases allowed: cold.
  278. This action will <<frozen-indices, freeze>> the index
  279. by calling the <<freeze-index-api, Freeze Index API>>.
  280. [source,console]
  281. --------------------------------------------------
  282. PUT _ilm/policy/my_policy
  283. {
  284. "policy": {
  285. "phases": {
  286. "cold": {
  287. "actions": {
  288. "freeze" : { }
  289. }
  290. }
  291. }
  292. }
  293. }
  294. --------------------------------------------------
  295. [IMPORTANT]
  296. ================================
  297. Freezing an index will close the index and reopen it within the same API call.
  298. This causes primaries to not be allocated for a short amount of time and
  299. causes the cluster to go red until the primaries are allocated again.
  300. This limitation might be removed in the future.
  301. ================================
  302. [[ilm-readonly-action]]
  303. ==== Read-Only
  304. Phases allowed: warm.
  305. This action will set the index to be read-only
  306. (see: <<dynamic-index-settings,index.blocks.write>>)
  307. This action does not have any options associated with it.
  308. [source,console]
  309. --------------------------------------------------
  310. PUT _ilm/policy/my_policy
  311. {
  312. "policy": {
  313. "phases": {
  314. "warm": {
  315. "actions": {
  316. "readonly" : { }
  317. }
  318. }
  319. }
  320. }
  321. }
  322. --------------------------------------------------
  323. [[ilm-rollover-action]]
  324. ==== Rollover
  325. Phases allowed: hot.
  326. [WARNING]
  327. index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
  328. [WARNING]
  329. The managed index must set `index.lifecycle.rollover_alias` as the
  330. alias to rollover. The index must also be the write index for the alias.
  331. [IMPORTANT]
  332. If a policy using the Rollover action is used on a <<ccr-put-follow,follower
  333. index>>, policy execution will wait until the leader index rolls over (or has
  334. <<skipping-rollover, otherwise been marked as complete>>), then convert the
  335. follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
  336. action>> had been used instead of rolling over.
  337. For example, if an index to be managed has an alias `my_data`. The managed
  338. index "my_index-000001" must be the write index for the alias. For more information, read
  339. <<indices-rollover-is-write-index,Write Index Alias Behavior>>.
  340. [source,console]
  341. --------------------------------------------------
  342. PUT my_index-000001
  343. {
  344. "settings": {
  345. "index.lifecycle.name": "my_policy",
  346. "index.lifecycle.rollover_alias": "my_data"
  347. },
  348. "aliases": {
  349. "my_data": {
  350. "is_write_index": true
  351. }
  352. }
  353. }
  354. --------------------------------------------------
  355. The Rollover Action rolls an alias over to a new index when the
  356. existing index meets one of the rollover conditions.
  357. [[ilm-rollover-options]]
  358. .Rollover Options
  359. [options="header"]
  360. |======
  361. | Name | Required | Default | Description
  362. | `max_size` | no | - | max primary shard index storage size.
  363. See <<byte-units, Byte Units>>
  364. for formatting
  365. | `max_docs` | no | - | max number of documents an
  366. index is to contain before
  367. rolling over.
  368. | `max_age` | no | - | max time elapsed from index
  369. creation. See
  370. <<time-units, Time Units>>
  371. for formatting
  372. |======
  373. At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
  374. three are required to be specified.
  375. ===== Example: Rollover when index is too large
  376. This example rolls the index over when it is at least 100 gigabytes.
  377. [source,console]
  378. --------------------------------------------------
  379. PUT _ilm/policy/my_policy
  380. {
  381. "policy": {
  382. "phases": {
  383. "hot": {
  384. "actions": {
  385. "rollover" : {
  386. "max_size": "100GB"
  387. }
  388. }
  389. }
  390. }
  391. }
  392. }
  393. --------------------------------------------------
  394. ===== Example: Rollover when index has too many documents
  395. This example rolls the index over when it contains at least
  396. 100000000 documents.
  397. [source,console]
  398. --------------------------------------------------
  399. PUT _ilm/policy/my_policy
  400. {
  401. "policy": {
  402. "phases": {
  403. "hot": {
  404. "actions": {
  405. "rollover" : {
  406. "max_docs": 100000000
  407. }
  408. }
  409. }
  410. }
  411. }
  412. }
  413. --------------------------------------------------
  414. ===== Example: Rollover when index is too old
  415. This example rolls the index over when it has been created at least
  416. 7 days ago.
  417. [source,console]
  418. --------------------------------------------------
  419. PUT _ilm/policy/my_policy
  420. {
  421. "policy": {
  422. "phases": {
  423. "hot": {
  424. "actions": {
  425. "rollover" : {
  426. "max_age": "7d"
  427. }
  428. }
  429. }
  430. }
  431. }
  432. }
  433. --------------------------------------------------
  434. ===== Example: Rollover when index is too old or too large
  435. This example rolls the index over when it has been created at least
  436. 7 days ago or it is at least 100 gigabytes. In this case, the index will be
  437. rolled over when any of the conditions is met.
  438. [source,console]
  439. --------------------------------------------------
  440. PUT _ilm/policy/my_policy
  441. {
  442. "policy": {
  443. "phases": {
  444. "hot": {
  445. "actions": {
  446. "rollover" : {
  447. "max_age": "7d",
  448. "max_size": "100GB"
  449. }
  450. }
  451. }
  452. }
  453. }
  454. }
  455. --------------------------------------------------
  456. ===== Example: Rollover condition stalls phase transition
  457. The Rollover action will only complete once one of its conditions is
  458. met. This means that any proceeding phases will be blocked until Rollover
  459. succeeds.
  460. [source,console]
  461. --------------------------------------------------
  462. PUT /_ilm/policy/rollover_policy
  463. {
  464. "policy": {
  465. "phases": {
  466. "hot": {
  467. "actions": {
  468. "rollover": {
  469. "max_size": "50G"
  470. }
  471. }
  472. },
  473. "delete": {
  474. "min_age": "1d",
  475. "actions": {
  476. "delete": {}
  477. }
  478. }
  479. }
  480. }
  481. }
  482. --------------------------------------------------
  483. The above example illustrates a policy which attempts to delete an
  484. index one day after the index has been rolled over. It does not
  485. delete the index one day after it has been created.
  486. [[ilm-set-priority-action]]
  487. ==== Set Priority
  488. Phases allowed: hot, warm, cold.
  489. This action sets the <<recovery-prioritization, index priority>> on the index as
  490. soon as the policy enters the hot, warm, or cold phase. Indices with a higher
  491. priority will be recovered before indices with lower priorities following a node
  492. restart. Generally, indexes in the hot phase should have the highest value and
  493. indexes in the cold phase should have the lowest values. For example:
  494. 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
  495. Indicies that don't set this value have an implicit default priority of 1.
  496. [[ilm-set-priority-options]]
  497. .Set Priority Options
  498. [options="header"]
  499. |======
  500. | Name | Required | Default | Description
  501. | `priority` | yes | - | The priority for the index. Must be 0 or greater.
  502. The value may also be set to null to remove the priority.
  503. |======
  504. [source,console]
  505. --------------------------------------------------
  506. PUT _ilm/policy/my_policy
  507. {
  508. "policy": {
  509. "phases": {
  510. "warm": {
  511. "actions": {
  512. "set_priority" : {
  513. "priority": 50
  514. }
  515. }
  516. }
  517. }
  518. }
  519. }
  520. --------------------------------------------------
  521. [[ilm-shrink-action]]
  522. ==== Shrink
  523. NOTE: Index will be be made read-only when this action is run
  524. (see: <<dynamic-index-settings,index.blocks.write>>)
  525. [IMPORTANT]
  526. If a policy using the Shrink action is used on a <<ccr-put-follow,follower
  527. index>>, policy execution will wait until the leader index rolls over (or has
  528. <<skipping-rollover, otherwise been marked as complete>>), then convert the
  529. follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
  530. action>> had been used before shrink is applied, as shrink cannot be safely
  531. applied to follower indices.
  532. This action shrinks an existing index into a new index with fewer primary
  533. shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
  534. Since allocating all the primary shards of the index to one node is a
  535. prerequisite, this action will first allocate the primary shards to a valid
  536. node. After shrinking, it will swap aliases pointing to the original index
  537. into the new shrunken index. The new index will also have a new name:
  538. "shrink-<origin-index-name>". So if the original index was called "logs",
  539. then the new index will be named "shrink-logs".
  540. [[ilm-shrink-options]]
  541. .Shrink Options
  542. [options="header"]
  543. |======
  544. | Name | Required | Default | Description
  545. | `number_of_shards` | yes | - | The number of shards
  546. to shrink to. must be
  547. a factor of the number
  548. of shards in the
  549. source index.
  550. |======
  551. [source,console]
  552. --------------------------------------------------
  553. PUT _ilm/policy/my_policy
  554. {
  555. "policy": {
  556. "phases": {
  557. "warm": {
  558. "actions": {
  559. "shrink" : {
  560. "number_of_shards": 1
  561. }
  562. }
  563. }
  564. }
  565. }
  566. }
  567. --------------------------------------------------
  568. [[ilm-unfollow-action]]
  569. ==== Unfollow
  570. [IMPORTANT]
  571. This action may be used explicitly, as shown below, but this action is also run
  572. before <<ilm-rollover-action,the Rollover action>> and <<ilm-shrink-action,the
  573. Shrink action>> as described in the documentation for those actions.
  574. This action turns a {ref}/ccr-apis.html[ccr] follower index
  575. into a regular index. This can be desired when moving follower
  576. indices into the next phase. Also certain actions like shrink
  577. and rollover can then be performed safely on follower indices.
  578. This action will wait until is it safe to convert a follower index into a
  579. regular index. In particular, the following conditions must be met:
  580. * The leader index must have `index.lifecycle.indexing_complete` set to `true`.
  581. This happens automatically if the leader index is rolled over using
  582. <<ilm-rollover-action,the Rollover action>>, or may be set manually using
  583. the <<indices-update-settings,Index Settings API>>.
  584. * All operations performed on the leader index must have been replicated to the
  585. follower index. This ensures that no operations will be lost when the index is
  586. converted into a regular index.
  587. If the unfollow action encounters a follower index then
  588. the following operations will be performed on it:
  589. * Pauses indexing following for the follower index.
  590. * Closes the follower index.
  591. * Unfollows the follower index.
  592. * Opens the follower index (which is at this point is a regular index).
  593. The unfollow action does not have any options and
  594. if it encounters a non follower index, then the
  595. unfollow action leaves that index untouched and
  596. lets the next action operate on this index.
  597. [source,console]
  598. --------------------------------------------------
  599. PUT _ilm/policy/my_policy
  600. {
  601. "policy": {
  602. "phases": {
  603. "hot": {
  604. "actions": {
  605. "unfollow" : {}
  606. }
  607. }
  608. }
  609. }
  610. }
  611. --------------------------------------------------
  612. === Full Policy
  613. With all of these actions, we can support complex management strategies for our
  614. indices. This policy will define an index that will start in the hot phase,
  615. rolling over every 50 GB or 7 days. After 30 days it enters the warm phase
  616. and increases the replicas to 2, force merges and shrinks. After 60 days
  617. it enters the cold phase and allocates to "cold" nodes, and after 90 days the
  618. index is deleted.
  619. [source,console]
  620. --------------------------------------------------
  621. PUT _ilm/policy/full_policy
  622. {
  623. "policy": {
  624. "phases": {
  625. "hot": {
  626. "actions": {
  627. "rollover": {
  628. "max_age": "7d",
  629. "max_size": "50G"
  630. }
  631. }
  632. },
  633. "warm": {
  634. "min_age": "30d",
  635. "actions": {
  636. "forcemerge": {
  637. "max_num_segments": 1
  638. },
  639. "shrink": {
  640. "number_of_shards": 1
  641. },
  642. "allocate": {
  643. "number_of_replicas": 2
  644. }
  645. }
  646. },
  647. "cold": {
  648. "min_age": "60d",
  649. "actions": {
  650. "allocate": {
  651. "require": {
  652. "type": "cold"
  653. }
  654. }
  655. }
  656. },
  657. "delete": {
  658. "min_age": "90d",
  659. "actions": {
  660. "delete": {}
  661. }
  662. }
  663. }
  664. }
  665. }
  666. --------------------------------------------------