[ILM] add docs explaining high-level concepts (#35322)

add docs explaining high-level concepts

docs/reference/ilm/index.asciidoc

@@ -52,10 +52,12 @@ hardware.
 
 include::getting-started-ilm.asciidoc[]
 
-include::using-policies-rollover.asciidoc[]
+include::policy-definitions.asciidoc[]
 
 include::set-up-lifecycle-policy.asciidoc[]
 
+include::using-policies-rollover.asciidoc[]
+
 include::update-lifecycle-policy.asciidoc[]
 
 include::get-index-lifecycle-information.asciidoc[]

docs/reference/ilm/policy-definitions.asciidoc

@@ -0,0 +1,549 @@
+[role="xpack"]
+[testenv="basic"]
+[[ilm-policy-definition]]
+== Policy Phases and Actions
+
+There are four stages in the index lifecycle, in the order
+they are executed.
+
+[options="header"]
+|======
+| Name     | Description
+| `hot`    | The index is actively being written to
+| `warm`   | The index is generally not being written to, but is still queried
+| `cold`   | The index is no longer being updated and is seldom queried. The
+information still needs to be searchable, but it's okay if those queries are
+slower.
+| `delete` | The index is no longer needed and can safely be deleted
+|======
+
+Each of these stages is called a "phase". A policy does not need to configure
+each phase for an index. For example, one policy may define only the hot
+phase and the delete phase, while another may define all four phases.
+
+=== Timing
+
+Indices enter phases based on a phase's `min_age` parameter.
+The index will not enter the phase until the index's age is older than that
+of the `min_age`. The parameter is configured using a time
+duration format (see <<time-units, Time Units>>).
+
+`min_age` defaults to zero seconds `0s` for each phase if not specified.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "min_age": "1d",
+        "actions": {
+          "allocate": {
+            "number_of_replicas": 1
+          }
+        }
+      },
+      "delete": {
+        "min_age": "30d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+The Above example configures a policy that moves the index into the warm
+phase after one day. Until then, the index is in a waiting state. After
+moving into the warm phase, it will wait until 30 days have elapsed before
+moving to the delete phase and deleting the index.
+
+`min_age` is usually the time elapsed from the time the index is created. If the
+index is rolled over, then `min_age` is the time elapsed from the time the index
+is rolled over. The intention here is to execute following phases and actions
+relative to when data was written last to a rolled over index.
+
+=== Actions
+
+The below list shows the actions which are available in each phase.
+
+* Hot
+  - <<ilm-rollover-action,Rollover>>
+* Warm
+  - <<ilm-allocate-action,Allocate>>
+  - <<ilm-readonly-action,Read-Only>>
+  - <<ilm-forcemerge-action,Force Merge>>
+  - <<ilm-shrink-action,Shrink>>
+* Cold
+  - <<ilm-allocate-action,Allocate>>
+* Delete
+  - <<ilm-delete-action,Delete>>
+
+[[ilm-allocate-action]]
+==== Allocate
+
+Phases allowed: warm, cold.
+
+The Allocate action allows you to specify which nodes are allowed to host the
+shards of the index and set the number of replicas.
+Behind the scenes, it is modifying the index settings
+for shard filtering and/or replica counts. When updating the number of replicas,
+configuring allocation rules is optional. When configuring allocation rules,
+setting number of replicas is optional. Although this action can be treated as
+two separate index settings updates, both can be configured at once.
+
+Read more about index replicas <<getting-started-shards-and-replicas,here>>.
+Read more about shard allocation filtering in
+the <<shard-allocation-filtering,Shard allocation filtering documentation>>.
+
+[[ilm-allocate-options]]
+.Allocate Options
+[options="header"]
+|======
+| Name                 | Required  | Default     | Description
+| `number_of_replicas` | no        | -           | The number of replicas to
+                                                   assign to the index
+| `include`            | no        | -           | assigns an index to nodes
+                                                   having at least _one_ of the attributes
+| `exclude`            | no        | -           | assigns an index to nodes having
+                                                   _none_ of the attributes
+| `require`            | no        | -           | assigns an index to nodes having
+                                                   _all_ of the attributes
+|======
+
+If `number_of_replicas` is not configured, then at least one of `include`,
+`exclude`, and `require` is required. An empty Allocate Action with no configuration
+is invalid.
+
+===== Example: Change number of replicas
+
+In this example, the index's number of replicas is changed to `2`, while allocation
+rules are unchanged.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "allocate" : {
+            "number_of_replicas" : 2
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+===== Example: Assign index to node with specific "box_type" attribute
+
+This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "allocate" : {
+            "include" : {
+              "box_type": "hot,warm"
+            }
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+===== Example: Assign index to a specific node and update replica settings
+
+This example updates the index to have one replica per shard and be allocated
+to nodes with a `box_type` attribute of "cold".
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "allocate" : {
+            "number_of_replicas": 1,
+            "require" : {
+              "box_type": "cold"
+            }
+        }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+[[ilm-delete-action]]
+==== Delete
+
+Phases allowed: delete.
+
+The Delete Action does just that, it deletes the index.
+
+This action does not have any options associated with it.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "delete": {
+        "actions": {
+          "delete" : { }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+[[ilm-forcemerge-action]]
+==== Force Merge
+
+Phases allowed: warm.
+
+NOTE: Index will be be made read-only when this action is run
+(see: <<dynamic-index-settings,index.blocks.write>>)
+
+The Force Merge Action <<indices-forcemerge,force merges>> the index into at
+most a specific number of <<indices-segments,segments>>.
+
+[[ilm-forcemerge-options]]
+.Force Merge Options
+[options="header"]
+|======
+| Name                 | Required  | Default             | Description
+| `max_num_segments`   | yes       | -                   | The number of
+                                                           segments to merge to.
+                                                           To fully merge the
+                                                           index, set it to `1`
+|======
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "forcemerge" : {
+            "max_num_segments": 1
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+[[ilm-readonly-action]]
+==== Read-Only
+
+Phases allowed: warm.
+
+This action will set the index to be read-only
+(see: <<dynamic-index-settings,index.blocks.write>>)
+
+This action does not have any options associated with it.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "readonly" : { }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+[[ilm-rollover-action]]
+==== Rollover
+
+Phases allowed: hot.
+
+[WARNING]
+index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
+[WARNING]
+The managed index must set `index.lifecycle.rollover_alias` as the
+alias to rollover. The index must also be the write index for the alias.
+
+For example, if an index to be managed has an alias `my_data`. The managed
+index "my_index" must be the write index for the alias. For more information, read
+<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
+
+[source,js]
+--------------------------------------------------
+PUT my_index
+{
+  "settings": {
+    "index.lifecycle.name": "my_policy",
+    "index.lifecycle.rollover_alias": "my_data"
+  },
+  "aliases": {
+    "my_data": {
+      "is_write_index": true
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+The Rollover Action rolls an alias over to a new index when the
+existing index meets one of the rollover conditions.
+
+
+[[ilm-rollover-options]]
+.Rollover Options
+[options="header"]
+|======
+| Name       | Required  | Default             | Description
+| `max_size` | no        | -                   | max index storage size.
+                                                 See <<byte-units, Byte Units>>
+                                                 for formatting
+| `max_docs` | no        | -                   | max number of documents an
+                                                 index is to contain before
+                                                 rolling over.
+| `max_age`  | no        | -                   | max time elapsed from index
+                                                 creation. See
+                                                 <<time-units, Time Units>>
+                                                 for formatting
+|======
+
+At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
+three are required to be specified.
+
+===== Example: Rollover when index is too large
+
+This example rolls the index over when it is at least 100 gigabytes.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover" : {
+            "max_size": "100GB"
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+===== Example: Rollover when index has too many documents
+
+This example rolls the index over when it contains at least
+1000000 documents.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover" : {
+            "max_docs": 1000000
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+===== Example: Rollover when index is too old
+
+This example rolls the index over when it has been created at least
+7 days ago.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover" : {
+            "max_age": "7d"
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+===== Example: Rollover when index is too old or too large
+
+This example rolls the index over when it has been created at least
+7 days ago or it is at least 100 gigabytes. In this case, the index will be
+rolled over when any of the conditions is met.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover" : {
+            "max_age": "7d",
+            "max_size": "100GB"
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+
+[[ilm-shrink-action]]
+==== Shrink
+
+NOTE: Index will be be made read-only when this action is run
+(see: <<dynamic-index-settings,index.blocks.write>>)
+
+This action shrinks an existing index into a new index with fewer primary
+shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
+Since allocating all the primary shards of the index to one node is a
+prerequisite, this action will first allocate the primary shards to a valid
+node. After shrinking, it will swap aliases pointing to the original index
+into the new shrunken index. The new index will also have a new name:
+"shrink-<origin-index-name>". So if the original index was called "logs",
+then the new index will be named "shrink-logs".
+
+[[ilm-shrink-options]]
+.Shrink Options
+[options="header"]
+|======
+| Name               | Required  | Default             | Description
+| `number_of_shards` | yes       | -                   | The number of shards
+                                                         to shrink to. must be
+                                                         a factor of the number
+                                                         of shards in the
+                                                         source index.
+|======
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "shrink" : {
+            "number_of_shards": 1
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
+
+=== Full Policy
+
+With all of these actions, we can support complex management strategies for our
+indices. This policy will define an index that will start in the hot phase,
+rolling over every 20g or 7 days. After 30 days it enters the warm phase
+and increases the replicas to 2, force merges and shrinks. After 60 days
+it enters the cold phase and allocates to "cold" nodes, and after 90 days the
+index is deleted.
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/full_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover": {
+            "max_age": "7d",
+            "max_size": "20G"
+          }
+        }
+      },
+      "warm": {
+        "min_age": "30d",
+        "actions": {
+          "forcemerge": {
+            "max_num_segments": 1
+          },
+          "shrink": {
+            "number_of_shards": 1
+          },
+          "allocate": {
+            "number_of_replicas": 2
+          }
+        }
+      },
+      "cold": {
+        "min_age": "60d",
+        "actions": {
+          "allocate": {
+            "require": {
+              "type": "cold"
+            }
+          }
+        }
+      },
+      "delete": {
+        "min_age": "90d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE