[ILM] getting started documentation (#35379)

docs/reference/ilm/getting-started-ilm.asciidoc

@@ -1,15 +1,228 @@
 [role="xpack"]
+[testenv="basic"]
 [[getting-started-index-lifecycle-management]]
 == Getting started with {ilm}
 
-Create a policy that rolls over after 1 day deletes an index after 30 days
+Let's jump into {ILM} by working through a hands-on scenario.
+This section will leverage many new concepts unique to {ILM} that
+you may not be familiar with. The following sections will explore
+these in more details.
 
-Show create policy API req/res
+The goal of this example is to set up a set of indices that will encapsulate
+the data from a time series data source. We can imagine there is a system
+like {filebeat-ref}[Filebeat] that continuously indexes documents into
+our writing index. We wish to roll over the index after it reaches a size
+of 50 gigabytes, or has been created 30 days ago, and then delete the index
+after 90 days.
 
-Show assign policy to index API req/res
+=== Setting up a new policy
 
-Show both the API and how it is done with `index.lifecyce.name` using the
-create-index API
+There are many new features introduced by {ILM}, but we will only focus on
+a few that are needed for our example. For starters, we will use the
+<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
+policies are defined in JSON and include specific
+<<ilm-policy-definition,phases and actions>>.
 
-Show explain API to show current state, but ignore the “step” related info,
-only focus on managed/phase/action
+[source,js]
+------------------------
+PUT _ilm/policy/datastream_policy   <1>
+{
+  "policy": {                       <2>
+    "phases": {
+      "hot": {                      <3>
+        "actions": {
+          "rollover": {             <4>
+            "max_size": "50GB",
+            "max_age": "30d"
+          }
+        }
+      },
+      "delete": {
+        "min_age": "90d",           <5>
+        "actions": {
+          "delete": {}              <6>
+        }
+      }
+    }
+  }
+}
+------------------------
+// CONSOLE
+// TEST
+<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
+    a new policy named "datastream_policy"
+<2> policy definition sub-object
+<3> the hot phase defined in the "phases" section. Optional `min_age` field
+    not defined -- defaults to `0ms`
+<4> rollover action definition
+<5> delete phase begins after 90 days
+<6> delete action definition
+
+
+Here we created the policy called `datastream_policy` which rolls over
+the index being written to after it reaches 50 gigabytes, or it is 30
+days old. The rollover will occur when either of these conditions is true.
+The index will be deleted 90 days after it is rolled over.
+
+=== Applying a policy to our index
+
+There are <<set-up-lifecycle-policy,a few ways>> to associate a
+policy to an index. Since we wish specific settings to be applied to
+the new index created from Rollover, we will set the policy via
+index templates.
+
+
+[source,js]
+-----------------------
+PUT _template/datastream_template
+{
+  "index_patterns": ["datastream-*"],                 <1>
+  "settings": {
+    "number_of_shards": 1,
+    "number_of_replicas": 1,
+    "index.lifecycle.name": "datastream_policy",      <2>
+    "index.lifecycle.rollover_alias": "datastream"    <3>
+  }
+}
+-----------------------
+// CONSOLE
+// TEST[continued]
+<1> match all indices starting with "datastream-". These will include all
+    newly created indices from actions like rollover
+<2> the name of the lifecycle policy managing the index
+<3> alias to use for the rollover action, required since a rollover action is
+    defined in the policy.
+
+The above index template introduces a few new settings specific to {ILM}.
+The first being `index.lifecycle.name`. This setting will configure
+the "datastream_policy" to the index applying this template. This means
+that all newly created indices prefixed "datastream-" will be managed by
+our policy. The other setting used here is `index.lifecycle.rollover_alias`.
+This setting is required when using a policy containing the rollover
+action and specifies which alias to rollover on behalf of this index.
+The intention here is that the rollover alias is also defined on the index.
+
+To begin, we will want to bootstrap our first index to write to.
+
+
+[source,js]
+-----------------------
+PUT datastream-000001
+{
+  "aliases": {
+    "datastream": {
+      "is_write_index": true
+    }
+  }
+}
+-----------------------
+// CONSOLE
+// TEST[continued]
+
+When creating our index, we have to consider a few important configurations
+that tie our index and our policy together correctly. We need to make sure
+that our index name matches our index template pattern of "datastream-*",
+which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
+requires that our index name ends with a number. In our case, we used
+`000001`. This is important so that Rollover can increment this number when
+naming the new index created from rolling over.
+
+Our index creation request leverages its template to apply our settings,
+but we must also configure our rollover alias: "datastream". To do this,
+we take advantage of <<aliases-write-index,write indices>>. This is a way
+to define an alias to be used for both reading and writing, with only one
+index being the index that is being written to at a time. Rollover swaps
+the write index to be the new index created from rollover, and sets the
+alias to be read-only for the source index.
+
+=== Checking progress
+
+Now that we have an index managed by our policy, how do we tell what is going
+on? Which phase are we in? Is something broken? This section will go over a
+few APIs and their responses to help us inspect our indices with respect
+to {ILM}.
+
+With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
+things like which phase we're in and when we entered that phase. The API
+will also provide further info if errors occurred, or if we are blocked on
+certain checks within actions.
+
+[source,js]
+--------------------------------------------------
+GET datastream-*/_ilm/explain
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+The above request will retrieve {ILM} execution information for all our
+managed indices.
+
+
+[source,js]
+--------------------------------------------------
+{
+  "indices": {
+    "datastream-000001": {
+      "index": "datastream-000001",
+      "managed": true,                           <1>
+      "policy": "datastream_policy",             <2>
+      "lifecycle_date_millis": 1538475653281,
+      "phase": "hot",                            <3>
+      "phase_time_millis": 1538475653317,
+      "action": "rollover",                      <4>
+      "action_time_millis": 1538475653317,
+      "step": "attempt_rollover",                <5>
+      "step_time_millis": 1538475653317,
+      "phase_execution": {
+        "policy": "datastream_policy",
+        "phase_definition": {                    <6>
+          "min_age": "0ms",
+          "actions": {
+            "rollover": {
+              "max_size": "50gb",
+              "max_age": "30d"
+            }
+          }
+        },
+        "version": 1,                            <7>
+        "modified_date_in_millis": 1539609701576
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+// TESTRESPONSE[s/"lifecycle_date_millis": 1538475653281/"lifecycle_date_millis": $body.indices.datastream-000001.lifecycle_date_millis/]
+// TESTRESPONSE[s/"phase_time_millis": 1538475653317/"phase_time_millis": $body.indices.datastream-000001.phase_time_millis/]
+// TESTRESPONSE[s/"action_time_millis": 1538475653317/"action_time_millis": $body.indices.datastream-000001.action_time_millis/]
+// TESTRESPONSE[s/"step_time_millis": 1538475653317/"step_time_millis": $body.indices.datastream-000001.step_time_millis/]
+// TESTRESPONSE[s/"modified_date_in_millis": 1539609701576/"modified_date_in_millis": $body.indices.datastream-000001.phase_execution.modified_date_in_millis/]
+<1> this index is managed by ILM
+<2> the policy in question, in this case, "datastream_policy"
+<3> what phase the index is currently in
+<4> what action the index is currently on
+<5> what step the index is currently on
+<6> the definition of the phase
+    (in this case, the "hot" phase) that the index is currently on
+<7> the version of the policy being used to execute the current phase
+
+You can read about the full details of this response in the
+<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
+the response details which phase, action, and step we're in. We are in the
+"hot" phase, and "rollover" action. Rollover will continue to be called
+by {ILM} until its conditions are met and it rolls over the index.
+Afterwards, the original index will stay in the hot phase until 90 more
+days pass and it is deleted in the delete phase.
+As time goes on, new indices will be created and deleted.
+With `datastream-000002` being created when the index mets the rollover
+conditions and `datastream-000003` created after that. We will be able
+to search across all of our managed indices using the "datastream" alias,
+and we will be able to write to our to-be-rolled-over write indices using
+that same alias.
+
+
+
+That's it! We have our first use-case managed by {ILM}.
+
+To learn more about all our APIs,
+check out <<index-lifecycle-management-api,ILM APIs>>.