Browse Source

[DOCS] Edit ILM GS tutorial (#51513)

* [DOCS] Edit ILM GS tutorial

* [DOCS] Incorporated review feedback from @andreidan.

* [DOCS] Removed test link & fixed anchor & title.

* Update docs/reference/ilm/getting-started-ilm.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
debadair 5 years ago
parent
commit
8c5cdfff07
1 changed files with 106 additions and 121 deletions
  1. 106 121
      docs/reference/ilm/getting-started-ilm.asciidoc

+ 106 - 121
docs/reference/ilm/getting-started-ilm.asciidoc

@@ -1,79 +1,93 @@
 [role="xpack"]
 [testenv="basic"]
+
 [[getting-started-index-lifecycle-management]]
-== Get started with {ilm}
+== Get started: Automate rollover with {ilm-init}
+
+++++
+<titleabbrev>Automate rollover</titleabbrev>
+++++
+
+This tutorial demonstrates how to use {ilm} ({ilm-init}) 
+to manage indices that contain time-series data. 
+
+When you continuously index timestamped documents into {es} using 
+Filebeat, Logstash, or some other mechanism, 
+you typically use an index alias so you can periodically roll over to a new index.
+This enables you to implement a hot-warm-cold architecture to meet your performance
+requirements for your newest data, control costs over time, enforce retention policies,
+and still get the most out of your data.
 
-Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario.
-This section will leverage many new concepts unique to {ilm-init} that
-you may not be familiar with. The following sections will explore
-these in more details.
+To automate rollover and management of time-series indices with {ilm-init}, you:
 
-The goal of this example is to set up a set of indices that will encapsulate
-the data from a time series data source. We can imagine there is a system
-like {filebeat-ref}[Filebeat] that continuously indexes documents into
-our writing index. We wish to roll over the index after it reaches a size
-of 50 gigabytes, or has been created 30 days ago, and then delete the index
-after 90 days.
+. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.  
+. <<ilm-gs-apply-policy, Create an index template>> to apply the policy to each new index.
+. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
+. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>> 
+as expected with the {ilm-init} explain API. 
 
 [float]
 [[ilm-gs-create-policy]]
-=== Setting up a policy
+=== Create a lifecycle policy
 
-There are many new features introduced by {ilm-init}, but we will only focus on
-a few that are needed for our example. For starters, we will use the
-<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
-policies are defined in JSON and include specific
-<<ilm-policy-definition,phases and actions>>.
+A lifecycle policy specifies the phases in the index lifecycle
+and the actions to perform in each phase. A lifecycle can have up to four phases:
+`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON 
+and added through the {ilm-init} put policy API. 
+
+For example, the following request creates a `datastream_policy` with two phases:
+ 
+* The `hot` phase defines a `rollover` action to specify that an index rolls over when it 
+reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
+* The `delete` phase uses `min_age` to remove the index 90 days after rollover.
+Note that this value is relative to the rollover time, not the index creation time. 
 
 [source,console]
 ------------------------
-PUT _ilm/policy/datastream_policy   <1>
+PUT _ilm/policy/datastream_policy 
 {
-  "policy": {                       <2>
+  "policy": {                       
     "phases": {
-      "hot": {                      <3>
+      "hot": {                      <1>
         "actions": {
-          "rollover": {             <4>
-            "max_size": "50GB",
+          "rollover": {             
+            "max_size": "50GB",     <2>
             "max_age": "30d"
           }
         }
       },
       "delete": {
-        "min_age": "90d",           <5>
+        "min_age": "90d",           <3>
         "actions": {
-          "delete": {}              <6>
+          "delete": {}              <4>
         }
       }
     }
   }
 }
 ------------------------
+<1> The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately. 
+<2> Trigger the `rollover` action when either of the conditions are met.
+<3> Move the index into the `delete` phase 90 days after rollover.
+<4> Trigger the `delete` action when the index enters the delete phase.
 
-<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
-    a new policy named "datastream_policy"
-<2> policy definition sub-object
-<3> the hot phase defined in the "phases" section. Optional `min_age` field
-    not defined -- defaults to `0ms`
-<4> rollover action definition
-<5> delete phase begins after 90 days
-<6> delete action definition
-
-
-Here we created the policy called `datastream_policy` which rolls over
-the index being written to after it reaches 50 gigabytes, or it is 30
-days old. The rollover will occur when either of these conditions is true.
-The index will be deleted 90 days after it is rolled over.
+See <<_actions>> for the complete list of actions available in each phase.
 
 [float]
 [[ilm-gs-apply-policy]]
-=== Applying a policy to our index
+=== Create an index template to apply the lifecycle policy
+
+To automaticaly apply a lifecycle policy to the new write index on rollover, 
+specify the policy in the index template used to create new indices.
 
-There are <<set-up-lifecycle-policy,a few ways>> to associate a
-policy to an index. Since we wish specific settings to be applied to
-the new index created from Rollover, we will set the policy via
-index templates.
+For example, the following request creates a `datastream_template` that is applied to new indices
+whose names match the `datastream-*` index pattern. 
+The template configures two {ilm-init} settings:
 
+* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match 
+the index pattern.  
+* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over 
+when the rollover action is triggered for an index.
 
 [source,console]
 -----------------------
@@ -90,6 +104,11 @@ PUT _template/datastream_template
 -----------------------
 // TEST[continued]
 
+<1> Apply the template to a new index if its name starts with `datastream-`. 
+<2> The name of the lifecycle policy to apply to each new index.
+<3> The name of the alias used to reference these indices. 
+Required for policies that use the rollover action.
+
 //////////////////////////
 
 [source,console]
@@ -100,23 +119,17 @@ DELETE /_template/datastream_template
 
 //////////////////////////
 
-<1> match all indices starting with "datastream-". These will include all
-    newly created indices from actions like rollover
-<2> the name of the lifecycle policy managing the index
-<3> alias to use for the rollover action, required since a rollover action is
-    defined in the policy.
-
-The above index template introduces a few new settings specific to {ilm-init}.
-The first being `index.lifecycle.name`. This setting will configure
-the "datastream_policy" to the index applying this template. This means
-that all newly created indices prefixed "datastream-" will be managed by
-our policy. The other setting used here is `index.lifecycle.rollover_alias`.
-This setting is required when using a policy containing the rollover
-action and specifies which alias to rollover on behalf of this index.
-The intention here is that the rollover alias is also defined on the index.
+[float]
+[[ilm-gs-bootstrap]]
+=== Bootstrap the initial time-series index
 
-To begin, we will want to bootstrap our first index to write to.
+To get things started, you need to bootstrap an initial index and 
+designate it as the write index for the rollover alias specified in your index template. 
+The name of this index must match the template's index pattern and end with a number. 
+On rollover, this value is incremented to generate a name for the new index. 
 
+For example, the following request creates an index called `datastream-000001` 
+and makes it the write index for the `datastream` alias.
 
 [source,console]
 -----------------------
@@ -131,35 +144,30 @@ PUT datastream-000001
 -----------------------
 // TEST[continued]
 
-When creating our index, we have to consider a few important configurations
-that tie our index and our policy together correctly. We need to make sure
-that our index name matches our index template pattern of "datastream-*",
-which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
-requires that our index name ends with a number. In our case, we used
-`000001`. This is important so that Rollover can increment this number when
-naming the new index created from rolling over.
-
-Our index creation request leverages its template to apply our settings,
-but we must also configure our rollover alias: "datastream". To do this,
-we take advantage of <<aliases-write-index,write indices>>. This is a way
-to define an alias to be used for both reading and writing, with only one
-index being the index that is being written to at a time. Rollover swaps
-the write index to be the new index created from rollover, and sets the
-alias to be read-only for the source index.
+When the rollover conditions are met, the `rollover` action:
+
+* Creates a new index called `datastream-000002`. 
+This matches the `datastream-*` pattern, so the settings from `datastream_template` are applied to the new index.
+* Designates the new index as the write index and makes the bootstrap index read-only.
+
+This process repeats each time rollover conditions are met. 
+You can search across all of the indices managed by the `datastream_policy` with the `datastream` alias. 
+Write operations are routed to the current write index. 
+
+For more information about write indices and rollover, see the <<rollover-index-api-desc, rollover API>>.
 
 [float]
 [[ilm-gs-check-progress]]
 === Checking progress
 
-Now that we have an index managed by our policy, how do we tell what is going
-on? Which phase are we in? Is something broken? This section will go over a
-few APIs and their responses to help us inspect our indices with respect
-to {ilm-init}.
+To get status information for managed indices, you use the {ilm-init} explain API. 
+This lets you find out things like:
 
-With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
-things like which phase we're in and when we entered that phase. The API
-will also provide further info if errors occurred, or if we are blocked on
-certain checks within actions.
+* What phase an index is in and when it entered that phase.
+* The current action and what step is being performed.
+* If any errors have occurred or progress is blocked.
+
+For example, the following request gets information about the `datastream` indices: 
 
 [source,console]
 --------------------------------------------------
@@ -167,8 +175,9 @@ GET datastream-*/_ilm/explain
 --------------------------------------------------
 // TEST[continued]
 
-The above request will retrieve {ilm-init} execution information for all our
-managed indices.
+The response below shows that the bootstrap index is waiting in the `hot` phase's `rollover` action.
+It remains in this state and {ilm-init} continues to call `attempt-rollover` 
+until the rollover conditions are met. 
 
 [[36818c6d9f434d387819c30bd9addb14]]
 [source,console-result]
@@ -177,19 +186,19 @@ managed indices.
   "indices": {
     "datastream-000001": {
       "index": "datastream-000001",
-      "managed": true,                           <1>
-      "policy": "datastream_policy",             <2>
+      "managed": true,                           
+      "policy": "datastream_policy",             <1>
       "lifecycle_date_millis": 1538475653281,
-      "age": "30s",                              <3>
-      "phase": "hot",                            <4>
+      "age": "30s",                              <2>
+      "phase": "hot",                            
       "phase_time_millis": 1538475653317,
-      "action": "rollover",                      <5>
+      "action": "rollover",                      
       "action_time_millis": 1538475653317,
-      "step": "attempt-rollover",                <6>
+      "step": "attempt-rollover",                <3>
       "step_time_millis": 1538475653317,
       "phase_execution": {
         "policy": "datastream_policy",
-        "phase_definition": {                    <7>
+        "phase_definition": {                    <4>
           "min_age": "0ms",
           "actions": {
             "rollover": {
@@ -198,7 +207,7 @@ managed indices.
             }
           }
         },
-        "version": 1,                            <8>
+        "version": 1,                            
         "modified_date_in_millis": 1539609701576
       }
     }
@@ -207,33 +216,9 @@ managed indices.
 --------------------------------------------------
 // TESTRESPONSE[skip:no way to know if we will get this response immediately]
 
-<1> this index is managed by ILM
-<2> the policy in question, in this case, "datastream_policy"
-<3> the current age of the index
-<4> what phase the index is currently in
-<5> what action the index is currently on
-<6> what step the index is currently on
-<7> the definition of the phase
-    (in this case, the "hot" phase) that the index is currently on
-<8> the version of the policy being used to execute the current phase
-
-You can read about the full details of this response in the
-<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
-the response details which phase, action, and step we're in. We are in the
-"hot" phase, and "rollover" action. Rollover will continue to be called
-by {ilm-init} until its conditions are met and it rolls over the index.
-Afterwards, the original index will stay in the hot phase until 90 more
-days pass and it is deleted in the delete phase.
-As time goes on, new indices will be created and deleted.
-With `datastream-000002` being created when the index mets the rollover
-conditions and `datastream-000003` created after that. We will be able
-to search across all of our managed indices using the "datastream" alias,
-and we will be able to write to our to-be-rolled-over write indices using
-that same alias.
-
-
-
-That's it! We have our first use-case managed by {ilm-init}.
-
-To learn more about all our APIs,
-check out <<index-lifecycle-management-api,ILM APIs>>.
+<1> The policy used to manage the index 
+<2> The age of the index
+<3> The step {ilm-init} is performing on the index
+<4> The definition of the current phase (the `hot` phase)
+
+See the <<index-lifecycle-management-api,ILM APIs>> for more information.