|
@@ -1,79 +1,93 @@
|
|
|
[role="xpack"]
|
|
|
[testenv="basic"]
|
|
|
+
|
|
|
[[getting-started-index-lifecycle-management]]
|
|
|
-== Get started with {ilm}
|
|
|
+== Get started: Automate rollover with {ilm-init}
|
|
|
+
|
|
|
+++++
|
|
|
+<titleabbrev>Automate rollover</titleabbrev>
|
|
|
+++++
|
|
|
+
|
|
|
+This tutorial demonstrates how to use {ilm} ({ilm-init})
|
|
|
+to manage indices that contain time-series data.
|
|
|
+
|
|
|
+When you continuously index timestamped documents into {es} using
|
|
|
+Filebeat, Logstash, or some other mechanism,
|
|
|
+you typically use an index alias so you can periodically roll over to a new index.
|
|
|
+This enables you to implement a hot-warm-cold architecture to meet your performance
|
|
|
+requirements for your newest data, control costs over time, enforce retention policies,
|
|
|
+and still get the most out of your data.
|
|
|
|
|
|
-Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario.
|
|
|
-This section will leverage many new concepts unique to {ilm-init} that
|
|
|
-you may not be familiar with. The following sections will explore
|
|
|
-these in more details.
|
|
|
+To automate rollover and management of time-series indices with {ilm-init}, you:
|
|
|
|
|
|
-The goal of this example is to set up a set of indices that will encapsulate
|
|
|
-the data from a time series data source. We can imagine there is a system
|
|
|
-like {filebeat-ref}[Filebeat] that continuously indexes documents into
|
|
|
-our writing index. We wish to roll over the index after it reaches a size
|
|
|
-of 50 gigabytes, or has been created 30 days ago, and then delete the index
|
|
|
-after 90 days.
|
|
|
+. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.
|
|
|
+. <<ilm-gs-apply-policy, Create an index template>> to apply the policy to each new index.
|
|
|
+. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
|
|
|
+. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
|
|
|
+as expected with the {ilm-init} explain API.
|
|
|
|
|
|
[float]
|
|
|
[[ilm-gs-create-policy]]
|
|
|
-=== Setting up a policy
|
|
|
+=== Create a lifecycle policy
|
|
|
|
|
|
-There are many new features introduced by {ilm-init}, but we will only focus on
|
|
|
-a few that are needed for our example. For starters, we will use the
|
|
|
-<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
|
|
|
-policies are defined in JSON and include specific
|
|
|
-<<ilm-policy-definition,phases and actions>>.
|
|
|
+A lifecycle policy specifies the phases in the index lifecycle
|
|
|
+and the actions to perform in each phase. A lifecycle can have up to four phases:
|
|
|
+`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON
|
|
|
+and added through the {ilm-init} put policy API.
|
|
|
+
|
|
|
+For example, the following request creates a `datastream_policy` with two phases:
|
|
|
+
|
|
|
+* The `hot` phase defines a `rollover` action to specify that an index rolls over when it
|
|
|
+reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
|
|
|
+* The `delete` phase uses `min_age` to remove the index 90 days after rollover.
|
|
|
+Note that this value is relative to the rollover time, not the index creation time.
|
|
|
|
|
|
[source,console]
|
|
|
------------------------
|
|
|
-PUT _ilm/policy/datastream_policy <1>
|
|
|
+PUT _ilm/policy/datastream_policy
|
|
|
{
|
|
|
- "policy": { <2>
|
|
|
+ "policy": {
|
|
|
"phases": {
|
|
|
- "hot": { <3>
|
|
|
+ "hot": { <1>
|
|
|
"actions": {
|
|
|
- "rollover": { <4>
|
|
|
- "max_size": "50GB",
|
|
|
+ "rollover": {
|
|
|
+ "max_size": "50GB", <2>
|
|
|
"max_age": "30d"
|
|
|
}
|
|
|
}
|
|
|
},
|
|
|
"delete": {
|
|
|
- "min_age": "90d", <5>
|
|
|
+ "min_age": "90d", <3>
|
|
|
"actions": {
|
|
|
- "delete": {} <6>
|
|
|
+ "delete": {} <4>
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
------------------------
|
|
|
+<1> The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately.
|
|
|
+<2> Trigger the `rollover` action when either of the conditions are met.
|
|
|
+<3> Move the index into the `delete` phase 90 days after rollover.
|
|
|
+<4> Trigger the `delete` action when the index enters the delete phase.
|
|
|
|
|
|
-<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
|
|
|
- a new policy named "datastream_policy"
|
|
|
-<2> policy definition sub-object
|
|
|
-<3> the hot phase defined in the "phases" section. Optional `min_age` field
|
|
|
- not defined -- defaults to `0ms`
|
|
|
-<4> rollover action definition
|
|
|
-<5> delete phase begins after 90 days
|
|
|
-<6> delete action definition
|
|
|
-
|
|
|
-
|
|
|
-Here we created the policy called `datastream_policy` which rolls over
|
|
|
-the index being written to after it reaches 50 gigabytes, or it is 30
|
|
|
-days old. The rollover will occur when either of these conditions is true.
|
|
|
-The index will be deleted 90 days after it is rolled over.
|
|
|
+See <<_actions>> for the complete list of actions available in each phase.
|
|
|
|
|
|
[float]
|
|
|
[[ilm-gs-apply-policy]]
|
|
|
-=== Applying a policy to our index
|
|
|
+=== Create an index template to apply the lifecycle policy
|
|
|
+
|
|
|
+To automaticaly apply a lifecycle policy to the new write index on rollover,
|
|
|
+specify the policy in the index template used to create new indices.
|
|
|
|
|
|
-There are <<set-up-lifecycle-policy,a few ways>> to associate a
|
|
|
-policy to an index. Since we wish specific settings to be applied to
|
|
|
-the new index created from Rollover, we will set the policy via
|
|
|
-index templates.
|
|
|
+For example, the following request creates a `datastream_template` that is applied to new indices
|
|
|
+whose names match the `datastream-*` index pattern.
|
|
|
+The template configures two {ilm-init} settings:
|
|
|
|
|
|
+* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match
|
|
|
+the index pattern.
|
|
|
+* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
|
|
|
+when the rollover action is triggered for an index.
|
|
|
|
|
|
[source,console]
|
|
|
-----------------------
|
|
@@ -90,6 +104,11 @@ PUT _template/datastream_template
|
|
|
-----------------------
|
|
|
// TEST[continued]
|
|
|
|
|
|
+<1> Apply the template to a new index if its name starts with `datastream-`.
|
|
|
+<2> The name of the lifecycle policy to apply to each new index.
|
|
|
+<3> The name of the alias used to reference these indices.
|
|
|
+Required for policies that use the rollover action.
|
|
|
+
|
|
|
//////////////////////////
|
|
|
|
|
|
[source,console]
|
|
@@ -100,23 +119,17 @@ DELETE /_template/datastream_template
|
|
|
|
|
|
//////////////////////////
|
|
|
|
|
|
-<1> match all indices starting with "datastream-". These will include all
|
|
|
- newly created indices from actions like rollover
|
|
|
-<2> the name of the lifecycle policy managing the index
|
|
|
-<3> alias to use for the rollover action, required since a rollover action is
|
|
|
- defined in the policy.
|
|
|
-
|
|
|
-The above index template introduces a few new settings specific to {ilm-init}.
|
|
|
-The first being `index.lifecycle.name`. This setting will configure
|
|
|
-the "datastream_policy" to the index applying this template. This means
|
|
|
-that all newly created indices prefixed "datastream-" will be managed by
|
|
|
-our policy. The other setting used here is `index.lifecycle.rollover_alias`.
|
|
|
-This setting is required when using a policy containing the rollover
|
|
|
-action and specifies which alias to rollover on behalf of this index.
|
|
|
-The intention here is that the rollover alias is also defined on the index.
|
|
|
+[float]
|
|
|
+[[ilm-gs-bootstrap]]
|
|
|
+=== Bootstrap the initial time-series index
|
|
|
|
|
|
-To begin, we will want to bootstrap our first index to write to.
|
|
|
+To get things started, you need to bootstrap an initial index and
|
|
|
+designate it as the write index for the rollover alias specified in your index template.
|
|
|
+The name of this index must match the template's index pattern and end with a number.
|
|
|
+On rollover, this value is incremented to generate a name for the new index.
|
|
|
|
|
|
+For example, the following request creates an index called `datastream-000001`
|
|
|
+and makes it the write index for the `datastream` alias.
|
|
|
|
|
|
[source,console]
|
|
|
-----------------------
|
|
@@ -131,35 +144,30 @@ PUT datastream-000001
|
|
|
-----------------------
|
|
|
// TEST[continued]
|
|
|
|
|
|
-When creating our index, we have to consider a few important configurations
|
|
|
-that tie our index and our policy together correctly. We need to make sure
|
|
|
-that our index name matches our index template pattern of "datastream-*",
|
|
|
-which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
|
|
|
-requires that our index name ends with a number. In our case, we used
|
|
|
-`000001`. This is important so that Rollover can increment this number when
|
|
|
-naming the new index created from rolling over.
|
|
|
-
|
|
|
-Our index creation request leverages its template to apply our settings,
|
|
|
-but we must also configure our rollover alias: "datastream". To do this,
|
|
|
-we take advantage of <<aliases-write-index,write indices>>. This is a way
|
|
|
-to define an alias to be used for both reading and writing, with only one
|
|
|
-index being the index that is being written to at a time. Rollover swaps
|
|
|
-the write index to be the new index created from rollover, and sets the
|
|
|
-alias to be read-only for the source index.
|
|
|
+When the rollover conditions are met, the `rollover` action:
|
|
|
+
|
|
|
+* Creates a new index called `datastream-000002`.
|
|
|
+This matches the `datastream-*` pattern, so the settings from `datastream_template` are applied to the new index.
|
|
|
+* Designates the new index as the write index and makes the bootstrap index read-only.
|
|
|
+
|
|
|
+This process repeats each time rollover conditions are met.
|
|
|
+You can search across all of the indices managed by the `datastream_policy` with the `datastream` alias.
|
|
|
+Write operations are routed to the current write index.
|
|
|
+
|
|
|
+For more information about write indices and rollover, see the <<rollover-index-api-desc, rollover API>>.
|
|
|
|
|
|
[float]
|
|
|
[[ilm-gs-check-progress]]
|
|
|
=== Checking progress
|
|
|
|
|
|
-Now that we have an index managed by our policy, how do we tell what is going
|
|
|
-on? Which phase are we in? Is something broken? This section will go over a
|
|
|
-few APIs and their responses to help us inspect our indices with respect
|
|
|
-to {ilm-init}.
|
|
|
+To get status information for managed indices, you use the {ilm-init} explain API.
|
|
|
+This lets you find out things like:
|
|
|
|
|
|
-With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
|
|
|
-things like which phase we're in and when we entered that phase. The API
|
|
|
-will also provide further info if errors occurred, or if we are blocked on
|
|
|
-certain checks within actions.
|
|
|
+* What phase an index is in and when it entered that phase.
|
|
|
+* The current action and what step is being performed.
|
|
|
+* If any errors have occurred or progress is blocked.
|
|
|
+
|
|
|
+For example, the following request gets information about the `datastream` indices:
|
|
|
|
|
|
[source,console]
|
|
|
--------------------------------------------------
|
|
@@ -167,8 +175,9 @@ GET datastream-*/_ilm/explain
|
|
|
--------------------------------------------------
|
|
|
// TEST[continued]
|
|
|
|
|
|
-The above request will retrieve {ilm-init} execution information for all our
|
|
|
-managed indices.
|
|
|
+The response below shows that the bootstrap index is waiting in the `hot` phase's `rollover` action.
|
|
|
+It remains in this state and {ilm-init} continues to call `attempt-rollover`
|
|
|
+until the rollover conditions are met.
|
|
|
|
|
|
[[36818c6d9f434d387819c30bd9addb14]]
|
|
|
[source,console-result]
|
|
@@ -177,19 +186,19 @@ managed indices.
|
|
|
"indices": {
|
|
|
"datastream-000001": {
|
|
|
"index": "datastream-000001",
|
|
|
- "managed": true, <1>
|
|
|
- "policy": "datastream_policy", <2>
|
|
|
+ "managed": true,
|
|
|
+ "policy": "datastream_policy", <1>
|
|
|
"lifecycle_date_millis": 1538475653281,
|
|
|
- "age": "30s", <3>
|
|
|
- "phase": "hot", <4>
|
|
|
+ "age": "30s", <2>
|
|
|
+ "phase": "hot",
|
|
|
"phase_time_millis": 1538475653317,
|
|
|
- "action": "rollover", <5>
|
|
|
+ "action": "rollover",
|
|
|
"action_time_millis": 1538475653317,
|
|
|
- "step": "attempt-rollover", <6>
|
|
|
+ "step": "attempt-rollover", <3>
|
|
|
"step_time_millis": 1538475653317,
|
|
|
"phase_execution": {
|
|
|
"policy": "datastream_policy",
|
|
|
- "phase_definition": { <7>
|
|
|
+ "phase_definition": { <4>
|
|
|
"min_age": "0ms",
|
|
|
"actions": {
|
|
|
"rollover": {
|
|
@@ -198,7 +207,7 @@ managed indices.
|
|
|
}
|
|
|
}
|
|
|
},
|
|
|
- "version": 1, <8>
|
|
|
+ "version": 1,
|
|
|
"modified_date_in_millis": 1539609701576
|
|
|
}
|
|
|
}
|
|
@@ -207,33 +216,9 @@ managed indices.
|
|
|
--------------------------------------------------
|
|
|
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
|
|
|
|
|
|
-<1> this index is managed by ILM
|
|
|
-<2> the policy in question, in this case, "datastream_policy"
|
|
|
-<3> the current age of the index
|
|
|
-<4> what phase the index is currently in
|
|
|
-<5> what action the index is currently on
|
|
|
-<6> what step the index is currently on
|
|
|
-<7> the definition of the phase
|
|
|
- (in this case, the "hot" phase) that the index is currently on
|
|
|
-<8> the version of the policy being used to execute the current phase
|
|
|
-
|
|
|
-You can read about the full details of this response in the
|
|
|
-<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
|
|
|
-the response details which phase, action, and step we're in. We are in the
|
|
|
-"hot" phase, and "rollover" action. Rollover will continue to be called
|
|
|
-by {ilm-init} until its conditions are met and it rolls over the index.
|
|
|
-Afterwards, the original index will stay in the hot phase until 90 more
|
|
|
-days pass and it is deleted in the delete phase.
|
|
|
-As time goes on, new indices will be created and deleted.
|
|
|
-With `datastream-000002` being created when the index mets the rollover
|
|
|
-conditions and `datastream-000003` created after that. We will be able
|
|
|
-to search across all of our managed indices using the "datastream" alias,
|
|
|
-and we will be able to write to our to-be-rolled-over write indices using
|
|
|
-that same alias.
|
|
|
-
|
|
|
-
|
|
|
-
|
|
|
-That's it! We have our first use-case managed by {ilm-init}.
|
|
|
-
|
|
|
-To learn more about all our APIs,
|
|
|
-check out <<index-lifecycle-management-api,ILM APIs>>.
|
|
|
+<1> The policy used to manage the index
|
|
|
+<2> The age of the index
|
|
|
+<3> The step {ilm-init} is performing on the index
|
|
|
+<4> The definition of the current phase (the `hot` phase)
|
|
|
+
|
|
|
+See the <<index-lifecycle-management-api,ILM APIs>> for more information.
|