|
@@ -8,11 +8,11 @@
|
|
|
<titleabbrev>Automate rollover</titleabbrev>
|
|
|
++++
|
|
|
|
|
|
-This tutorial demonstrates how to use {ilm} ({ilm-init})
|
|
|
-to manage indices that contain time-series data.
|
|
|
+This tutorial demonstrates how to use {ilm} ({ilm-init})
|
|
|
+to manage indices that contain time-series data.
|
|
|
|
|
|
-When you continuously index timestamped documents into {es} using
|
|
|
-Filebeat, Logstash, or some other mechanism,
|
|
|
+When you continuously index timestamped documents into {es} using
|
|
|
+Filebeat, Logstash, or some other mechanism,
|
|
|
you typically use an index alias so you can periodically roll over to a new index.
|
|
|
This enables you to implement a hot-warm-cold architecture to meet your performance
|
|
|
requirements for your newest data, control costs over time, enforce retention policies,
|
|
@@ -20,11 +20,11 @@ and still get the most out of your data.
|
|
|
|
|
|
To automate rollover and management of time-series indices with {ilm-init}, you:
|
|
|
|
|
|
-. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.
|
|
|
+. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.
|
|
|
. <<ilm-gs-apply-policy, Create an index template>> to apply the policy to each new index.
|
|
|
. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
|
|
|
-. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
|
|
|
-as expected with the {ilm-init} explain API.
|
|
|
+. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
|
|
|
+as expected with the {ilm-init} explain API.
|
|
|
|
|
|
[float]
|
|
|
[[ilm-gs-create-policy]]
|
|
@@ -32,25 +32,25 @@ as expected with the {ilm-init} explain API.
|
|
|
|
|
|
A lifecycle policy specifies the phases in the index lifecycle
|
|
|
and the actions to perform in each phase. A lifecycle can have up to four phases:
|
|
|
-`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON
|
|
|
-and added through the {ilm-init} put policy API.
|
|
|
+`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON
|
|
|
+and added through the {ilm-init} put policy API.
|
|
|
|
|
|
-For example, the following request creates a `datastream_policy` with two phases:
|
|
|
-
|
|
|
-* The `hot` phase defines a `rollover` action to specify that an index rolls over when it
|
|
|
+For example, the following request creates a `timeseries_policy` with two phases:
|
|
|
+
|
|
|
+* The `hot` phase defines a `rollover` action to specify that an index rolls over when it
|
|
|
reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
|
|
|
* The `delete` phase uses `min_age` to remove the index 90 days after rollover.
|
|
|
-Note that this value is relative to the rollover time, not the index creation time.
|
|
|
+Note that this value is relative to the rollover time, not the index creation time.
|
|
|
|
|
|
[source,console]
|
|
|
------------------------
|
|
|
-PUT _ilm/policy/datastream_policy
|
|
|
+PUT _ilm/policy/timeseries_policy
|
|
|
{
|
|
|
- "policy": {
|
|
|
+ "policy": {
|
|
|
"phases": {
|
|
|
"hot": { <1>
|
|
|
"actions": {
|
|
|
- "rollover": {
|
|
|
+ "rollover": {
|
|
|
"max_size": "50GB", <2>
|
|
|
"max_age": "30d"
|
|
|
}
|
|
@@ -66,7 +66,7 @@ PUT _ilm/policy/datastream_policy
|
|
|
}
|
|
|
}
|
|
|
------------------------
|
|
|
-<1> The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately.
|
|
|
+<1> The `min_age` defaults to `0ms`, so new indices enter the `hot` phase immediately.
|
|
|
<2> Trigger the `rollover` action when either of the conditions are met.
|
|
|
<3> Move the index into the `delete` phase 90 days after rollover.
|
|
|
<4> Trigger the `delete` action when the index enters the delete phase.
|
|
@@ -77,43 +77,43 @@ See <<_actions>> for the complete list of actions available in each phase.
|
|
|
[[ilm-gs-apply-policy]]
|
|
|
=== Create an index template to apply the lifecycle policy
|
|
|
|
|
|
-To automaticaly apply a lifecycle policy to the new write index on rollover,
|
|
|
+To automaticaly apply a lifecycle policy to the new write index on rollover,
|
|
|
specify the policy in the index template used to create new indices.
|
|
|
|
|
|
-For example, the following request creates a `datastream_template` that is applied to new indices
|
|
|
-whose names match the `datastream-*` index pattern.
|
|
|
+For example, the following request creates a `timeseries_template` that is applied to new indices
|
|
|
+whose names match the `timeseries-*` index pattern.
|
|
|
The template configures two {ilm-init} settings:
|
|
|
|
|
|
-* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match
|
|
|
-the index pattern.
|
|
|
-* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
|
|
|
+* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match
|
|
|
+the index pattern.
|
|
|
+* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
|
|
|
when the rollover action is triggered for an index.
|
|
|
|
|
|
[source,console]
|
|
|
-----------------------
|
|
|
-PUT _template/datastream_template
|
|
|
+PUT _template/timeseries_template
|
|
|
{
|
|
|
- "index_patterns": ["datastream-*"], <1>
|
|
|
+ "index_patterns": ["timeseries-*"], <1>
|
|
|
"settings": {
|
|
|
"number_of_shards": 1,
|
|
|
"number_of_replicas": 1,
|
|
|
- "index.lifecycle.name": "datastream_policy", <2>
|
|
|
- "index.lifecycle.rollover_alias": "datastream" <3>
|
|
|
+ "index.lifecycle.name": "timeseries_policy", <2>
|
|
|
+ "index.lifecycle.rollover_alias": "timeseries" <3>
|
|
|
}
|
|
|
}
|
|
|
-----------------------
|
|
|
// TEST[continued]
|
|
|
|
|
|
-<1> Apply the template to a new index if its name starts with `datastream-`.
|
|
|
+<1> Apply the template to a new index if its name starts with `timeseries-`.
|
|
|
<2> The name of the lifecycle policy to apply to each new index.
|
|
|
-<3> The name of the alias used to reference these indices.
|
|
|
+<3> The name of the alias used to reference these indices.
|
|
|
Required for policies that use the rollover action.
|
|
|
|
|
|
//////////////////////////
|
|
|
|
|
|
[source,console]
|
|
|
--------------------------------------------------
|
|
|
-DELETE /_template/datastream_template
|
|
|
+DELETE /_template/timeseries_template
|
|
|
--------------------------------------------------
|
|
|
// TEST[continued]
|
|
|
|
|
@@ -123,20 +123,20 @@ DELETE /_template/datastream_template
|
|
|
[[ilm-gs-bootstrap]]
|
|
|
=== Bootstrap the initial time-series index
|
|
|
|
|
|
-To get things started, you need to bootstrap an initial index and
|
|
|
-designate it as the write index for the rollover alias specified in your index template.
|
|
|
-The name of this index must match the template's index pattern and end with a number.
|
|
|
-On rollover, this value is incremented to generate a name for the new index.
|
|
|
+To get things started, you need to bootstrap an initial index and
|
|
|
+designate it as the write index for the rollover alias specified in your index template.
|
|
|
+The name of this index must match the template's index pattern and end with a number.
|
|
|
+On rollover, this value is incremented to generate a name for the new index.
|
|
|
|
|
|
-For example, the following request creates an index called `datastream-000001`
|
|
|
-and makes it the write index for the `datastream` alias.
|
|
|
+For example, the following request creates an index called `timeseries-000001`
|
|
|
+and makes it the write index for the `timeseries` alias.
|
|
|
|
|
|
[source,console]
|
|
|
-----------------------
|
|
|
-PUT datastream-000001
|
|
|
+PUT timeseries-000001
|
|
|
{
|
|
|
"aliases": {
|
|
|
- "datastream": {
|
|
|
+ "timeseries": {
|
|
|
"is_write_index": true
|
|
|
}
|
|
|
}
|
|
@@ -146,13 +146,13 @@ PUT datastream-000001
|
|
|
|
|
|
When the rollover conditions are met, the `rollover` action:
|
|
|
|
|
|
-* Creates a new index called `datastream-000002`.
|
|
|
-This matches the `datastream-*` pattern, so the settings from `datastream_template` are applied to the new index.
|
|
|
+* Creates a new index called `timeseries-000002`.
|
|
|
+This matches the `timeseries-*` pattern, so the settings from `timeseries_template` are applied to the new index.
|
|
|
* Designates the new index as the write index and makes the bootstrap index read-only.
|
|
|
|
|
|
-This process repeats each time rollover conditions are met.
|
|
|
-You can search across all of the indices managed by the `datastream_policy` with the `datastream` alias.
|
|
|
-Write operations are routed to the current write index.
|
|
|
+This process repeats each time rollover conditions are met.
|
|
|
+You can search across all of the indices managed by the `timeseries_policy` with the `timeseries` alias.
|
|
|
+Write operations are routed to the current write index.
|
|
|
|
|
|
For more information about write indices and rollover, see the <<rollover-index-api-desc, rollover API>>.
|
|
|
|
|
@@ -160,44 +160,44 @@ For more information about write indices and rollover, see the <<rollover-index-
|
|
|
[[ilm-gs-check-progress]]
|
|
|
=== Checking progress
|
|
|
|
|
|
-To get status information for managed indices, you use the {ilm-init} explain API.
|
|
|
+To get status information for managed indices, you use the {ilm-init} explain API.
|
|
|
This lets you find out things like:
|
|
|
|
|
|
* What phase an index is in and when it entered that phase.
|
|
|
* The current action and what step is being performed.
|
|
|
* If any errors have occurred or progress is blocked.
|
|
|
|
|
|
-For example, the following request gets information about the `datastream` indices:
|
|
|
+For example, the following request gets information about the `timeseries` indices:
|
|
|
|
|
|
[source,console]
|
|
|
--------------------------------------------------
|
|
|
-GET datastream-*/_ilm/explain
|
|
|
+GET timeseries-*/_ilm/explain
|
|
|
--------------------------------------------------
|
|
|
// TEST[continued]
|
|
|
|
|
|
The response below shows that the bootstrap index is waiting in the `hot` phase's `rollover` action.
|
|
|
-It remains in this state and {ilm-init} continues to call `attempt-rollover`
|
|
|
-until the rollover conditions are met.
|
|
|
+It remains in this state and {ilm-init} continues to call `attempt-rollover`
|
|
|
+until the rollover conditions are met.
|
|
|
|
|
|
[[36818c6d9f434d387819c30bd9addb14]]
|
|
|
[source,console-result]
|
|
|
--------------------------------------------------
|
|
|
{
|
|
|
"indices": {
|
|
|
- "datastream-000001": {
|
|
|
- "index": "datastream-000001",
|
|
|
- "managed": true,
|
|
|
- "policy": "datastream_policy", <1>
|
|
|
+ "timeseries-000001": {
|
|
|
+ "index": "timeseries-000001",
|
|
|
+ "managed": true,
|
|
|
+ "policy": "timeseries_policy", <1>
|
|
|
"lifecycle_date_millis": 1538475653281,
|
|
|
"age": "30s", <2>
|
|
|
- "phase": "hot",
|
|
|
+ "phase": "hot",
|
|
|
"phase_time_millis": 1538475653317,
|
|
|
- "action": "rollover",
|
|
|
+ "action": "rollover",
|
|
|
"action_time_millis": 1538475653317,
|
|
|
"step": "attempt-rollover", <3>
|
|
|
"step_time_millis": 1538475653317,
|
|
|
"phase_execution": {
|
|
|
- "policy": "datastream_policy",
|
|
|
+ "policy": "timeseries_policy",
|
|
|
"phase_definition": { <4>
|
|
|
"min_age": "0ms",
|
|
|
"actions": {
|
|
@@ -207,7 +207,7 @@ until the rollover conditions are met.
|
|
|
}
|
|
|
}
|
|
|
},
|
|
|
- "version": 1,
|
|
|
+ "version": 1,
|
|
|
"modified_date_in_millis": 1539609701576
|
|
|
}
|
|
|
}
|
|
@@ -216,7 +216,7 @@ until the rollover conditions are met.
|
|
|
--------------------------------------------------
|
|
|
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
|
|
|
|
|
|
-<1> The policy used to manage the index
|
|
|
+<1> The policy used to manage the index
|
|
|
<2> The age of the index
|
|
|
<3> The step {ilm-init} is performing on the index
|
|
|
<4> The definition of the current phase (the `hot` phase)
|