|
@@ -1,20 +1,36 @@
|
|
|
[role="xpack"]
|
|
|
[[tutorial-migrate-data-stream-from-ilm-to-dsl]]
|
|
|
-=== Tutorial: Migrate ILM managed data stream to Data stream lifecycle
|
|
|
+=== Tutorial: Migrate ILM managed data stream to data stream lifecycle
|
|
|
|
|
|
preview::[]
|
|
|
|
|
|
-In this tutorial we'll look at migrating an existing data stream from {ilm-init} to
|
|
|
-Data stream lifecycle. The existing {ilm-init} managed backing indices will continue
|
|
|
+In this tutorial we'll look at migrating an existing data stream from Index Lifecycle Management ({ilm-init}) to
|
|
|
+data stream lifecycle. The existing {ilm-init} managed backing indices will continue
|
|
|
to be managed by {ilm-init} until they age out and get deleted by {ilm-init}; however,
|
|
|
-the new backing indices will be managed by Data stream lifecycle.
|
|
|
-This way, a data stream is gradually migrated away from being managed by {ilm-cap} to
|
|
|
-being managed by Data stream lifecycle. As we'll see, {ilm-cap} and Data stream lifecycle
|
|
|
-can co-manage a data stream; however, an index can only be managed by one system at
|
|
|
+the new backing indices will be managed by data stream lifecycle.
|
|
|
+This way, a data stream is gradually migrated away from being managed by {ilm-init} to
|
|
|
+being managed by data stream lifecycle. As we'll see, {ilm-init} and data stream lifecycle
|
|
|
+can co-manage a data stream; however, an index can only be managed by one system at
|
|
|
a time.
|
|
|
|
|
|
-Let's first create a data stream with two backing indices managed by {ilm-cap}.
|
|
|
-We first create an {ilm-cap} policy:
|
|
|
+[discrete]
|
|
|
+[[migrate-dsl-ilm-tldr]]
|
|
|
+==== TL;DR
|
|
|
+To migrate a data stream from {ilm-init} to data stream lifecycle we'll have to execute
|
|
|
+two steps:
|
|
|
+
|
|
|
+1. Update the index template that's backing the data stream to set <<index-lifecycle-prefer-ilm, prefer_ilm>>
|
|
|
+to `false`, and to configure data stream lifecycle.
|
|
|
+2. Configure the data stream lifecycle for the _existing_ data stream using
|
|
|
+the <<data-streams-put-lifecycle, lifecycle API>>.
|
|
|
+
|
|
|
+For more details see the <<migrate-from-ilm-to-dsl, migrate to data stream lifecycle>> section.
|
|
|
+
|
|
|
+[discrete]
|
|
|
+[[setup-test-data]]
|
|
|
+==== Setup ILM managed data stream
|
|
|
+Let's first create a data stream with two backing indices managed by {ilm-init}.
|
|
|
+We first create an {ilm-init} policy:
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
@@ -40,7 +56,7 @@ PUT _ilm/policy/pre-dsl-ilm-policy
|
|
|
}
|
|
|
----
|
|
|
|
|
|
-And let's create an index template that'll back the data stream and configures {ilm-cap}:
|
|
|
+And let's create an index template that'll back the data stream and configures {ilm-init}:
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
@@ -153,11 +169,14 @@ currently richer in features).
|
|
|
stream (which will be configured on all the new backing indices, as long as it exists
|
|
|
in the index template).
|
|
|
|
|
|
+[discrete]
|
|
|
+[[migrate-from-ilm-to-dsl]]
|
|
|
+==== Migrate data stream to data stream lifecycle
|
|
|
To migrate the `dsl-data-stream` to data stream lifecycle we'll have to execute
|
|
|
two steps:
|
|
|
|
|
|
-1. Update the index template that's backing the index template to configure <<index-lifecycle-prefer-ilm, prefer_ilm>>
|
|
|
-to `false`, and to configure data stream lifecycle.
|
|
|
+1. Update the index template that's backing the data stream to set <<index-lifecycle-prefer-ilm, prefer_ilm>>
|
|
|
+to `false`, and to configure data stream lifecycle.
|
|
|
2. Configure the data stream lifecycle for the _existing_ `dsl-data-stream` using
|
|
|
the <<data-streams-put-lifecycle, lifecycle API>>.
|
|
|
|
|
@@ -191,14 +210,14 @@ PUT _index_template/dsl-data-stream-template
|
|
|
// TEST[continued]
|
|
|
|
|
|
<1> The `prefer_ilm` setting will now be configured on the **new** backing indices
|
|
|
-(created by rolling over the data stream) such that {ilm-init} does _not_ take
|
|
|
-precedence over Data stream lifecycle.
|
|
|
-<2> We're configuring the data stream lifecycle so _new_ data streams will be
|
|
|
-managed by Data stream lifecycle.
|
|
|
+(created by rolling over the data stream) such that {ilm-init} does _not_ take
|
|
|
+precedence over data stream lifecycle.
|
|
|
+<2> We're configuring the data stream lifecycle so _new_ data streams will be
|
|
|
+managed by data stream lifecycle.
|
|
|
|
|
|
-We've now made sure that new data streams will be managed by Data stream lifecycle.
|
|
|
+We've now made sure that new data streams will be managed by data stream lifecycle.
|
|
|
|
|
|
-Let's update our existing `dsl-data-stream` and configure Data stream lifecycle:
|
|
|
+Let's update our existing `dsl-data-stream` and configure data stream lifecycle:
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
@@ -209,8 +228,8 @@ PUT _data_stream/dsl-data-stream/_lifecycle
|
|
|
----
|
|
|
// TEST[continued]
|
|
|
|
|
|
-We can inspect the data stream to check that the next generation will indeed be
|
|
|
-managed by Data stream lifecycle:
|
|
|
+We can inspect the data stream to check that the next generation will indeed be
|
|
|
+managed by data stream lifecycle:
|
|
|
|
|
|
[source,console]
|
|
|
--------------------------------------------------
|
|
@@ -273,8 +292,8 @@ GET _data_stream/dsl-data-stream
|
|
|
<4> The `prefer_ilm` setting value we configured in the index template is reflected
|
|
|
and will be configured accordingly for new backing indices.
|
|
|
|
|
|
-We'll now rollover the data stream to see the new generation index being managed by
|
|
|
-Data stream lifecycle:
|
|
|
+We'll now rollover the data stream to see the new generation index being managed by
|
|
|
+data stream lifecycle:
|
|
|
|
|
|
[source,console]
|
|
|
----
|
|
@@ -352,14 +371,17 @@ GET _data_stream/dsl-data-stream
|
|
|
in the index template
|
|
|
<4> The new write index is managed by `Data stream lifecycle`
|
|
|
|
|
|
-We can easily change this data stream to be managed by {ilm-cap} because we didn't remove
|
|
|
-the {ilm-cap} policy when we <<update-index-template-for-dsl, updated
|
|
|
-the index template>>.
|
|
|
+[discrete]
|
|
|
+[[migrate-from-dsl-to-ilm]]
|
|
|
+==== Migrate data stream back to ILM
|
|
|
+We can easily change this data stream to be managed by {ilm-init} because we didn't remove
|
|
|
+the {ilm-init} policy when we <<update-index-template-for-dsl, updated
|
|
|
+the index template>>.
|
|
|
|
|
|
We can achieve this in two ways:
|
|
|
|
|
|
1. <<data-streams-delete-lifecycle, Delete the lifecycle>> from the data streams
|
|
|
-2. Disable Data stream lifecycle by configuring the `enabled` flag to `false`.
|
|
|
+2. Disable data stream lifecycle by configuring the `enabled` flag to `false`.
|
|
|
|
|
|
Let's implement option 2 and disable the data stream lifecycle:
|
|
|
|
|
@@ -439,13 +461,13 @@ GET _data_stream/dsl-data-stream
|
|
|
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000003"/"index_name": $body.data_streams.0.indices.2.index_name/]
|
|
|
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8abcd1"/"index_uuid": $body.data_streams.0.indices.2.index_uuid/]
|
|
|
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW","failure_indices":[],"failure_store":false/]
|
|
|
-<1> The write index is now managed by {ilm-cap}
|
|
|
+<1> The write index is now managed by {ilm-init}
|
|
|
<2> The `lifecycle` configured on the data stream is now disabled.
|
|
|
-<3> The next write index will be managed by {ilm-cap}
|
|
|
+<3> The next write index will be managed by {ilm-init}
|
|
|
|
|
|
-Had we removed the {ilm-cap} policy from the index template when we <<update-index-template-for-dsl, updated>>
|
|
|
+Had we removed the {ilm-init} policy from the index template when we <<update-index-template-for-dsl, updated>>
|
|
|
it, the write index of the data stream will now be `Unmanaged` because the index
|
|
|
-wouldn't have the {ilm-cap} policy configured to fallback onto.
|
|
|
+wouldn't have the {ilm-init} policy configured to fallback onto.
|
|
|
|
|
|
//////////////////////////
|
|
|
[source,console]
|