elasticsearch/docs/reference/data-streams/lifecycle/tutorial-migrate-data-stream-from-ilm-to-dsl.asciidoc

[role="xpack"]
[[tutorial-migrate-data-stream-from-ilm-to-dsl]]
=== Tutorial: Migrate ILM managed data stream to data stream lifecycle

In this tutorial we'll look at migrating an existing data stream from <<index-lifecycle-management,Index Lifecycle Management ({ilm-init})>> to
<<data-stream-lifecycle,data stream lifecycle>>. The existing {ilm-init} managed backing indices will continue
to be managed by {ilm-init} until they age out and get deleted by {ilm-init}; however,
the new backing indices will be managed by data stream lifecycle.
This way, a data stream is gradually migrated away from being managed by {ilm-init} to
being managed by data stream lifecycle. As we'll see, {ilm-init} and data stream lifecycle
can co-manage a data stream; however, an index can only be managed by one system at
a time.

[discrete]
[[migrate-dsl-ilm-tldr]]
==== TL;DR
To migrate a data stream from {ilm-init} to data stream lifecycle we'll have to execute
two steps:

1. Update the index template that's backing the data stream to set <<index-lifecycle-prefer-ilm, prefer_ilm>>
to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the _existing_ data stream using
the <<data-streams-put-lifecycle, lifecycle API>>.

For more details see the <<migrate-from-ilm-to-dsl, migrate to data stream lifecycle>> section.

[discrete]
[[setup-test-data]]
==== Setup ILM managed data stream
Let's first create a data stream with two backing indices managed by {ilm-init}.
We first create an {ilm-init} policy:

[source,console]
----
PUT _ilm/policy/pre-dsl-ilm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----

And let's create an index template that'll back the data stream and configures {ilm-init}:

[source,console]
----
PUT _index_template/dsl-data-stream-template
{
  "index_patterns": ["dsl-data-stream*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.lifecycle.name": "pre-dsl-ilm-policy"
    }
  }
}
----
// TEST[continued]

We'll now index a document targetting `dsl-data-stream` to create the data stream
and we'll also manually rollover the data stream to have another generation index created:

[source,console]
----
POST dsl-data-stream/_doc?
{
  "@timestamp": "2023-10-18T16:21:15.000Z",
  "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
----
// TEST[continued]

[source,console]
----
POST dsl-data-stream/_rollover
----
// TEST[continued]

We'll use the <<indices-get-data-stream, GET _data_stream>> API to inspect the state of
the data stream:

[source,console]
--------------------------------------------------
GET _data_stream/dsl-data-stream
--------------------------------------------------
// TEST[continued]

Inspecting the response we'll see that both backing indices are managed by {ilm-init}
and that the next generation index will also be managed by {ilm-init}:

[source,console-result]
----
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",    <1>
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,                                       <2>
          "ilm_policy": "pre-dsl-ilm-policy",                       <3>
          "managed_by": "Index Lifecycle Management"                <4>
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "next_generation_managed_by": "Index Lifecycle Management",   <5>
      "prefer_ilm": true,                                           <6>
      "ilm_policy": "pre-dsl-ilm-policy",                           <7>
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
----
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000001"/"index_name": $body.data_streams.0.indices.0.index_name/]
// TESTRESPONSE[s/"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000002"/"index_name": $body.data_streams.0.indices.1.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW","failure_store":{"enabled": false, "indices": [], "rollover_on_write": true}/]

<1> The name of the backing index.
<2> For each backing index we display the value of the <<index-lifecycle-prefer-ilm, prefer_ilm>>
configuration which will indicate if {ilm-init} takes precedence over data stream lifecycle in case
both systems are configured for an index.
<3> The {ilm-init} policy configured for this index.
<4> The system that manages this index (possible values are "Index Lifecycle Management",
"Data stream lifecycle", or "Unmanaged")
<5> The system that will manage the next generation index (the new write index of this
data stream, once the data stream is rolled over). The possible values are
"Index Lifecycle Management", "Data stream lifecycle", or "Unmanaged".
<6> The <<index-lifecycle-prefer-ilm, prefer_ilm>> value configured in the index template
that's backing the data stream. This value will be configured for all the new backing indices.
If it's not configured in the index template the backing indices will receive the `true`
default value ({ilm-init} takes precedence over data stream lifecycle by default as it's
currently richer in features).
<7> The {ilm-init} policy configured in the index template that's backing this data
stream (which will be configured on all the new backing indices, as long as it exists
in the index template).

[discrete]
[[migrate-from-ilm-to-dsl]]
==== Migrate data stream to data stream lifecycle
To migrate the `dsl-data-stream` to data stream lifecycle we'll have to execute
two steps:

1. Update the index template that's backing the data stream to set <<index-lifecycle-prefer-ilm, prefer_ilm>>
to `false`, and to configure data stream lifecycle.
2. Configure the data stream lifecycle for the _existing_ `dsl-data-stream` using
the <<data-streams-put-lifecycle, lifecycle API>>.

IMPORTANT: The data stream lifecycle configuration that's added to the index template,
being a data stream configuration, will only apply to **new** data streams.
Our data stream exists already, so even though we added a data stream lifecycle
configuration in the index template it will not be applied to `dsl-data-stream`.


[[update-index-template-for-dsl]]
Let's update the index template:

[source,console]
----
PUT _index_template/dsl-data-stream-template
{
  "index_patterns": ["dsl-data-stream*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "settings": {
      "index.lifecycle.name": "pre-dsl-ilm-policy",
      "index.lifecycle.prefer_ilm": false             <1>
    },
    "lifecycle": {
      "data_retention": "7d"                          <2>
    }
  }
}
----
// TEST[continued]

<1> The `prefer_ilm` setting will now be configured on the **new** backing indices
(created by rolling over the data stream) such that {ilm-init} does _not_ take
precedence over data stream lifecycle.
<2> We're configuring the data stream lifecycle so _new_ data streams will be
managed by data stream lifecycle.

We've now made sure that new data streams will be managed by data stream lifecycle.

Let's update our existing `dsl-data-stream` and configure data stream lifecycle:

[source,console]
----
PUT _data_stream/dsl-data-stream/_lifecycle
{
    "data_retention": "7d"
}
----
// TEST[continued]

We can inspect the data stream to check that the next generation will indeed be
managed by data stream lifecycle:

[source,console]
--------------------------------------------------
GET _data_stream/dsl-data-stream
--------------------------------------------------
// TEST[continued]

[source,console-result]
----
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                <1>
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                <2>
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "7d",
        "effective_retention": "7d",
        "retention_determined_by": "data_stream_configuration"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Data stream lifecycle",         <3>
      "prefer_ilm": false,                                           <4>
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
----
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000001"/"index_name": $body.data_streams.0.indices.0.index_name/]
// TESTRESPONSE[s/"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000002"/"index_name": $body.data_streams.0.indices.1.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW","failure_store":{"enabled": false, "indices": [], "rollover_on_write": true}/]

<1> The existing backing index will continue to be managed by {ilm-init}
<2> The existing backing index will continue to be managed by {ilm-init}
<3> The next generation index will be managed by Data stream lifecycle
<4> The `prefer_ilm` setting value we configured in the index template is reflected
and will be configured accordingly for new backing indices.

We'll now rollover the data stream to see the new generation index being managed by
data stream lifecycle:

[source,console]
----
POST dsl-data-stream/_rollover
----
// TEST[continued]

[source,console]
----
GET _data_stream/dsl-data-stream
----
// TEST[continued]

[source,console-result]
----
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                <1>
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                <2>
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000003",
          "index_uuid": "PA_JquKGSiKcAKBA8abcd1",
          "prefer_ilm": false,                                      <3>
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Data stream lifecycle"                     <4>
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": true,
        "data_retention": "7d",
        "effective_retention": "7d",
        "retention_determined_by": "data_stream_configuration"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Data stream lifecycle",
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
----
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000001"/"index_name": $body.data_streams.0.indices.0.index_name/]
// TESTRESPONSE[s/"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000002"/"index_name": $body.data_streams.0.indices.1.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000003"/"index_name": $body.data_streams.0.indices.2.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8abcd1"/"index_uuid": $body.data_streams.0.indices.2.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW","failure_store":{"enabled": false, "indices": [], "rollover_on_write": true}/]

<1> The backing indices that existed before rollover will continue to be managed by {ilm-init}
<2> The backing indices that existed before rollover will continue to be managed by {ilm-init}
<3> The new write index received the `false` value for the `prefer_ilm` setting, as we configured
in the index template
<4> The new write index is managed by `Data stream lifecycle`

[discrete]
[[migrate-from-dsl-to-ilm]]
==== Migrate data stream back to ILM
We can easily change this data stream to be managed by {ilm-init} because we didn't remove
the {ilm-init} policy when we <<update-index-template-for-dsl, updated
the index template>>.

We can achieve this in two ways:

1. <<data-streams-delete-lifecycle, Delete the lifecycle>> from the data streams
2. Disable data stream lifecycle by configuring the `enabled` flag to `false`.

Let's implement option 2 and disable the data stream lifecycle:

[source,console]
----
PUT _data_stream/dsl-data-stream/_lifecycle
{
    "data_retention": "7d",
    "enabled": false <1>
}
----
// TEST[continued]
<1> The `enabled` flag can be ommitted and defaults to `true` however, here we
explicitly configure it to `false`
Let's check the state of the data stream:

[source,console]
----
GET _data_stream/dsl-data-stream
----
// TEST[continued]

[source,console-result]
----
{
  "data_streams": [
    {
      "name": "dsl-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000001",
          "index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000002",
          "index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
          "prefer_ilm": true,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-dsl-data-stream-2023.10.19-000003",
          "index_uuid": "PA_JquKGSiKcAKBA8abcd1",
          "prefer_ilm": false,
          "ilm_policy": "pre-dsl-ilm-policy",
          "managed_by": "Index Lifecycle Management"                <1>
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "dsl-data-stream-template",
      "lifecycle": {
        "enabled": false,                                          <2>
        "data_retention": "7d"
      },
      "ilm_policy": "pre-dsl-ilm-policy",
      "next_generation_managed_by": "Index Lifecycle Management",  <3>
      "prefer_ilm": false,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}
----
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000001"/"index_name": $body.data_streams.0.indices.0.index_name/]
// TESTRESPONSE[s/"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000002"/"index_name": $body.data_streams.0.indices.1.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"index_name": ".ds-dsl-data-stream-2023.10.19-000003"/"index_name": $body.data_streams.0.indices.2.index_name/]
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8abcd1"/"index_uuid": $body.data_streams.0.indices.2.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW","failure_store":{"enabled": false, "indices": [], "rollover_on_write": true}/]
<1> The write index is now managed by {ilm-init}
<2> The `lifecycle` configured on the data stream is now disabled.
<3> The next write index will be managed by {ilm-init}

Had we removed the {ilm-init} policy from the index template when we <<update-index-template-for-dsl, updated>>
it, the write index of the data stream will now be `Unmanaged` because the index
wouldn't have the {ilm-init} policy configured to fallback onto.

//////////////////////////
[source,console]
--------------------------------------------------
DELETE _data_stream/dsl-data-stream
DELETE _index_template/dsl-data-stream-template
DELETE _ilm/policy/pre-dsl-ilm-policy
--------------------------------------------------
// TEST[continued]

//////////////////////////