Start with data stream lifecycle documentation (#95326)

docs/reference/data-management.asciidoc

@@ -20,17 +20,32 @@ so you can move it to less expensive, less performant hardware.
 For your oldest data, what matters is that you have access to the data. 
 It's ok if queries take longer to complete.
 
-To help you manage your data, {es} enables you to:
+To help you manage your data, {es} offers you:
 
+* <<index-lifecycle-management, {ilm-cap}>> ({ilm-init}) to manage both indices and data streams and it is fully customisable, and
+* <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
+common lifecycle management needs.
+
+preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+
+**{ilm-init}** can be used to manage both indices and data streams and it allows you to:
+
+* Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
+Data older than this period can be deleted by {es}.
 * Define <<data-tiers, multiple tiers>> of data nodes with different performance characteristics.
-* Automatically transition indices through the data tiers according to your performance needs and retention policies
-with <<index-lifecycle-management, {ilm}>> ({ilm-init}). 
+* Automatically transition indices through the data tiers according to your performance needs and retention policies.
 * Leverage <<searchable-snapshots, searchable snapshots>> stored in a remote repository to provide resiliency 
 for your older indices while reducing operating costs and maintaining search performance.  
 * Perform <<async-search-intro, asynchronous searches>> of data stored on less-performant hardware.
+
+**Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily:
+
+* Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
+Data older than this period can be deleted by {es} at a later time.
+* Improve the performance of your data stream by performing background operations that will optimise the way your data
+stream is stored.
 --
 
 include::ilm/index.asciidoc[]
 
 include::datatiers.asciidoc[]
-

docs/reference/data-streams/data-stream-apis.asciidoc

@@ -12,6 +12,14 @@ The following APIs are available for managing <<data-streams,data streams>>:
 * <<promote-data-stream-api>>
 * <<modify-data-streams-api>>
 
+[[data-stream-lifecycle-api]]
+The following APIs are available for managing the built-in lifecycle of data streams:
+
+* <<data-streams-put-lifecycle,Update data stream lifecycle>> preview:[]
+* <<data-streams-get-lifecycle,Get data stream lifecycle>> preview:[]
+* <<data-streams-delete-lifecycle,Delete data stream lifecycle>> preview:[]
+* <<data-streams-explain-lifecycle,Explain data stream lifecycle>> preview:[]
+
 The following API is available for <<tsds,time series data streams>>:
 
 * <<indices-downsample-data-stream>>
@@ -33,4 +41,12 @@ include::{es-repo-dir}/data-streams/promote-data-stream-api.asciidoc[]
 
 include::{es-repo-dir}/data-streams/modify-data-streams-api.asciidoc[]
 
+include::{es-repo-dir}/data-streams/lifecycle/apis/put-lifecycle.asciidoc[]
+
+include::{es-repo-dir}/data-streams/lifecycle/apis/get-lifecycle.asciidoc[]
+
+include::{es-repo-dir}/data-streams/lifecycle/apis/delete-lifecycle.asciidoc[]
+
+include::{es-repo-dir}/data-streams/lifecycle/apis/explain-lifecycle.asciidoc[]
+
 include::{es-repo-dir}/indices/downsample-data-stream.asciidoc[]

docs/reference/data-streams/data-streams.asciidoc

@@ -135,3 +135,4 @@ include::set-up-a-data-stream.asciidoc[]
 include::use-a-data-stream.asciidoc[]
 include::change-mappings-and-settings.asciidoc[]
 include::tsds.asciidoc[]
+include::lifecycle/index.asciidoc[]

docs/reference/dlm/apis/delete-lifecycle.asciidoc → docs/reference/data-streams/lifecycle/apis/delete-lifecycle.asciidoc

@@ -1,10 +1,10 @@
-[[dlm-delete-lifecycle]]
+[[data-streams-delete-lifecycle]]
 === Delete the lifecycle of a data stream
 ++++
 <titleabbrev>Delete Data Stream Lifecycle</titleabbrev>
 ++++
 
-experimental::[]
+preview::[]
 
 Deletes the lifecycle from a set of data streams.
 
@@ -14,18 +14,18 @@ Deletes the lifecycle from a set of data streams.
 * If the {es} {security-features} are enabled, you must have the `manage_data_stream_lifecycle` index privilege or higher to
 use this API. For more information, see <<security-privileges>>.
 
-[[dlm-delete-lifecycle-request]]
+[[data-streams-delete-lifecycle-request]]
 ==== {api-request-title}
 
 `DELETE _data_stream/<data-stream>/_lifecycle`
 
-[[dlm-delete-lifecycle-desc]]
+[[data-streams-delete-lifecycle-desc]]
 ==== {api-description-title}
 
 Deletes the lifecycle from the specified data streams. If multiple data streams are provided but at least one of them
 does not exist, then the deletion of the lifecycle will fail for all of them and the API will respond with `404`.
 
-[[dlm-delete-lifecycle-path-params]]
+[[data-streams-delete-lifecycle-path-params]]
 ==== {api-path-parms-title}
 
 `<data-stream>`::
@@ -41,7 +41,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
 +
 Defaults to `open`.
 
-[[dlm-delete-lifecycle-example]]
+[[data-streams-delete-lifecycle-example]]
 ==== {api-examples-title}
 
 ////

docs/reference/dlm/apis/explain-data-lifecycle.asciidoc → docs/reference/data-streams/lifecycle/apis/explain-lifecycle.asciidoc

@@ -1,41 +1,39 @@
-[[dlm-explain-lifecycle]]
-=== Explain Lifecycle API
+[[data-streams-explain-lifecycle]]
+=== Explain data stream lifecycle
 ++++
-<titleabbrev>Explain Data Lifecycle</titleabbrev>
+<titleabbrev>Explain Data Stream Lifecycle</titleabbrev>
 ++++
 
-experimental::[]
+preview::[]
 
 Retrieves the current data lifecycle status for one or more data stream backing indices.
 
 [[explain-lifecycle-api-prereqs]]
 ==== {api-prereq-title}
 
-* Nit: would rephrase as:
-
 If the {es} {security-features} are enabled, you must have at least the `manage_data_stream_lifecycle` index privilege or
 `view_index_metadata` index privilege to use this API. For more information, see <<security-privileges>>.
 
-[[dlm-explain-lifecycle-request]]
+[[data-streams-explain-lifecycle-request]]
 ==== {api-request-title}
 
 `GET <target>/_lifecycle/explain`
 
-[[dlm-explain-lifecycle-desc]]
+[[data-streams-explain-lifecycle-desc]]
 ==== {api-description-title}
 
-Retrieves information about the index's current DLM lifecycle state, such as
+Retrieves information about the index or data stream's current data stream lifecycle state, such as
 time since index creation, time since rollover, the lifecycle configuration
 managing the index, or any error that {es} might've encountered during the lifecycle
 execution.
 
-[[dlm-explain-lifecycle-path-params]]
+[[data-streams-explain-lifecycle-path-params]]
 ==== {api-path-parms-title}
 
 `<target>`::
 (Required, string) Comma-separated list of indices.
 
-[[dlm-explain-lifecycle-query-params]]
+[[data-streams-explain-lifecycle-query-params]]
 ==== {api-query-parms-title}
 
 `include_defaults`::
@@ -44,7 +42,7 @@ execution.
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 
-[[dlm-explain-lifecycle-example]]
+[[data-streams-explain-lifecycle-example]]
 ==== {api-examples-title}
 
 The following example retrieves the lifecycle state of the index `.ds-metrics-2023.03.22-000001`:
@@ -53,9 +51,9 @@ The following example retrieves the lifecycle state of the index `.ds-metrics-20
 --------------------------------------------------
 GET .ds-metrics-2023.03.22-000001/_lifecycle/explain
 --------------------------------------------------
-// TEST[skip:we're not setting up DLM in these tests]
+// TEST[skip:we're not setting up data stream lifecycle in these tests]
 
-If the index is managed by DLM `explain` will show the `managed_by_lifecycle` field
+If the index is managed by a data stream lifecycle `explain` will show the `managed_by_lifecycle` field
 set to `true` and the rest of the response will contain information about the
 lifecycle execution status for this index:
 
@@ -77,8 +75,8 @@ lifecycle execution status for this index:
 --------------------------------------------------
 // TESTRESPONSE[skip:the result is for illustrating purposes only]
 
-<1> Shows if the index is being managed by DLM. If the index is not managed by
-DLM the other fields will not be shown
+<1> Shows if the index is being managed by data stream lifecycle. If the index is not managed by
+a data stream lifecycle the other fields will not be shown
 <2> When the index was created, this timestamp is used to determine when to
 rollover
 <3> The time since the index creation (used for calculating when to rollover

docs/reference/dlm/apis/get-lifecycle.asciidoc → docs/reference/data-streams/lifecycle/apis/get-lifecycle.asciidoc

@@ -1,10 +1,10 @@
-[[dlm-get-lifecycle]]
+[[data-streams-get-lifecycle]]
 === Get the lifecycle of a data stream
 ++++
 <titleabbrev>Get Data Stream Lifecycle</titleabbrev>
 ++++
 
-experimental::[]
+preview::[]
 
 Gets the lifecycle of a set of data streams.
 
@@ -15,12 +15,12 @@ Gets the lifecycle of a set of data streams.
 <<privileges-list-indices,index privilege>>, the `manage_data_stream_lifecycle` index privilege, or the
 `view_index_metadata` privilege to use this API. For more information, see <<security-privileges>>.
 
-[[dlm-get-lifecycle-request]]
+[[data-streams-get-lifecycle-request]]
 ==== {api-request-title}
 
 `GET _data_stream/<data-stream>/_lifecycle`
 
-[[dlm-get-lifecycle-desc]]
+[[data-streams-get-lifecycle-desc]]
 ==== {api-description-title}
 
 Gets the lifecycle of the specified data streams. If multiple data streams are requested but at least one of them
@@ -28,7 +28,7 @@ does not exist, then the API will respond with `404` since at least one of the r
 If the requested data streams do not have a lifecycle configured they will still be included in the API response but the
 `lifecycle` key will be missing.
 
-[[dlm-get-lifecycle-path-params]]
+[[data-streams-get-lifecycle-path-params]]
 ==== {api-path-parms-title}
 
 `<data-stream>`::
@@ -75,12 +75,12 @@ duration the document could be deleted. When undefined, every document in this d
 `rollover`::
 (Optional, object)
 The conditions which will trigger the rollover of a backing index as configured by the cluster setting
-`cluster.lifecycle.default.rollover`. This property is an implementation detail and it will only be retrieved when the query
-param `include_defaults` is set to `true`. The contents of this field are subject to change.
+`cluster.lifecycle.default.rollover`. This property is an implementation detail and it will only be retrieved
+when the query param `include_defaults` is set to `true`. The contents of this field are subject to change.
 =====
 ====
 
-[[dlm-get-lifecycle-example]]
+[[data-streams-get-lifecycle-example]]
 ==== {api-examples-title}
 
 ////

docs/reference/dlm/apis/put-lifecycle.asciidoc → docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc

@@ -1,10 +1,10 @@
-[[dlm-put-lifecycle]]
+[[data-streams-put-lifecycle]]
 === Set the lifecycle of a data stream
 ++++
 <titleabbrev>Put Data Stream Lifecycle</titleabbrev>
 ++++
 
-experimental::[]
+preview::[]
 
 Configures the data lifecycle for the targeted data streams.
 
@@ -14,18 +14,18 @@ Configures the data lifecycle for the targeted data streams.
 If the {es} {security-features} are enabled, you must have the `manage_data_stream_lifecycle` index privilege or higher to use this API.
 For more information, see <<security-privileges>>.
 
-[[dlm-put-lifecycle-request]]
+[[data-streams-put-lifecycle-request]]
 ==== {api-request-title}
 
 `PUT _data_stream/<data-stream>/_lifecycle`
 
-[[dlm-put-lifecycle-desc]]
+[[data-streams-put-lifecycle-desc]]
 ==== {api-description-title}
 
 Configures the data lifecycle for the targeted data streams. If multiple data streams are provided but at least one of them
 does not exist, then the update of the lifecycle will fail for all of them and the API will respond with `404`.
 
-[[dlm-put-lifecycle-path-params]]
+[[data-streams-put-lifecycle-path-params]]
 ==== {api-path-parms-title}
 
 `<data-stream>`::
@@ -55,7 +55,7 @@ If defined, every document added to this data stream will be stored at least for
 duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.
 ====
 
-[[dlm-put-lifecycle-example]]
+[[data-streams-put-lifecycle-example]]
 ==== {api-examples-title}
 
 The following example sets the lifecycle of `my-data-stream`:

docs/reference/data-streams/lifecycle/index.asciidoc

@@ -0,0 +1,64 @@
+[role="xpack"]
+[[data-stream-lifecycle]]
+== Data stream lifecycle
+
+preview::[]
+
+A data stream lifecycle is the built-in mechanism data streams use to manage their lifecycle. It enables you to easily
+automate the management of your data streams according to your retention requirements. For example, you could configure
+the lifecycle to:
+
+* Ensure that data indexed in the data stream will be kept at least for the retention time you defined.
+* Ensure that data older than the retention period will be deleted automatically by {es} at a later time.
+
+To achieve that, it supports:
+
+* Automatic <<index-rollover,rollover>>, which chunks your incoming data in smaller pieces to facilitate better performance
+and backwards incompatible mapping changes.
+* Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored.
+{es} is allowed at a later time to delete data older than this time period.
+
+[discrete]
+[[data-streams-lifecycle-how-it-works]]
+=== How does it work?
+
+In intervals configured by <<data-streams-lifecycle-poll-interval,`data_streams.lifecycle.poll_interval`>>, {es} goes over
+each data stream and performs the following steps:
+
+1. Checks if the data stream has a data lifecycle configured, skipping any indices not part of a managed data stream.
+2. Rolls over the write index of the data stream, if it fulfills the conditions defined by
+<<cluster-lifecycle-default-rollover,`cluster.lifecycle.default.rollover`>>.
+3. Applies retention to the remaining backing indices. This means deleting the backing indices whose
+`generation_time` is longer than the configured retention period. The `generation_time` is only applicable to rolled over backing
+indices and it is either the time since the backing index got rolled over, or the time optionally configured in the
+<<index-data-stream-lifecycle-origination-date,`index.lifecycle.origination_date`>> setting.
+
+IMPORTANT: We use the `generation_time` instead of the creation time because this ensures that all data in the backing
+index have passed the retention period. As a result, the retention period is not the exact time data gets deleted, but
+the minimum time data will be stored.
+
+NOTE: The steps `2` and `3` apply only to backing indices that are not already managed by {ilm-init}, meaning that these indices either do
+not have an {ilm-init} policy defined, or if they do, they have <<index-lifecycle-prefer-ilm,`index.lifecycle.prefer_ilm`>>
+set to `false`.
+
+[discrete]
+[[data-stream-lifecycle-configuration]]
+=== Configuring data stream lifecycle
+
+Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and
+on an existing one differ.
+
+In the following sections, we will go through the following tutorials:
+
+* To create a new data stream with a lifecycle, you need to add the data lifecycle as part of the index template
+that matches the name of your data stream (see <<tutorial-manage-new-data-stream>>). When a write operation
+with the name of your data stream reaches {es} then the data stream will be created with the respective data lifecycle.
+* To update the lifecycle of an existing data stream you need to use the <<data-stream-lifecycle-api, data stream lifecycle APIs>>
+to edit the lifecycle on the data stream itself (see <<tutorial-manage-existing-data-stream>>).
+
+NOTE: Updating the data lifecycle of an existing data stream is different from updating the settings or the mapping,
+because it is applied on the data stream level and not on the individual backing indices.
+
+include::tutorial-manage-new-data-stream.asciidoc[]
+
+include::tutorial-manage-existing-data-stream.asciidoc[]

docs/reference/data-streams/lifecycle/tutorial-manage-existing-data-stream.asciidoc

@@ -0,0 +1,136 @@
+[role="xpack"]
+[[tutorial-manage-existing-data-stream]]
+=== Tutorial: Update existing data stream
+
+preview::[]
+
+To update the lifecycle of an existing data stream you do the following actions:
+
+. <<set-lifecycle>>
+. <<delete-lifecycle>>
+
+[discrete]
+[[set-lifecycle]]
+==== Set a data stream's lifecycle
+
+To add or to change the retention period of your data stream you can use the <<data-streams-put-lifecycle, PUT lifecycle API>>.
+
+* You can set infinite retention period, meaning that your data should never be deleted. For example:
++
+[source,console]
+----
+PUT _data_stream/my-data-stream/_lifecycle
+{ } <1>
+----
+// TEST[setup:my_data_stream]
+<1> An empty payload means that your data stream is still managed but the data will never be deleted. Managing a time
+series data stream such as logs or metrics enables {es} to better store your data even if you do not use a retention period.
+
+* Or you can set the retention period of your choice. For example:
++
+[source,console]
+----
+PUT _data_stream/my-data-stream/_lifecycle
+{
+  "data_retention": "30d" <1>
+}
+----
+// TEST[continued]
+<1> The retention period of this data stream is set to 30 days. This means that {es} is allowed to delete data that is
+older than 30 days at its own discretion.
+
+The changes in the lifecycle are applied on all backing indices of the data stream. You can see the effect of the change
+via the <<data-streams-explain-lifecycle, explain API>>:
+
+[source,console]
+--------------------------------------------------
+GET .ds-my-data-stream-*/_lifecycle/explain
+--------------------------------------------------
+// TEST[continued]
+
+The response will look like:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "indices": {
+    ".ds-my-data-stream-2023.04.19-000002": {
+      "index": ".ds-my-data-stream-2023.04.19-000002",  <1>
+      "managed_by_lifecycle": true,                           <2>
+      "index_creation_date_millis": 1681919221417,
+      "time_since_index_creation": "6.85s",             <3>
+      "lifecycle": {
+        "data_retention": "30d"                         <4>
+      }
+    },
+    ".ds-my-data-stream-2023.04.17-000001": {
+      "index": ".ds-my-data-stream-2023.04.17-000001",  <5>
+      "managed_by_lifecycle": true,                           <6>
+      "index_creation_date_millis": 1681745209501,
+      "time_since_index_creation": "48d",               <7>
+      "rollover_date_millis": 1681919221419,
+      "time_since_rollover": "6.84s",                   <8>
+      "generation_time": "6.84s",                       <9>
+      "lifecycle": {
+        "data_retention": "30d"                         <10>
+      }
+    }
+  }
+}
+--------------------------------------------------
+// TEST[continued]
+// TESTRESPONSE[skip:the result is for illustrating purposes only]
+<1> The name of the backing index.
+<2> This index is managed by a data stream lifecycle.
+<3> The time that has passed since this index has been created.
+<4> The data retention for this index is at least 30 days, as it was recently updated.
+<5> The name of the backing index.
+<6> This index is managed by the built-in data stream lifecycle.
+<7> The time that has passed since this index has been created.
+<8> The time that has passed since this index was <<index-rollover,rolled over>>.
+<9> The time that will be used to determine when it's safe to delete this index and all its data.
+<10> The data retention for this index as well is at least 30 days, as it was recently updated.
+
+[discrete]
+[[delete-lifecycle]]
+==== Remove lifecycle for a data stream
+
+To remove the lifecycle of a data stream you can use the <<data-streams-delete-lifecycle-request,delete lifecycle API>>. As consequence,
+the maintenance operations that were applied by the lifecycle will no longer be applied to the data stream and all its
+backing indices. For example:
+
+[source,console]
+--------------------------------------------------
+DELETE _data_stream/my-data-stream/_lifecycle
+--------------------------------------------------
+// TEST[continued]
+
+You can then use the <<data-streams-explain-lifecycle, explain API>> again to see that the indices are no longer managed.
+
+[source,console]
+--------------------------------------------------
+GET .ds-my-data-stream-*/_lifecycle/explain
+--------------------------------------------------
+// TEST[continued]
+// TEST[teardown:data_stream_cleanup]
+
+[source,console-result]
+--------------------------------------------------
+{
+  "indices": {
+    ".ds-my-data-stream-2023.04.19-000002": {
+      "index": ".ds-my-data-stream-2023.04.19-000002",  <1>
+      "managed_by_lifecycle": false                           <2>
+    },
+    ".ds-my-data-stream-2023.04.17-000001": {
+      "index": ".ds-my-data-stream-2023.04.19-000001",  <3>
+      "managed_by_lifecycle": false                           <4>
+    }
+  }
+}
+--------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only]
+<1> The name of the backing index.
+<2> Indication that the index is not managed by the data stream lifecycle.
+<3> The name of another backing index.
+<4> Indication that the index is not managed by the data stream lifecycle.

docs/reference/data-streams/lifecycle/tutorial-manage-new-data-stream.asciidoc

@@ -0,0 +1,148 @@
+[role="xpack"]
+[[tutorial-manage-new-data-stream]]
+=== Tutorial: Create a data stream with a lifecycle
+
+preview::[]
+
+To create a data stream with a built-in lifecycle, follow these steps:
+
+. <<create-index-template-with-lifecycle>>
+. <<create-data-stream-with-lifecycle>>
+. <<retrieve-lifecycle-information>>
+
+[discrete]
+[[create-index-template-with-lifecycle]]
+==== Create an index template
+
+A data stream requires a matching <<index-templates,index template>>. You can configure the data stream lifecycle by
+setting the `lifecycle` field in the index template the same as you do for mappings and index settings. You can define an
+index template that sets a lifecycle as follows:
+
+* Include the `data_stream` object to enable data streams.
+
+* Define the lifecycle in the template section or include a composable template that defines the lifecycle.
+
+* Use a priority higher than `200` to avoid collisions with built-in templates.
+See <<avoid-index-pattern-collisions>>.
+
+You can use the <<indices-put-template,create index template API>>.
+
+[source,console]
+--------------------------------------------------
+PUT _index_template/my-index-template
+{
+  "index_patterns": ["my-data-stream*"],
+  "data_stream": { },
+  "priority": 500,
+  "template": {
+    "lifecycle": {
+      "data_retention": "7d"
+    }
+  },
+  "_meta": {
+    "description": "Template with data stream lifecycle"
+  }
+}
+--------------------------------------------------
+
+[discrete]
+[[create-data-stream-with-lifecycle]]
+==== Create a data stream
+
+You can create a data stream in two ways:
+
+. By manually creating the stream using the <<indices-create-data-stream,create data stream API>>. The stream's name must
+still match one of your template's index patterns.
++
+[source,console]
+--------------------------------------------------
+PUT _data_stream/my-data-stream
+--------------------------------------------------
+// TEST[continued]
+
+. By <<add-documents-to-a-data-stream,indexing requests>> that
+target the stream's name. This name must match one of your index template's index patterns.
++
+[source,console]
+--------------------------------------------------
+PUT my-data-stream/_bulk
+{ "create":{ } }
+{ "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" }
+{ "create":{ } }
+{ "@timestamp": "2099-05-06T16:25:42.000Z", "message": "192.0.2.255 - - [06/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" }
+--------------------------------------------------
+// TEST[continued]
+
+[discrete]
+[[retrieve-lifecycle-information]]
+==== Retrieve lifecycle information
+
+You can use the <<data-streams-get-lifecycle,get data stream lifecycle API>> to see the data lifecycle of your data stream and
+the <<data-streams-explain-lifecycle,explain data lifecycle API>> to see the exact state of each backing index.
+
+[source,console]
+--------------------------------------------------
+GET _data_stream/my-data-stream/_lifecycle
+--------------------------------------------------
+// TEST[continued]
+
+The result will look like this:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "data_streams": [
+    {
+      "name": "my-data-stream",<1>
+      "lifecycle": {
+        "data_retention": "7d" <2>
+      }
+    }
+  ]
+}
+--------------------------------------------------
+<1> The name of your data stream.
+<2> The retention period of the data indexed in this data stream, this means that the data in this data stream will
+be kept at least for 7 days. After that {es} can delete it at its own discretion.
+
+If you want to see more information about how the data stream lifecycle is applied on individual backing indices use the
+<<data-streams-explain-lifecycle,explain data lifecycle API>>:
+
+[source,console]
+--------------------------------------------------
+GET .ds-my-data-stream-*/_lifecycle/explain
+--------------------------------------------------
+// TEST[continued]
+The result will look like this:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "indices": {
+    ".ds-my-data-stream-2023.04.19-000001": {
+      "index": ".ds-my-data-stream-2023.04.19-000001",      <1>
+      "managed_by_lifecycle": true,                               <2>
+      "index_creation_date_millis": 1681918009501,
+      "time_since_index_creation": "1.6m",                  <3>
+      "lifecycle": {                                        <4>
+        "data_retention": "7d"
+      }
+    }
+  }
+}
+--------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only]
+<1> The name of the backing index.
+<2> If it is managed by the built-in data stream lifecycle.
+<3> Time since the index was created.
+<4> The lifecycle configuration that is applied on this backing index.
+
+//////////////////////////
+[source,console]
+--------------------------------------------------
+DELETE _data_stream/my-data-stream
+DELETE _index_template/my-index-template
+--------------------------------------------------
+// TEST[continued]
+
+//////////////////////////

docs/reference/dlm/apis/dlm-api.asciidoc

@@ -1,20 +0,0 @@
-[[data-lifecycle-management-api]]
-== Data Lifecycle Management APIs
-
-You use the following APIs to configure the data lifecycle management for data streams
-and to retrieve lifecycle information for backing indices.
-
-[discrete]
-[[dlm-api-management-endpoint]]
-=== Operation management APIs
-
-* <<dlm-put-lifecycle,Update data stream lifecycle>>
-* <<dlm-get-lifecycle,Get data stream lifecycle>>
-* <<dlm-delete-lifecycle,Delete data stream lifecycle>>
-* <<dlm-explain-lifecycle,Explain Lifecycle API>>
-
-include::put-lifecycle.asciidoc[]
-include::get-lifecycle.asciidoc[]
-include::delete-lifecycle.asciidoc[]
-include::explain-data-lifecycle.asciidoc[]
-

docs/reference/ilm/index-rollover.asciidoc

@@ -42,7 +42,7 @@ On each rollover, the new index becomes the write index.
 [[ilm-automatic-rollover]]
 === Automatic rollover
 
-{ilm-init} enables you to automatically roll over to a new index based
+{ilm-init} and the data stream lifecycle (in preview:[]]) enable you to automatically roll over to a new index based
 on conditions like the index size, document count, or age. When a rollover is triggered, a new
 index is created, the write alias is updated to point to the new index, and all
 subsequent updates are written to the new index.

docs/reference/indices/get-component-template.asciidoc

@@ -72,7 +72,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=local]
 include::{docdir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 `include_defaults`::
-(Optional, Boolean) Functionality in experimental:[]. If `true`, return all default settings in the response.
+(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
 Defaults to `false`.
 
 [[get-component-template-api-example]]

docs/reference/indices/get-data-stream.asciidoc

@@ -100,7 +100,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
 Defaults to `open`.
 
 `include_defaults`::
-(Optional, Boolean) Functionality in experimental:[]. If `true`, return all default settings in the response.
+(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
 Defaults to `false`.
 
 [role="child_attributes"]
@@ -223,7 +223,7 @@ cluster can not write into this data stream or change its mappings.
 
 `lifecycle`::
 (object)
-Functionality in experimental:[]. Contains the configuration for the data lifecycle management of this data stream.
+Functionality in preview:[]. Contains the configuration for the data lifecycle management of this data stream.
 +
 .Properties of `lifecycle`
 [%collapsible%open]

docs/reference/indices/get-index-template.asciidoc

@@ -64,7 +64,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=local]
 include::{docdir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 `include_defaults`::
-(Optional, Boolean) Functionality in experimental:[]. If `true`, return all default settings in the response.
+(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
 Defaults to `false`.
 
 [[get-template-api-example]]

docs/reference/indices/simulate-index.asciidoc

@@ -63,7 +63,7 @@ Name of the index to simulate.
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 `include_defaults`::
-(Optional, Boolean) Functionality in experimental:[]. If `true`, return all default settings in the response.
+(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
 Defaults to `false`.
 
 [role="child_attributes"]

docs/reference/indices/simulate-template.asciidoc

@@ -95,7 +95,7 @@ Defaults to `false`.
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 `include_defaults`::
-(Optional, Boolean) Functionality in experimental:[]. If `true`, return all default settings in the response.
+(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
 Defaults to `false`.
 
 [role="child_attributes"]

docs/reference/redirects.asciidoc

@@ -1882,6 +1882,26 @@ Refer to <<example-watches>> for other Watcher examples.
 
 Refer to <<fix-watermark-errors>>.
 
+[role="exclude",id="dlm-delete-lifecycle"]
+=== Delete the lifecycle of a data stream
+
+Refer to <<data-streams-delete-lifecycle,Delete data stream lifecycle API>>.
+
+[role="exclude",id="dlm-explain-lifecycle"]
+=== Explain the lifecycle of a data stream
+
+Refer to <<data-streams-explain-lifecycle,Explain data stream lifecycle API>>.
+
+[role="exclude",id="dlm-get-lifecycle"]
+=== Get the lifecycle of a data stream
+
+Refer to <<data-streams-get-lifecycle,Get data stream lifecycle API>>.
+
+[role="exclude",id="dlm-put-lifecycle"]
+=== Update the lifecycle of a data stream
+
+Refer to <<data-streams-delete-lifecycle,Update data stream lifecycle API>>.
+
 [role="exclude",id="get-synonym-rule"]
 === Get synonym rule API
 

docs/reference/rest-api/index.asciidoc

@@ -20,7 +20,6 @@ not be included yet.
 * <<features-apis,Features APIs>>
 * <<ccr-apis,{ccr-cap} APIs>>
 * <<data-stream-apis,Data stream APIs>>
-* <<data-lifecycle-management-api, Data Lifecycle Management APIs>>
 * <<docs, Document APIs>>
 * <<enrich-apis,Enrich APIs>>
 * <<eql-apis,EQL search APIs>>
@@ -98,4 +97,3 @@ include::{es-repo-dir}/transform/apis/index.asciidoc[]
 include::usage.asciidoc[]
 include::{xes-repo-dir}/rest-api/watcher.asciidoc[]
 include::defs.asciidoc[]
-include::{es-repo-dir}/dlm/apis/dlm-api.asciidoc[]

docs/reference/settings/data-stream-lifecycle-settings.asciidoc

@@ -0,0 +1,51 @@
+[role="xpack"]
+[[data-stream-lifecycle-settings]]
+=== Data stream lifecycle settings in {es}
+[subs="attributes"]
+++++
+<titleabbrev>Data stream lifecycle settings</titleabbrev>
+++++
+
+preview::[]
+
+These are the settings available for configuring <<data-stream-lifecycle, data stream lifecycle>>.
+
+==== Cluster level settings
+
+[[data-streams-lifecycle-poll-interval]]
+`data_streams.lifecycle.poll_interval`::
+(<<dynamic-cluster-setting,Dynamic>>, <<time-units, time unit value>>)
+How often {es} checks what is the next action for all data streams with a built-in lifecycle. Defaults to `10m`.
+
+[[cluster-lifecycle-default-rollover]]
+`cluster.lifecycle.default.rollover`::
+(<<dynamic-cluster-setting,Dynamic>>, string)
+This property accepts a key value pair formatted string and configures the conditions that would trigger a data stream
+to <<index-rollover,rollover>> when it has `lifecycle` configured. This property is an implementation detail and subject to
+change. Currently, it defaults to `max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000`,
+this means that your data stream will rollover if any of the following conditions are met:
+
+* Either any primary shard reaches the size of 50GB,
+* or any primary shard contains 200.000.000 documents
+* or the index reaches a certain age which depends on the retention time of your data stream,
+* **and** has at least one document.
+
+==== Index level settings
+The following index-level settings are typically configured on the backing indices of a data stream.
+
+[[index-lifecycle-prefer-ilm]]
+`index.lifecycle.prefer_ilm`::
+(<<indices-update-settings,Dynamic>>, boolean)
+This setting determines which feature is managing the backing index of a data stream if, and only if, the backing index
+has an <<index-lifecycle-management,{ilm}>> ({ilm-init}) policy and the data stream has also a built-in lifecycle. When
+`true` this index is managed by {ilm-init}, when `false` the backing index is managed by the data stream lifecycle.
+Defaults to `true`.
+
+[[index-data-stream-lifecycle-origination-date]]
+`index.lifecycle.origination_date`::
+(<<indices-update-settings,Dynamic>>, long)
+If specified, this is the timestamp used to calculate the backing index generation age after this backing index has been
+<<index-rollover,rolled over>>. The generation age is used to determine data retention, consequently, you can use this
+setting if you create a backing index that contains older data and want to ensure that the retention period or
+other parts of the lifecycle will be applied based on the data's original timestamp and not the timestamp they got
+indexed. Specified as a Unix epoch value in milliseconds.

docs/reference/setup.asciidoc

@@ -54,6 +54,8 @@ include::settings/health-diagnostic-settings.asciidoc[]
 
 include::settings/ilm-settings.asciidoc[]
 
+include::settings/data-stream-lifecycle-settings.asciidoc[]
+
 include::modules/indices/index_management.asciidoc[]
 
 include::modules/indices/recovery.asciidoc[]

rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_lifecycle.json

@@ -1,7 +1,7 @@
 {
   "indices.delete_data_lifecycle":{
     "documentation":{
-      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/dlm-delete-lifecycle.html",
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-delete-lifecycle.html",
       "description":"Deletes the data lifecycle of the selected data streams."
     },
     "stability":"experimental",

rest-api-spec/src/main/resources/rest-api-spec/api/indices.explain_data_lifecycle.json

@@ -1,8 +1,8 @@
 {
   "indices.explain_data_lifecycle": {
     "documentation": {
-      "url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/dlm-explain-lifecycle.html",
-      "description": "Retrieves information about the index's current DLM lifecycle, such as any potential encountered error, time since creation etc."
+      "url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams-explain-lifecycle.html",
+      "description": "Retrieves information about the index's current data stream lifecycle, such as any potential encountered error, time since creation etc."
     },
     "stability": "experimental",
     "visibility": "public",

rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_lifecycle.json

@@ -1,7 +1,7 @@
 {
   "indices.get_data_lifecycle":{
     "documentation":{
-      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/dlm-get-lifecycle.html",
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-get-lifecycle.html",
       "description":"Returns the data lifecycle of the selected data streams."
     },
     "stability":"experimental",

rest-api-spec/src/main/resources/rest-api-spec/api/indices.put_data_lifecycle.json

@@ -1,7 +1,7 @@
 {
   "indices.put_data_lifecycle":{
     "documentation":{
-      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/dlm-put-lifecycle.html",
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-put-lifecycle.html",
       "description":"Updates the data lifecycle of the selected data streams."
     },
     "stability":"experimental",