elasticsearch/docs/reference/data-streams/lifecycle/tutorial-manage-data-stream-retention.asciidoc

[role="xpack"]
[[tutorial-manage-data-stream-retention]]
=== Tutorial: Data stream retention

preview::[]

In this tutorial, we are going to go over the data stream lifecycle retention, define it, go over how it can be configured and how
it can be applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle.

. <<what-is-retention>>
. <<retention-configuration>>
. <<effective-retention-calculation>>
. <<effective-retention-application>>

You can verify if a data steam is managed by the data stream lifecycle via the <<data-streams-get-lifecycle,get data stream lifecycle API>>:

////
[source,console]
----
PUT /_index_template/template
{
  "index_patterns": ["my-data-stream*"],
  "template": {
    "lifecycle": {}
  },
  "data_stream": { }
}

PUT /_data_stream/my-data-stream
----
// TESTSETUP
////

////
[source,console]
----
DELETE /_data_stream/my-data-stream*
DELETE /_index_template/template
DELETE /_data_stream/_global_retention
----
// TEARDOWN
////

[source,console]
--------------------------------------------------
GET _data_stream/my-data-stream/_lifecycle
--------------------------------------------------

The result should look like this:

[source,console-result]
--------------------------------------------------
{
  "data_streams": [
    {
      "name": "my-data-stream",                                   <1>
      "lifecycle": {
        "enabled": true                                           <2>
      }
    }
  ]
}
--------------------------------------------------
// TESTRESPONSE[skip:the result is for illustrating purposes only]
<1> The name of your data stream.
<2> Ensure that the lifecycle is enabled, meaning this should be `true`.

[discrete]
[[what-is-retention]]
==== What is data stream retention?

We define retention as the least amount of time the data of a data stream are going to be kept in {es}. After this time period
has passed, {es} is allowed to remove these data to free up space and/or manage costs.

NOTE: Retention does not define the period that the data will be removed, but the minimum time period they will be kept.

We define 4 different types of retention:

* The data stream retention, or `data_retention`, which is the retention configured on the data stream level. It can be
set via an <<index-templates,index template>> for future data streams or via the <<data-streams-put-lifecycle, PUT data
stream lifecycle API>> for an existing data stream. When the data stream retention is not set, it implies that the data
need to be kept forever.
* The global default retention, or `default_retention`, which is a retention configured on a cluster level and will be
applied to all data streams managed by data stream lifecycle that do not have `data_retention` configured. Effectively,
it ensures that there will be no data streams keeping their data forever. This can be set via the
<<data-streams-put-global-retention, PUT global retention API>>.
* The global max retention, or `max_retention`, which is a retention configured on a cluster level and will be applied to
all data streams managed by data stream lifecycle. Effectively, it ensures that there will be no data streams whose retention
will exceed this time period. This can be set via the <<data-streams-put-global-retention, PUT global retention API>>.
* The effective retention, or `effective_retention`, which is the retention applied at a data stream on a given moment.
Effective retention cannot be set, it is derived by taking into account all the configured retention listed above and is
calculated as it is described <<effective-retention-calculation,here>>.

[discrete]
[[retention-configuration]]
==== How to configure retention?

- By setting the `data_retention` on the data stream level. This retention can be configured in two ways:
+
-- For new data streams, it can be defined in the index template that would be applied during the data stream's creation.
You can use the <<indices-put-template,create index template API>>, for example:
+
[source,console]
--------------------------------------------------
PUT _index_template/template
{
  "index_patterns": ["my-data-stream*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "lifecycle": {
      "data_retention": "7d"
    }
  },
  "_meta": {
    "description": "Template with data stream lifecycle"
  }
}
--------------------------------------------------
-- For an existing data stream, it can be set via the <<data-streams-put-lifecycle, PUT lifecycle API>>.
+
[source,console]
----
PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "30d" <1>
}
----
// TEST[continued]
<1> The retention period of this data stream is set to 30 days.

- By setting the global retention via the `default_retention` and `max_retention` that are set on a cluster level. You
can set them via the <<data-streams-put-global-retention, PUT global retention API>>. For example:
+
[source,console]
--------------------------------------------------
PUT _data_stream/_global_retention
{
  "default_retention": "7d",
  "max_retention": "90d"
}
--------------------------------------------------
// TEST[continued]

[discrete]
[[effective-retention-calculation]]
==== How is the effective retention calculated?
The effective is calculated in the following way:

- The `effective_retention` is the `default_retention`, when `default_retention` is defined and the data stream does not
have `data_retention`.
- The `effective_retention` is the `data_retention`, when `data_retention` is defined and if `max_retention` is defined,
it is less than the `max_retention`.
- The `effective_retention` is the `max_retention`, when `max_retention` is defined, and the data stream has either no
`data_retention` or its `data_retention` is greater than the `max_retention`.

The above is demonstrated in the examples below:

|===
|`default_retention`    |`max_retention`    |`data_retention`   |`effective_retention`  |Retention determined by

|Not set                |Not set            |Not set            |Infinite               |N/A
|Not relevant           |12 months          |**30 days**        |30 days                |`data_retention`
|Not relevant           |Not set            |**30 days**        |30 days                |`data_retention`
|**30 days**            |12 months          |Not set            |30 days                |`default_retention`
|**30 days**            |30 days            |Not set            |30 days                |`default_retention`
|Not relevant           |**30 days**        |12 months          |30 days                |`max_retention`
|Not set                |**30 days**        |Not set            |30 days                |`max_retention`
|===

[discrete]
[[effective-retention-application]]
==== How is the effective retention applied?

Retention is applied to the remaining backing indices of a data stream as the last step of
<<data-streams-lifecycle-how-it-works, a data stream lifecycle run>>. Data stream lifecycle will retrieve the backing indices
whose `generation_time` is longer than the effective retention period and delete them. The `generation_time` is only
applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time
optionally configured in the <<index-data-stream-lifecycle-origination-date,`index.lifecycle.origination_date`>> setting.

IMPORTANT: We use the `generation_time` instead of the creation time because this ensures that all data in the backing
index have passed the retention period. As a result, the retention period is not the exact time data get deleted, but
the minimum time data will be stored.