[DOCS] Add TSDS docs (#86905)

* [DOCS] Add TSDB docs

* Update docs/build.gradle

Co-authored-by: Adam Locke <adam.locke@elastic.co>

* Address Nik's comments, part 1

* Address Nik's comments, part deux

* Reword write index

* Add feature flags

* Wrap one more section in feature flag

* Small fixes

* set index.routing_path to optional

* Update storage reduction value

* Update create index template code example

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>

docs/build.gradle

@@ -220,6 +220,58 @@ teardowns['data_stream_cleanup'] = '''
   - is_true: acknowledged
 '''
 
+// Used for TSDS examples
+setups['tsds_template'] = '''
+  - do:
+      indices.put_index_template:
+        name: my-weather-sensor-index-template
+        body: |
+          {
+            "index_patterns": [ "metrics-weather_sensors-*" ],
+            "data_stream": {
+              "index_mode": "time_series"
+            },
+            "template": {
+              "settings": {
+                "index.routing_path": [ "sensor_id", "location" ]
+              },
+              "mappings": {
+                "properties": {
+                  "sensor_id": {
+                    "type": "keyword",
+                    "time_series_dimension": true
+                  },
+                  "location": {
+                    "type": "keyword",
+                    "time_series_dimension": true
+                  }
+                }
+              }
+            },
+            "priority": 500
+          }
+'''
+
+setups['tsds'] = setups['tsds_template'] + '''
+  - do:
+      raw:
+        method: PUT
+        path: '_data_stream/metrics-weather_sensors-dev'
+'''
+
+teardowns['tsds_cleanup'] = '''
+  - do:
+      raw:
+        method: DELETE
+        path: '_data_stream/*'
+  - is_true: acknowledged
+  - do:
+      raw:
+        method: DELETE
+        path: '_index_template/*'
+  - is_true: acknowledged
+'''
+
 // Used for several full-text search and agg examples
 buildRestTests.setups['messages'] = '''
   - do:

docs/reference/data-streams/data-streams.asciidoc

@@ -126,3 +126,4 @@ alias with a write index instead of a data stream. See
 include::set-up-a-data-stream.asciidoc[]
 include::use-a-data-stream.asciidoc[]
 include::change-mappings-and-settings.asciidoc[]
+include::tsds.asciidoc[]

docs/reference/data-streams/set-up-tsds.asciidoc

@@ -0,0 +1,351 @@
+[[set-up-tsds]]
+=== Set up a time series data stream (TSDS)
+++++
+<titleabbrev>Set up a TSDS</titleabbrev>
+++++
+
+preview::[]
+
+To set up a <<tsds,time series data stream (TSDS)>>, follow these steps:
+
+. Check the <<tsds-prereqs,prerequisites>>.
+. <<tsds-ilm-policy>>.
+. <<tsds-create-mappings-component-template>>.
+. <<tsds-create-index-settings-component-template>>.
+. <<create-tsds-index-template>>.
+. <<create-tsds>>.
+. <<secure-tsds>>.
+
+[discrete]
+[[tsds-prereqs]]
+==== Prerequisites
+
+* Before you create a TSDS, you should be familiar with <<data-streams,data
+streams>> and <<tsds,TSDS concepts>>.
+
+* To follow this tutorial, you must have the following permissions:
+
+** <<privileges-list-cluster,Cluster privileges>>: `manage_ilm` and
+`manage_index_templates`.
+** <<privileges-list-indices,Index privileges>>: `create_doc` and `create_index`
+for any TSDS you create or convert. To roll over a TSDS, you must have the
+`manage` privilege.
+
+[discrete]
+[[tsds-ilm-policy]]
+==== Create an index lifecycle policy
+
+While optional, we recommend using {ilm-init} to automate the management of your
+TSDS’s backing indices. {ilm-init} requires an index lifecycle policy.
+
+We recommend you specify a `max_age` criteria for the `rollover` action in the
+policy. This ensures the <<time-bound-indices,`@timestamp` ranges>> for the
+TSDS's backing indices are consistent. For example, setting a `max_age` of `1d`
+for the `rollover` action ensures your backing indices consistently contain one
+day's worth of data.
+
+////
+[source,console]
+----
+PUT /_snapshot/found-snapshots
+{
+ "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+PUT _ilm/policy/my-weather-sensor-lifecycle-policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover": {
+            "max_age": "1d",
+            "max_primary_shard_size": "50gb"
+          }
+        }
+      },
+      "warm": {
+        "min_age": "30d",
+        "actions": {
+          "shrink": {
+            "number_of_shards": 1
+          },
+          "forcemerge": {
+            "max_num_segments": 1
+          }
+        }
+      },
+      "cold": {
+        "min_age": "60d",
+        "actions": {
+          "searchable_snapshot": {
+            "snapshot_repository": "found-snapshots"
+          }
+        }
+      },
+      "frozen": {
+        "min_age": "90d",
+        "actions": {
+          "searchable_snapshot": {
+            "snapshot_repository": "found-snapshots"
+          }
+        }
+      },
+      "delete": {
+        "min_age": "735d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}
+----
+
+[discrete]
+[[tsds-create-mappings-component-template]]
+==== Create a mappings component template
+
+A TSDS requires a matching index template. In most cases, you compose this index
+template using one or more component templates. You typically use separate
+component templates for mappings and index settings. This lets you reuse the
+component templates in multiple index templates.
+
+For a TSDS, the mappings component template must include mappings for:
+
+* One or more <<time-series-dimension,dimension fields>> with a
+`time_series_dimension` value of `true`. At least one of these dimensions must
+be a plain `keyword` field.
+
+Optionally, the template can also include mappings for:
+
+* One or more <<time-series-metric,metric fields>>, marked using the
+`time_series_metric` mapping parameter.
+
+* A `date` or `date_nanos` mapping for the `@timestamp` field. If you don’t
+specify a mapping, Elasticsearch maps `@timestamp` as a `date` field with
+default options.
+
+[source,console]
+----
+PUT _component_template/my-weather-sensor-mappings
+{
+  "template": {
+    "mappings": {
+      "properties": {
+        "sensor_id": {
+          "type": "keyword",
+          "time_series_dimension": true
+        },
+        "location": {
+          "type": "keyword",
+          "time_series_dimension": true
+        },
+        "temperature": {
+          "type": "half_float",
+          "time_series_metric": "gauge"
+        },
+        "humidity": {
+          "type": "half_float",
+          "time_series_metric": "gauge"
+        },
+        "@timestamp": {
+          "type": "date",
+          "format": "strict_date_optional_time"
+        }
+      }
+    }
+  },
+  "_meta": {
+    "description": "Mappings for weather sensor data"
+  }
+}
+----
+// TEST[continued]
+
+[discrete]
+[[tsds-create-index-settings-component-template]]
+==== Create an index settings component template
+
+Optionally, the index settings component template for a TSDS can include:
+
+* Your lifecycle policy in the `index.lifecycle.name` index setting.
+* The <<tsds-look-ahead-time,`index.look_ahead_time`>> index setting.
+* Other index settings, such as <<index-codec,`index.codec`>>, for your TSDS's
+backing indices.
+
+IMPORTANT: Don't specify the `index.routing_path` index setting in a component
+template. If you choose, you can configure `index.routing_path` directly in the
+index template, as shown in the following step.
+
+[source,console]
+----
+PUT _component_template/my-weather-sensor-settings
+{
+  "template": {
+    "settings": {
+      "index.lifecycle.name": "my-lifecycle-policy",
+      "index.look_ahead_time": "3h",
+      "index.codec": "best_compression"
+    }
+  },
+  "_meta": {
+    "description": "Index settings for weather sensor data"
+  }
+}
+----
+// TEST[continued]
+
+[discrete]
+[[create-tsds-index-template]]
+==== Create an index template
+
+Use your component templates to create an index template. In the index template,
+specify:
+
+* One or more index patterns that match the TSDS's name. We recommend
+using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream
+naming scheme].
+
+* A `data_stream` object with an `index_mode` of `time_series`.
+
+* Optional: The `index.routing_path` index setting. The setting value should
+only match plain `keyword` dimension fields and should be set directly in the
+index template. When not defined explicitly, the `index.routing_path` setting is
+generated from the mapping using all mappings that are set with
+`time_series_dimension` set to `true`.
+
+* The component templates containing your mappings and other index settings.
+
+* A priority higher than `200` to avoid collisions with built-in templates.
+See <<avoid-index-pattern-collisions>>.
+
+[source,console]
+----
+PUT _index_template/my-weather-sensor-index-template
+{
+  "index_patterns": ["metrics-weather_sensors-*"],
+  "template": {
+    "settings": {
+      "index.mode": "time_series",
+      "index.routing_path": [ "sensor_id", "location" ]
+    }
+  },
+  "composed_of": [ "my-weather-sensor-mappings", "my-weather-sensor-settings" ],
+  "priority": 500,
+  "_meta": {
+    "description": "Template for my weather sensor data"
+  }
+}
+----
+// TEST[continued]
+
+////
+[source,console]
+----
+DELETE _data_stream/*
+DELETE _index_template/*
+DELETE _component_template/my-*
+DELETE _ilm/policy/my-weather-sensor-lifecycle-policy
+----
+// TEST[continued]
+////
+
+[discrete]
+[[create-tsds]]
+==== Create the TSDS
+
+<<add-documents-to-a-data-stream,Indexing requests>> add documents to a TSDS.
+Documents in a TSDS must include:
+
+* A `@timestamp` field
+* One or more dimension fields. At least one dimension must be a `keyword` field
+that matches the `index.routing_path` index setting, if specified. If not specified
+explicitly, `index.routing_path` is set automatically to whichever mappings have
+ `time_series_dimension` set to `true`.
+
+To automatically create your TSDS, submit an indexing request that
+targets the TSDS's name. This name must match one of your index template's
+index patterns.
+
+[source,console]
+----
+PUT metrics-weather_sensors-dev/_bulk
+{ "create":{ } }
+{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 }
+{ "create":{ } }
+{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }
+
+POST metrics-weather_sensors-dev/_doc
+{
+  "@timestamp": "2099-05-06T16:21:15.000Z",
+  "sensor_id": "SYKENET-000001",
+  "location": "swamp",
+  "temperature": 32.4,
+  "humidity": 88.9
+}
+----
+// TEST[skip: The @timestamp value won't match an accepted range in the TSDS]
+
+You can also manually create the TSDS using the
+<<indices-create-data-stream,create data stream API>>. The TSDS's name must
+still match one of your template's index patterns.
+
+[source,console]
+----
+PUT _data_stream/metrics-weather_sensors-dev
+----
+// TEST[setup:tsds_template]
+// TEST[teardown:tsds_cleanup]
+
+[discrete]
+[[secure-tsds]]
+==== Secure the TSDS
+
+Use <<privileges-list-indices,index privileges>> to control access to a TSDS.
+Granting privileges on a TSDS grants the same privileges on its backing indices.
+
+For an example, refer to <<data-stream-privileges>>.
+
+[discrete]
+[[convert-existing-data-stream-to-tsds]]
+==== Convert an existing data stream to a TSDS
+
+You can also use the above steps to convert an existing regular data stream to
+a TSDS. In this case, you'll want to:
+
+* Edit your existing index lifecycle policy, component templates, and index
+templates instead of creating new ones.
+
+* Instead of creating the TSDS, manually roll over its write index. This ensures
+the current write index and any new backing indices have an
+<<time-series-mode,`index.mode` of `time_series`>>.
++
+You can manually roll over the write index using the
+<<indices-rollover-index,rollover API>>.
++
+[source,console]
+----
+POST metrics-weather_sensors-dev/_rollover
+----
+// TEST[setup:tsds]
+// TEST[teardown:tsds_cleanup]
+
+[discrete]
+[[set-up-tsds-whats-next]]
+==== What's next?
+
+Now that you've set up your TSDS, you can manage and use it like a regular
+data stream. For more information, refer to:
+
+* <<use-a-data-stream>>
+* <<data-streams-change-mappings-and-settings>>
+* <<data-stream-apis,data stream APIs>>

docs/reference/data-streams/tsds-index-settings.asciidoc

@@ -0,0 +1,55 @@
+ifeval::["{release-state}"=="unreleased"]
+[[tsds-index-settings]]
+=== Time series index settings
+
+preview::[]
+
+Backing indices in a <<tsds,time series data stream (TSDS)>> support the
+following index settings.
+
+[[index-mode]]
+`index.mode`::
+preview:[] (<<_static_index_settings,Static>>, string) Mode for the index.
+Valid values are <<time-series-mode,`time_series`>> and `null` (no mode).
+Defaults to `null`.
+
+[[index-time-series-start-time]]
+`index.time_series.start_time`::
+preview:[] (<<_static_index_settings,Static>>, string) Earliest `@timestamp`
+value (inclusive) accepted by the index. Only indices with an `index.mode` of
+<<time-series-mode,`time_series`>> support this setting. For more information,
+refer to <<time-bound-indices>>.
+
+[[index-time-series-end-time]]
+`index.time_series.end_time`::
+preview:[] (<<dynamic-index-settings,Dynamic>>, string) Latest `@timestamp`
+value (exclusive) accepted by the index. Only indices with an `index.mode` of
+`time_series` support this setting. For more information, refer to
+<<time-bound-indices>>.
+
+[[index-look-ahead-time]]
+`index.look_ahead_time`::
+preview:[] (<<_static_index_settings,Static>>, <<time-units,time units>>)
+Interval used to calculate the `index.time_series.end_time` for a TSDS's write
+index. Defaults to `2h` (2 hours). Accepts `1m` (one minute) to `7d` (seven
+days). Only indices with an `index.mode` of `time_series` support this setting.
+For more information, refer to <<tsds-look-ahead-time>>.
+
+[[index-routing-path]] `index.routing_path`:: preview:[]
+(<<_static_index_settings,Static>>, string or array of strings) Plain `keyword`
+fields used to route documents in a TSDS to index shards. Supports wildcards
+(`*`). Only indices with an `index.mode` of `time_series` support this setting.
+Defaults to the list of <<time-series-dimension,dimension fields>> with a
+`time_series_dimension` value of `true` defined in your component templates. For
+more information, refer to <<dimension-based-routing>>.
+
+[[index-mapping-dimension-fields-limit]]
+// tag::dimensions-limit[]
+`index.mapping.dimension_fields.limit`::
+preview:[] (<<dynamic-index-settings,Dynamic>>, integer)
+//Maximum number of <<time-series-dimension,time series dimensions>> for the
+//index. Defaults to `16`.
+Maximum number of time series dimensions for the
+index. Defaults to `16`.
+// end::dimensions-limit[]
+endif::[]

docs/reference/data-streams/tsds.asciidoc

@@ -0,0 +1,308 @@
+ifeval::["{release-state}"=="unreleased"]
+[[tsds]]
+== Time series data stream (TSDS)
+
+preview::[]
+
+A time series data stream (TSDS) models timestamped metrics data as one or
+more time series.
+
+// TODO: Replace XX% with actual percentage
+You can use a TSDS to store metrics data more efficiently. In our benchmarks,
+metrics data stored in a TSDS used 44% less disk space than a regular data
+stream.
+
+[discrete]
+[[when-to-use-tsds]]
+=== When to use a TSDS
+
+Both a <<data-streams,regular data stream>> and a TSDS can store timestamped
+metrics data. Only use a TSDS if you typically add metrics data to {es} in near
+real-time and `@timestamp` order.
+
+A TSDS is only intended for metrics data. For other timestamped data, such as
+logs or traces, use a regular data stream.
+
+[discrete]
+[[differences-from-regular-data-stream]]
+=== Differences from a regular data stream
+
+A TSDS works like a regular data stream with some key differences:
+
+* The matching index template for a TSDS requires a `data_stream` object with
+the <<time-series-mode,`index_mode: time_series`>> option. This option enables
+most TSDS-related functionality.
+
+* In addition to a `@timestamp`, each document in a TSDS must contain one or
+more <<time-series-dimension,dimension fields>>. The matching index template for
+a TSDS must contain mappings for at least one `keyword` dimension.
++
+TSDS documents also typically
+contain one or more <<time-series-metric,metric fields>>.
+
+* {es} generates a hidden <<tsid,`_tsid`>> metadata field for each document in a
+TSDS.
+
+* A TSDS uses <<time-bound-indices,time-bound backing indices>> to store data
+from the same time period in the same backing index.
+
+* The matching index template for a TSDS must contain the `index.routing_path`
+index setting. A TSDS uses this setting to perform
+<<dimension-based-routing,dimension-based routing>>.
+
+* A TSDS uses internal <<index-modules-index-sorting,index sorting>> to order
+shard segments by `_tsid` and `@timestamp`.
+
+* TSDS documents only support auto-generated document `_id` values. For TSDS
+documents, the document `_id` is a hash of the document's dimensions and
+`@timestamp`. A TSDS doesn't support custom document `_id` values.
+
+[discrete]
+[[time-series]]
+=== What is a time series?
+
+A time series is a sequence of observations for a specific entity. Together,
+these observations let you track changes to the entity over time. For example, a
+time series can track:
+
+* CPU and disk usage for a computer
+* The price of a stock
+* Temperature and humidity readings from a weather sensor.
+
+.Time series of weather sensor readings plotted as a graph
+image::images/data-streams/time-series-chart.svg[align="center"]
+
+In a TSDS, each {es} document represents an observation, or data point, in a
+specific time series. Although a TSDS can contain multiple time series, a
+document can only belong to one time series. A time series can't span multiple
+data streams.
+
+[discrete]
+[[time-series-dimension]]
+==== Dimensions
+
+Dimensions are field names and values that, in combination, identify a
+document's time series. In most cases, a dimension describes some aspect of the
+entity you're measuring. For example, documents related to the same weather
+sensor may always have the same `sensor_id` and `location` values.
+
+A TSDS document is uniquely identified by its time series and timestamp, both of
+which are used to generate the document `_id`. So, two documents with the same
+dimensions and the same timestamp are considered to be duplicates. When you use
+the `_bulk` endpoint to add documents to a TSDS, a second document with the same
+timestamp and dimensions overwrites the first. When you use the 
+`PUT /<target>/_create/<_id>` format to add an individual document and a document
+with the same `_id` already exists, an error is generated.
+
+You mark a field as a dimension using the boolean `time_series_dimension`
+mapping parameter. The following field types support the `time_series_dimension`
+parameter:
+
+* <<keyword-field-type,`keyword`>>
+* <<ip,`ip`>>
+* <<number,`byte`>>
+* <<number,`short`>>
+* <<number,`integer`>>
+* <<number,`long`>>
+* <<number,`unsigned_long`>>
+
+[[dimension-limits]] 
+.Dimension limits 
+**** 
+In a TSDS, {es} uses dimensions to
+generate the document `_id` and <<tsid,`_tsid`>> values. The resulting `_id` is
+always a short encoded hash. To prevent the `_tsid` value from being overly
+large, {es} limits the number of dimensions for an index using the
+<<index-mapping-dimension-fields-limit,`index.mapping.dimension_fields.limit`>>
+index setting. While you can increase this limit, the resulting document `_tsid`
+value can't exceed 32KB. 
+****
+
+[discrete]
+[[time-series-metric]]
+==== Metrics
+
+Metrics are fields that contain numeric measurements, as well as aggregations
+and/or downsampling values based off of those measurements. While not required,
+documents in a TSDS typically contain one or more metric fields.
+
+Metrics differ from dimensions in that while dimensions generally remain
+constant, metrics are expected to change over time, even if rarely or slowly.
+
+To mark a field as a metric, you must specify a metric type using the
+`time_series_metric` mapping parameter. The following field types support the
+`time_series_metric` parameter:
+
+* <<aggregate-metric-double,`aggregate_metric_double`>>
+* <<histogram,`histogram`>>
+* All <<number,numeric field types>>
+
+Accepted metric types vary based on the field type:
+
+.Valid values for `time_series_metric`
+[%collapsible%open]
+====
+// tag::time-series-metric-counter[]
+`counter`:: A number that only increases or resets to `0` (zero). For
+example, a count of errors or completed tasks.
+// end::time-series-metric-counter[]
++
+Only numeric and `aggregate_metric_double` fields support the `counter` metric
+type.
+
+// tag::time-series-metric-gauge[]
+`gauge`:: A number that can increase or decrease. For example, a temperature or
+available disk space.
+// end::time-series-metric-gauge[]
++
+Only numeric and `aggregate_metric_double` fields support the `gauge` metric
+type.
+
+// tag::time-series-metric-histogram[]
+`histogram`:: A pair of numeric arrays that measure the distribution of values
+across predefined buckets. For example, server response times by percentile.
+// end::time-series-metric-histogram[]
++
+Only `histogram` fields support the `histogram` metric type.
+
+// tag::time-series-metric-summary[]
+`summary`:: An array of aggregated values, such as `sum`, `avg`, `value_count`,
+`min`, and `max`.
+// end::time-series-metric-summary[]
++
+Only `aggregate_metric_double` fields support the `gauge` metric type.
+
+// tag::time-series-metric-null[]
+`null` (Default):: Not a time series metric.
+// end::time-series-metric-null[]
+====
+
+[discrete]
+[[time-series-mode]]
+=== Time series mode
+
+The matching index template for a TSDS must contain a `data_stream` object with
+the `index_mode: time_series` option. This option ensures the TSDS creates
+backing indices with an <<index-mode,`index.mode`>> setting of `time_series`.
+This setting enables most TSDS-related functionality in the backing indices.
+
+If you convert an existing data stream to a TSDS, only backing indices created
+after the conversion have an `index.mode` of `time_series`. You can't
+change the `index.mode` of an existing backing index.
+
+[discrete]
+[[tsid]]
+==== `_tsid` metadata field
+
+When you add a document to a TSDS, {es} automatically generates a `_tsid`
+metadata field for the document. The `_tsid` is an object containing the
+document's dimensions. Documents in the same TSDS with the same `_tsid` are part
+of the same time series.
+
+The `_tsid` field is not queryable or updatable. You also can't retrieve a
+document's `_tsid` using a <<docs-get,get document>> request. However, you can
+use the `_tsid` field in aggregations and retrieve the `_tsid` value in searches
+using the <<search-fields-param,`fields` parameter>>.
+
+[discrete]
+[[time-bound-indices]]
+==== Time-bound indices
+
+In a TSDS, each backing index, including the most recent backing index, has a
+range of accepted `@timestamp` values. This range is defined by the
+<<index-time-series-start-time,`index.time_series.start_time`>> and
+<<index-time-series-end-time,`index.time_series.end_time`>> index settings.
+
+When you add a document to a TSDS, {es} adds the document to the appropriate
+backing index based on its `@timestamp` value. As a result, a TSDS can add
+documents to any TSDS backing index that can receive writes. This applies even
+if the index isn't the most recent backing index.
+
+image::images/data-streams/time-bound-indices.svg[align="center"]
+
+TIP: Some {ilm-init} actions, such as <<ilm-forcemerge,`forcemerge`>>,
+<<ilm-shrink,`shrink`>>, and <<ilm-searchable-snapshot,`searchable_snapshot`>>,
+make a backing index read-only. You cannot add documents to read-only indices.
+Keep this in mind when defining the index lifecycle policy for your TSDS.
+
+If no backing index can accept a document's `@timestamp` value, {es} rejects the
+document.
+
+
+{es} automatically configures `index.time_series.start_time` and
+`index.time_series.end_time` settings as part of the index creation and rollover
+process.
+
+[discrete]
+[[tsds-look-ahead-time]]
+==== Look-ahead time
+
+Use the <<index-look-ahead-time,`index.look_ahead_time`>> index setting to
+configure how far into the future you can add documents to an index. When you
+create a new write index for a TSDS, {es} calculates the index's
+`index.time_series.end_time` value as:
+
+`now + index.look_ahead_time`
+
+At the <<indices-lifecycle-poll-interval,`indices.lifecycle.poll_interval`>>,
+{es} checks if the write index has met the rollover criteria in its index
+lifecycle policy. If not, {es} refreshes the `now` value and updates the write
+index's `index.time_series.end_time` to:
+
+`now + index.look_ahead_time + indices.lifecycle.poll_interval`
+
+This process continues until the write index rolls over. When the index rolls
+over, {es} sets a final `index.time_series.end_time` value for the index. This
+value borders the `index.time_series.start_time` for the new write index. This
+ensures the `@timestamp` ranges for neighboring backing indices always border
+but never overlap.
+
+[discrete]
+[[dimension-based-routing]]
+==== Dimension-based routing
+
+Within each TSDS backing index, {es} uses the
+<<index-routing-path,`index.routing_path`>> index setting to route documents
+with the same dimensions to the same shards.
+
+When you create the matching index template for a TSDS, you must specify one or
+more dimensions in the `index.routing_path` setting. Each document in a TSDS
+must contain one or more dimensions that match the `index.routing_path` setting.
+
+Dimensions in the `index.routing_path` setting must be plain `keyword` fields.
+The `index.routing_path` setting accepts wildcard patterns (for example `dim.*`)
+and can dynamically match new fields. However, {es} will reject any mapping
+updates that add scripted, runtime, or non-dimension, non-`keyword` fields that
+match the `index.routing_path` value.
+
+TSDS documents don't support a custom `_routing` value. Similarly, you can't
+require a `_routing` value in mappings for a TSDS.
+
+[discrete]
+[[tsds-index-sorting]]
+==== Index sorting
+
+{es} uses <<index-codec,compression algorithms>> to compress repeated values.
+This compression works best when repeated values are stored near each other — in
+the same index, on the same shard, and side-by-side in the same shard segment.
+
+Most time series data contains repeated values. Dimensions are repeated across
+documents in the same time series. The metric values of a time series may also
+change slowly over time.
+
+Internally, each TSDS backing index uses <<index-modules-index-sorting,index
+sorting>> to order its shard segments by `_tsid` and `@timestamp`. This makes it
+more likely that these repeated values are stored near each other for better
+compression. A TSDS doesn't support any
+<<index-modules-index-sorting,`index.sort.*`>> index settings.
+
+[discrete]
+[[tsds-whats-next]]
+=== What's next?
+
+Now that you know the basics, you're ready to <<set-up-tsds,create a TSDS>> or
+<<set-up-tsds,convert an existing data stream to a TSDS>>.
+
+include::set-up-tsds.asciidoc[]
+include::tsds-index-settings.asciidoc[]
+endif::[]

docs/reference/indices/put-index-template.asciidoc

@@ -103,13 +103,22 @@ See <<create-index-template,create an index template>>.
 .Properties of `data_stream`
 [%collapsible%open]
 ====
+`allow_custom_routing`::
+(Optional, Boolean) If `true`, the data stream supports
+<<mapping-routing-field,custom routing>>. Defaults to `false`.
+
 `hidden`::
 (Optional, Boolean) If `true`, the data stream is <<hidden,hidden>>. Defaults to
 `false`.
 
-`allow_custom_routing`::
-(Optional, Boolean) If `true`, the data stream supports
-<<mapping-routing-field,custom routing>>. Defaults to `false`.
+ifeval::["{release-state}"=="unreleased"]
+`index_mode`::
+(Optional, string) Type of data stream to create. Valid values are `null`
+(regular data stream) and `time_series` (<<tsds,time series data stream>>).
++
+If `time_series`, each backing index has an `index.mode` index setting of
+`time_series`.
+endif::[]
 ====
 
 `index_patterns`::

docs/reference/mapping/mapping-settings-limit.asciidoc

@@ -48,18 +48,4 @@ If your field mappings contain a large, arbitrary set of keys, consider using th
     unless a user starts to add a huge number of fields with really long names. Default is
     `Long.MAX_VALUE` (no limit).
 
-[[index-mapping-dimension-fields-limit]]
-`index.mapping.dimension_fields.limit`::
-+
---
-experimental:[] (<<dynamic-index-settings,Dynamic>>, integer)
-
-.For internal use by Elastic only.
-[%collapsible]
-====
-Maximum number of time series dimensions for the index. Defaults to `16`.
-
-You can mark a field as a dimension using the `time_series_dimension` mapping
-parameter.
-====
---
+include::{es-repo-dir}/data-streams/tsds-index-settings.asciidoc[tag=dimensions-limit]

docs/reference/mapping/types/aggregate-metric-double.asciidoc

@@ -38,6 +38,7 @@ PUT my-index
 }
 ----
 
+[role="child_attributes"]
 [[aggregate-metric-double-params]]
 ==== Parameters for `aggregate_metric_double` fields
 
@@ -56,21 +57,23 @@ specify at least one value.
 Default metric sub-field to use for queries, scripts, and aggregations that
 don't use a sub-field. Must be a value from the `metrics` array.
 
+ifeval::["{release-state}"=="unreleased"]
 `time_series_metric`::
+preview:[] (Optional, string)
+include::numeric.asciidoc[tag=time_series_metric]
 +
---
-experimental:[]
-(Optional, string)
-
-.For internal use by Elastic only.
-[%collapsible]
+.Valid `time_series_metric` values for `aggregate_metric_double` fields
+[%collapsible%open]
 ====
-include::numeric.asciidoc[tag=time_series_metric]
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-counter]
+
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-gauge]
+
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-summary]
 
-For `aggregate_metric_double` fields, this parameter accepts `counter`, `gauge`,
-and `summary`. You can't update this parameter for existing fields.
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-null]
 ====
---
+endif::[]
 
 [[aggregate-metric-double-uses]]
 ==== Uses

docs/reference/mapping/types/histogram.asciidoc

@@ -23,24 +23,23 @@ per document. Nested arrays are not supported.
 * `histogram` fields do not support sorting.
 ========
 
+[role="child_attributes"]
 [[histogram-params]]
 ==== Parameters
 
+ifeval::["{release-state}"=="unreleased"]
 `time_series_metric`::
+preview:[] (Optional, string)
+include::numeric.asciidoc[tag=time_series_metric]
 +
---
-experimental:[]
-(Optional, string)
-
-.For internal use by Elastic only.
-[%collapsible]
+.Valid `time_series_metric` values for `histogram` fields
+[%collapsible%open]
 ====
-include::numeric.asciidoc[tag=time_series_metric]
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-histogram]
 
-For `histogram` fields, this parameter accepts `histogram`. You can't update
-this parameter for existing fields.
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-null]
 ====
---
+endif::[]
 
 [[histogram-uses]]
 ==== Uses

docs/reference/mapping/types/ip.asciidoc

@@ -95,16 +95,10 @@ The following parameters are accepted by `ip` fields:
     (default).
 
 `time_series_dimension`::
+preview:[] (Optional, Boolean)
 +
 --
-experimental:[]
-(Optional, Boolean)
-
-.For internal use by Elastic only.
-[%collapsible]
-====
 include::keyword.asciidoc[tag=dimension]
-====
 --
 
 [[query-ip-fields]]

docs/reference/mapping/types/keyword.asciidoc

@@ -149,19 +149,21 @@ The following parameters are accepted by `keyword` fields:
     Accepts `true` or `false` (default).
 
 `time_series_dimension`::
+preview:[] (Optional, Boolean)
 +
 --
-experimental:[]
-(Optional, Boolean)
-
-.For internal use by Elastic only.
-[%collapsible]
-====
 // tag::dimension[]
-Marks the field as a time series dimension. Defaults to `false`.
+ifeval::["{release-state}"=="unreleased"]
+Marks the field as a `time-series-dimension,time series dimension`. Defaults
+to `false`.
 
-The <<index-mapping-dimension-fields-limit,`index.mapping.dimension_fields.limit`>>
+The
+<<index-mapping-dimension-fields-limit,`index.mapping.dimension_fields.limit`>>
 index setting limits the number of dimensions in an index.
+endif::[]
+ifeval::["{release-state}"!="unreleased"]
+Marks the field as a time series dimension. Defaults to `false`.
+endif::[]
 
 Dimension fields have the following constraints:
 
@@ -170,7 +172,6 @@ Dimension fields have the following constraints:
 // end::dimension[]
 * Field values cannot be larger than 1024 bytes.
 * The field cannot use a <<normalizer,`normalizer`>>.
-====
 --
 
 ifeval::["{release-state}"=="unreleased"]

docs/reference/mapping/types/numeric.asciidoc

@@ -111,6 +111,7 @@ the data as both a `keyword` _and_ a numeric data type.
 // end::map-ids-as-keyword[]
 ====
 
+[role="child_attributes"]
 [[number-params]]
 ==== Parameters for numeric fields
 
@@ -178,44 +179,38 @@ The following parameters are accepted by numeric types:
     (default).
 
 `time_series_dimension`::
+preview:[] (Optional, Boolean)
 +
 --
-experimental:[]
-(Optional, Boolean)
-
-.For internal use by Elastic only.
-[%collapsible]
-====
 include::keyword.asciidoc[tag=dimension]
 
 Of the numeric field types, only `byte`, `short`, `integer`, `long`, and
 `unsigned_long` fields support this parameter.
 
 A numeric field can't be both a time series dimension and a time series metric.
-====
 --
 
+ifeval::["{release-state}"=="unreleased"]
 `time_series_metric`::
-+
---
-experimental:[]
-(Optional, string)
-
-.For internal use by Elastic only.
-[%collapsible]
-====
+preview:[] (Optional, string)
 // tag::time_series_metric[]
-Marks the field as a time series metric. The value is the metric type. Defaults
-to `null` (Not a time series metric).
+Marks the field as a <<time-series-metric,time series metric>>. The value is the
+metric type. You can't update this parameter for existing fields.
 // end::time_series_metric[]
++
+.Valid `time_series_metric` values for numeric fields
+[%collapsible%open]
+====
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-counter]
 
-For numeric fields, this parameter accepts `gauge` and `counter`. You can't
-update this parameter for existing fields.
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-gauge]
 
+include::{es-repo-dir}/data-streams/tsds.asciidoc[tag=time-series-metric-null]
+====
++
 For a numeric time series metric, the `doc_values` parameter must be `true`. A
 numeric field can't be both a time series dimension and a time series metric.
-====
---
+endif::[]
 
 [[scaled-float-params]]
 ==== Parameters for `scaled_float`

docs/reference/search/field-caps.asciidoc

@@ -136,11 +136,11 @@ field types are all described as the `keyword` type family.
   Whether this field can be aggregated on all indices.
 
 `time_series_dimension`::
-  experimental:[]
+  preview:[]
   Whether this field is used as a time series dimension.
 
 `time_series_metric`::
-  experimental:[]
+  preview:[]
   Contains metric type if this fields is used as a time series metrics, absent if the field is not used as metric.
 
 `indices`::

docs/reference/settings/ilm-settings.asciidoc

@@ -21,6 +21,7 @@ Whether ILM's history index is enabled. If enabled, ILM will record the
 history of actions taken as part of ILM policies to the `ilm-history-*`
 indices. Defaults to `true`.
 
+[[indices-lifecycle-poll-interval]]
 `indices.lifecycle.poll_interval`::
 (<<dynamic-cluster-setting,Dynamic>>, <<time-units, time unit value>>) 
 How often {ilm} checks for indices that meet policy criteria. Defaults to `10m`.