elasticsearch/docs/reference/data-streams/set-up-tsds.asciidoc

[[set-up-tsds]]
=== Set up a time series data stream (TSDS)
++++
<titleabbrev>Set up a TSDS</titleabbrev>
++++

To set up a <<tsds,time series data stream (TSDS)>>, follow these steps:

. Check the <<tsds-prereqs,prerequisites>>.
. <<tsds-ilm-policy>>.
. <<create-tsds-index-template>>.
. <<create-tsds>>.
. <<secure-tsds>>.

[discrete]
[[tsds-prereqs]]
==== Prerequisites

* Before you create a TSDS, you should be familiar with <<data-streams,data
streams>> and <<tsds,TSDS concepts>>.

* To follow this tutorial, you must have the following permissions:

** <<privileges-list-cluster,Cluster privileges>>: `manage_ilm` and
`manage_index_templates`.
** <<privileges-list-indices,Index privileges>>: `create_doc` and `create_index`
for any TSDS you create or convert. To roll over a TSDS, you must have the
`manage` privilege.

[discrete]
[[tsds-ilm-policy]]
==== Create an index lifecycle policy

While optional, we recommend using {ilm-init} to automate the management of your
TSDS's backing indices. {ilm-init} requires an index lifecycle policy.

We recommend you specify a `max_age` criteria for the `rollover` action in the
policy. This ensures the <<time-bound-indices,`@timestamp` ranges>> for the
TSDS's backing indices are consistent. For example, setting a `max_age` of `1d`
for the `rollover` action ensures your backing indices consistently contain one
day's worth of data.

////
[source,console]
----
PUT /_snapshot/found-snapshots
{
 "type": "fs",
  "settings": {
    "location": "my_backup_location"
  }
}
----
// TESTSETUP
////

[source,console]
----
PUT _ilm/policy/my-weather-sensor-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----

[discrete]
[[create-tsds-index-template]]
==== Create an index template

To setup a TSDS create an index template with the following details:

* One or more index patterns that match the TSDS's name. We recommend
using our {fleet-guide}/data-streams.html#data-streams-naming-scheme[data stream
naming scheme].

* Enable data streams.

* Specify a mapping that defines your dimensions and metrics:

** One or more <<time-series-dimension,dimension fields>> with a `time_series_dimension` value of `true`.
   At least one of these dimensions must be a plain `keyword` field.

** One or more <<time-series-metric,metric fields>>, marked using the `time_series_metric` mapping parameter.

** Optional: A `date` or `date_nanos` mapping for the `@timestamp` field. If you don’t specify a mapping,
   Elasticsearch maps `@timestamp` as a `date` field with default options.

* Define index settings:

** Set `index.mode` setting to `time_series`.

** Your lifecycle policy in the `index.lifecycle.name` index setting.

** Optional: Other index settings, such as <<dynamic-index-number-of-replicas,`index.number_of_replicas`>>,
   for your TSDS's backing indices.

* A priority higher than `200` to avoid collisions with built-in templates.
See <<avoid-index-pattern-collisions>>.

* Optional: Component templates containing your mappings and other index settings.

[source,console]
----
PUT _index_template/my-weather-sensor-index-template
{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy"
    },
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date"
        }
      }
    }
  },
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}
----
// TEST[continued]

////
[source,console]
----
DELETE _data_stream/*
DELETE _index_template/*
DELETE _ilm/policy/my-weather-sensor-lifecycle-policy
----
// TEST[continued]
////

[discrete]
[[create-tsds]]
==== Create the TSDS

<<add-documents-to-a-data-stream,Indexing requests>> add documents to a TSDS.
Documents in a TSDS must include:

* A `@timestamp` field
* One or more dimension fields. At least one dimension must be a `keyword` field
that matches the `index.routing_path` index setting, if specified. If not specified
explicitly, `index.routing_path` is set automatically to whichever mappings have
 `time_series_dimension` set to `true`.

To automatically create your TSDS, submit an indexing request that
targets the TSDS's name. This name must match one of your index template's
index patterns.

IMPORTANT: To test the following example, update the timestamps to within three hours of
your current time. Data added to a TSDS must always fall within an
<<tsds-accepted-time-range,accepted time range>>.

[source,console]
----
PUT metrics-weather_sensors-dev/_bulk
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }

POST metrics-weather_sensors-dev/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SYKENET-000001",
  "location": "swamp",
  "temperature": 32.4,
  "humidity": 88.9
}
----
// TEST[skip: The @timestamp value won't match an accepted range in the TSDS]

You can also manually create the TSDS using the
<<indices-create-data-stream,create data stream API>>. The TSDS's name must
still match one of your template's index patterns.

[source,console]
----
PUT _data_stream/metrics-weather_sensors-dev
----
// TEST[setup:tsds_template]
// TEST[teardown:tsds_cleanup]

[discrete]
[[secure-tsds]]
==== Secure the TSDS

Use <<privileges-list-indices,index privileges>> to control access to a TSDS.
Granting privileges on a TSDS grants the same privileges on its backing indices.

For an example, refer to <<data-stream-privileges>>.

[discrete]
[[convert-existing-data-stream-to-tsds]]
==== Convert an existing data stream to a TSDS

You can also use the above steps to convert an existing regular data stream to
a TSDS. In this case, you'll want to:

* Edit your existing index lifecycle policy, component templates, and index
templates instead of creating new ones.

* Instead of creating the TSDS, manually roll over its write index. This ensures
the current write index and any new backing indices have an
<<time-series-mode,`index.mode` of `time_series`>>.
+
You can manually roll over the write index using the
<<indices-rollover-index,rollover API>>.
+
[source,console]
----
POST metrics-weather_sensors-dev/_rollover
----
// TEST[setup:tsds]
// TEST[teardown:tsds_cleanup]

[discrete]
[[set-up-component-templates]]
==== A note about component templates and index.mode setting

Configuring a TSDS via an index template that uses component templates is a bit more complicated.
Typically with component templates mappings and settings get scattered across multiple component templates.
When configuring the `index.mode` setting in a component template, the `index.routing_path` setting needs to
be defined in the same component template. Additionally the fields mentioned in the `index.routing_path`
also need to be defined in the same component template with the `time_series_dimension` attribute enabled.

The reasons for this is that each component template needs to be valid on its own and the time series index mode
requires the `index.routing_path` setting. When configuring the  `index.mode` setting in an index template, the `index.routing_path` setting is configured automatically. It is derived  from
the field mappings with `time_series_dimension` attribute enabled.

[discrete]
[[set-up-tsds-whats-next]]
==== What's next?

Now that you've set up your TSDS, you can manage and use it like a regular
data stream. For more information, refer to:

* <<use-a-data-stream>>
* <<data-streams-change-mappings-and-settings>>
* <<data-stream-apis,data stream APIs>>