elasticsearch/docs/reference/data-streams/set-up-a-data-stream.asciidoc

[role="xpack"]
[[set-up-a-data-stream]]
== Set up a data stream

To set up a data stream, follow these steps:

. Check the <<data-stream-prereqs, prerequisites>>.
. <<configure-a-data-stream-ilm-policy>>.
. <<create-a-data-stream-template>>.
. <<create-a-data-stream>>.
. <<get-info-about-a-data-stream>> to verify it exists.
. <<secure-a-data-stream>>.

After you set up a data stream, you can <<use-a-data-stream, use the data
stream>> for indexing, searches, and other supported operations.

If you no longer need it, you can <<delete-a-data-stream,delete a data stream>>
and its backing indices.

[discrete]
[[data-stream-prereqs]]
=== Prerequisites

* {es} data streams are intended for time series data only. Each document
indexed to a data stream must contain the `@timestamp` field. This field must be
mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field data type.

* Data streams are best suited for time-based,
<<data-streams-append-only,append-only>> use cases. If you frequently need to
update or delete existing documents, we recommend using an index alias and an
index template instead.


[discrete]
[[configure-a-data-stream-ilm-policy]]
=== Optional: Configure an {ilm-init} lifecycle policy for a data stream

You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to automatically
manage a data stream's backing indices. For example, you could use {ilm-init}
to:

* Spin up a new write index for the data stream when the current one reaches a
  certain size or age.
* Move older backing indices to slower, less expensive hardware.
* Delete stale backing indices to enforce data retention standards.

To use {ilm-init} with a data stream, you must
<<set-up-lifecycle-policy,configure a lifecycle policy>>. This lifecycle policy
should contain the automated actions to take on backing indices and the
triggers for such actions.

TIP: While optional, we recommend using {ilm-init} to manage the backing indices
associated with a data stream.

You can create the policy through the Kibana UI. In Kibana, open the menu and go
to *Stack Management > Index Lifecycle Policies*. Click *Index Lifecycle
Policies*.

[role="screenshot"]
image::images/ilm/create-policy.png[Index Lifecycle Policies page]

You can also create a policy using the <<ilm-put-lifecycle,create lifecycle
policy API>>.

The following request configures the `my-data-stream-policy` lifecycle policy.
The policy uses the <<ilm-rollover,`rollover` action>> to create a
new <<data-stream-write-index,write index>> for the data stream when the current
one reaches 25GB in size. The policy also deletes backing indices 30 days after
their rollover.

[source,console]
----
PUT /_ilm/policy/my-data-stream-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "25GB"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
----


[discrete]
[[create-a-data-stream-template]]
=== Create an index template for a data stream

A data stream uses an index template to configure its backing indices. A
template for a data stream must specify:

* One or more index patterns that match the name of the stream.

* The mappings and settings for the stream's backing indices.

* That the template is used exclusively for data streams.

* A priority for the template.

[IMPORTANT]
====
{es} has built-in index templates for the `metrics-*-*` and `logs-*-*` index
patterns. {ingest-guide}/ingest-management-overview.html[{agent}] uses these
templates to create data streams. If you use {agent}, assign your index
templates a priority lower than `100` to avoid an override of the built-in
templates.

Otherwise, to avoid accidentally applying the built-in templates, use a
non-overlapping index pattern, or assign your templates a `priority` higher or
lower than `100`.
====

Every document indexed to a data stream must have a `@timestamp` field. This
field can be mapped as a <<date,`date`>> or <<date_nanos,`date_nanos`>> field
data type by the stream's index template. This mapping can include other
<<mapping-params,mapping parameters>>, such as <<mapping-date-format,`format`>>.
If the template does not specify a mapping, the `@timestamp` field is mapped as
a `date` field  with default options.

We recommend using {ilm-init} to manage a data stream's backing indices. Specify
the name of the lifecycle policy with the `index.lifecycle.name` setting.

TIP: We recommend you carefully consider which mappings and settings to include
in this template before creating a data stream. Later changes to the mappings or
settings of a stream's backing indices may require reindexing. See
<<data-streams-change-mappings-and-settings>>.

You can create an index template through the Kibana UI:

. From Kibana, open the menu and go to *Stack Management > Index Management*.
. In the *Index Templates* tab, click *Create template*.
. In the Create template wizard, use the *Data stream* toggle to indicate the
template is used exclusively for data streams.

[role="screenshot"]
image::images/data-streams/create-index-template.png[Create template page]

You can also create a template using the <<indices-put-template,put index
template API>>. The template must include a `data_stream` object with an empty
body (`{ }`). This object indicates the template is used exclusively for data
streams.

The following request configures the `my-data-stream-template` index template.
Because no field mapping is specified, the `@timestamp` field uses the `date`
field data type by default.

[source,console]
----
PUT /_index_template/my-data-stream-template
{
  "index_patterns": [ "my-data-stream*" ],
  "data_stream": { },
  "priority": 200,
  "template": {
    "settings": {
      "index.lifecycle.name": "my-data-stream-policy"
    }
  }
}
----
// TEST[continued]

Alternatively, the following template maps `@timestamp` as a `date_nanos` field.

[source,console]
----
PUT /_index_template/my-data-stream-template
{
  "index_patterns": [ "my-data-stream*" ],
  "data_stream": { },
  "priority": 200,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date_nanos" }    <1>
      }
    },
    "settings": {
      "index.lifecycle.name": "my-data-stream-policy"
    }
  }
}
----
// TEST[continued]

<1> Maps `@timestamp` as a `date_nanos` field. You can include other supported
mapping parameters in this field mapping.

NOTE: You cannot delete an index template that's in use by a data stream.
This would prevent the data stream from creating new backing indices.

[discrete]
[[create-a-data-stream]]
=== Create a data stream

You can create a data stream using one of two methods:

* <<index-documents-to-create-a-data-stream>>
* <<manually-create-a-data-stream>>

[discrete]
[[index-documents-to-create-a-data-stream]]
====  Index documents to create a data stream

You can automatically create a data stream using an indexing request. Submit 
an <<add-documents-to-a-data-stream,indexing request>> to a target
matching the index pattern defined in the template's `index_patterns`
property.

If the indexing request's target doesn't exist, {es} creates the data stream and
uses the target name as the name for the stream.

NOTE: Data streams support only specific types of indexing requests. See
<<add-documents-to-a-data-stream>>.

The following <<docs-index_,index API>> request targets `my-data-stream`, which
matches the index pattern for `my-data-stream-template`. Because
no existing index or data stream uses this name, this request creates the
`my-data-stream` data stream and indexes the document to it.

[source,console]
----
POST /my-data-stream/_doc/
{
  "@timestamp": "2020-12-06T11:04:05.000Z",
  "user": {
    "id": "vlb44hny"
  },
  "message": "Login attempt failed"
}
----
// TEST[continued]

The API returns the following response. Note the `_index` property contains
`.ds-my-data-stream-000001`, indicating the document was indexed to the write
index of the new data stream.

[source,console-result]
----
{
  "_index": ".ds-my-data-stream-000001",
  "_id": "qecQmXIBT4jB8tq1nG0j",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}
----
// TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/]

[discrete]
[[manually-create-a-data-stream]]
====  Manually create a data stream

You can use the <<indices-create-data-stream,create data stream API>> to
manually create a data stream. The name of the data stream must match the index
pattern defined in the template's `index_patterns` property.

The following create data stream request targets `my-data-stream-alt`, which
matches the index pattern for `my-data-stream-template`. Because
no existing index or data stream uses this name, this request creates the
`my-data-stream-alt` data stream.

[source,console]
----
PUT /_data_stream/my-data-stream-alt
----
// TEST[continued]

[discrete]
[[get-info-about-a-data-stream]]
=== Get information about a data stream

To view information about a data stream in Kibana, open the menu and go to
*Stack Management > Index Management*. In the *Data Streams* tab, click a data
stream's name to view information about the stream.

[role="screenshot"]
image::images/data-streams/data-streams-list.png[Data Streams tab]

You can also use the <<indices-get-data-stream,get data stream API>> to retrieve
the following information about one or more data streams:

* The current backing indices, which is returned as an array. The last item in
  the array contains information about the stream's current write index.
* The current generation
* The data stream's health status
* The index template used to create the stream's backing indices
* The current {ilm-init} lifecycle policy in the stream's matching index
template

The following get data stream API request retrieves information about
`my-data-stream`.

////
[source,console]
----
POST /my-data-stream/_rollover/
----
// TEST[continued]
////

[source,console]
----
GET /_data_stream/my-data-stream
----
// TEST[continued]

The API returns the following response. Note the `indices` property contains an
array of the stream's current backing indices. The last item in this array
contains information about the stream's write index, `.ds-my-data-stream-000002`.

[source,console-result]
----
{
  "data_streams": [
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-my-data-stream-000001",
          "index_uuid": "krR78LfvTOe6gr5dj2_1xQ"
        },
        {
          "index_name": ".ds-my-data-stream-000002",        <1>
          "index_uuid": "C6LWyNJHQWmA08aQGvqRkA"
        }
      ],
      "generation": 2,
      "status": "GREEN",
      "template": "my-data-stream-template",
      "ilm_policy": "my-data-stream-policy"
    }
  ]
}
----
// TESTRESPONSE[s/"index_uuid": "krR78LfvTOe6gr5dj2_1xQ"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
// TESTRESPONSE[s/"index_uuid": "C6LWyNJHQWmA08aQGvqRkA"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW"/]

<1> Last item in the `indices` array for `my-data-stream`. This
item contains information about the stream's current write index,
`.ds-my-data-stream-000002`.

[discrete]
[[secure-a-data-stream]]
=== Secure a data stream

You can use {es} {security-features} to control access to a data stream and its
data. See <<data-stream-privileges>>.

[discrete]
[[delete-a-data-stream]]
=== Delete a data stream

You can use the Kibana UI to delete a data stream and its backing indices. In
Kibana, open the menu and go to *Stack Management > Index Management*. In the
*Data Streams* tab, click the trash can icon to delete a stream and its backing
indices.

[role="screenshot"]
image::images/data-streams/data-streams-list.png[Data Streams tab]

You can also use the the <<indices-delete-data-stream,delete data stream API>>
to delete a data stream. The following delete data stream API request deletes
`my-data-stream`. This request also deletes the stream's backing
indices and any data they contain.

[source,console]
----
DELETE /_data_stream/my-data-stream
----
// TEST[continued]

////
[source,console]
----
DELETE /_data_stream/*
DELETE /_index_template/*
DELETE /_ilm/policy/my-data-stream-policy
----
// TEST[continued]
////