elasticsearch/docs/reference/data-streams.asciidoc

[chapter]
[[data-streams]]
= Data Streams

You can use data streams to index time-based data that's continuously generated.
A data stream groups indices from the same time-based data source.
A data stream tracks its indices, known as _backing indices_, using an ordered
list.

A data stream's backing indices are <<index-hidden,hidden>>.
While all backing indices handle read requests, the most recently created
backing index is the data stream's only write index.  A data stream only
accepts <<docs-index_,index requests>> with `op_type` set to `create`. To update
or delete specific documents in a data stream, submit a <<docs-delete,delete>>
or <<docs-update,update>> API request to the backing index containing the
document.

To create a data stream, set up a <<indices-templates,composable index
template>> containing:

* A name or wildcard pattern for the data stream in the `index_patterns` property.
* A `data_stream` definition that contains the `timestamp_field` property.
  The `timestamp_field` must be the primary timestamp field 
   for the data source. This field must be included in every
   document indexed to the data stream.

When you index one or more documents to a not-yet-existent target matching
the template's name or pattern, {es} automatically creates the corresponding
data stream. You can also manually create a data stream using the
<<indices-create-data-stream,create data stream API>>. However, a composable
template for the stream is still required.

You can use the <<indices-rollover-index,rollover API>> to roll a data stream
over to a new index when the current write index meets specified criteria, such
as a maximum age or size. A rollover creates a new backing index and updates the
data stream's list of backing indices. This new index then becomes the stream's
new write index. See <<rollover-data-stream-ex>>.

[discrete]
[[create-data-stream]]
== Create a data stream

Create a composable template with a `data_stream` definition:

[source,console]
-----------------------------------
PUT /_index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "data_stream": {
    "timestamp_field": "@timestamp"
  }
}
-----------------------------------

Start indexing data to a target matching composable template's wildcard pattern:

[source,console]
----
POST /logs-foobar/_doc
{
  "@timestamp": "2050-11-15T14:12:12",
  ...
}
----
// TEST[continued]
// TEST[s/,//]
// TEST[s/\.\.\.//]

Response:

[source,console-result]
--------------------------------------------------
{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 1
    },
    "_index" : "logs-foobar-000001",
    "_id" : "W0tpsmIBdwcYyG50zbta",
    "_version" : 1,
    "_seq_no" : 0,
    "_primary_term" : 1,
    "result": "created"
}
--------------------------------------------------
// TESTRESPONSE[s/W0tpsmIBdwcYyG50zbta/$body._id/]

Or create a data stream using the create data stream API:

[source,console]
--------------------------------------------------
PUT /_data_stream/logs-barbaz
--------------------------------------------------
// TEST[continued]

////
[source,console]
-----------------------------------
DELETE /_data_stream/logs-foobar
DELETE /_data_stream/logs-barbaz
DELETE /_index_template/logs_template
-----------------------------------
// TEST[continued]
////

[discrete]
[[data-streams-apis]]
== Data stream APIs

The following APIs are available for managing data streams:

* To get information about data streams, use the <<indices-get-data-stream, get data stream API>>.
* To delete data streams, use the <<indices-delete-data-stream, delete data stream API>>.
* To manually create a data stream, use the <<indices-create-data-stream, create data stream API>>.