elasticsearch/docs/reference/data-streams/data-streams.asciidoc

[[data-streams]]
= Data streams

[partintro]
--
You can use data streams to index time-based data that's continuously generated.
A data stream groups indices from the same time-based data source.
A data stream tracks its indices, known as _backing indices_, using an ordered
list.

A data stream's backing indices are <<index-hidden,hidden>>.
While all backing indices handle read requests, the most recently created
backing index is the data stream's only write index.  A data stream only
accepts <<docs-index_,index requests>> with `op_type` set to `create`. To update
or delete specific documents in a data stream, submit a <<docs-delete,delete>>
or <<docs-update,update>> API request to the backing index containing the
document.

To create a data stream, set up a <<indices-templates,composable index
template>> containing:

* A name or wildcard pattern for the data stream in the `index_patterns` property.
* A `data_stream` definition that contains the `timestamp_field` property.
  The `timestamp_field` must be the primary timestamp field
   for the data source. This field must be included in every
   document indexed to the data stream.

When you index one or more documents to a not-yet-existent target matching
the template's name or pattern, {es} automatically creates the corresponding
data stream. You can also manually create a data stream using the
<<indices-create-data-stream,create data stream API>>. However, a composable
template for the stream is still required.

You can use the <<indices-rollover-index,rollover API>> to roll a data stream
over to a new index when the current write index meets specified criteria, such
as a maximum age or size. A rollover creates a new backing index and updates the
data stream's list of backing indices. This new index then becomes the stream's
new write index. See <<rollover-data-stream-ex>>.

Backing indices are generated with the naming convention
`.ds-<data-stream-name>-zzzzzz`, where `zzzzzz` is the six-digit, zero-padded
generation of the data stream. For example, a data stream named
`web-server-logs` with a generation of 34 would have a write index named
`.ds-web-server-logs-000034`. Data streams may have backing indices that do not
conform to this naming convention if operations such as
<<indices-shrink-index,shrink>> have been performed on them.

[discrete]
[[data-streams-apis]]
== Data stream APIs

The following APIs are available for managing data streams:

* To get information about data streams, use the <<indices-get-data-stream, get data stream API>>.
* To delete data streams, use the <<indices-delete-data-stream, delete data stream API>>.
* To manually create a data stream, use the <<indices-create-data-stream, create data stream API>>.

[discrete]
[[data-streams-toc]]
== In this section

* <<set-up-a-data-stream>>
* <<use-a-data-stream>>
--

include::set-up-a-data-stream.asciidoc[]
include::use-a-data-stream.asciidoc[]