elasticsearch/docs/reference/indices/downsample-data-stream.asciidoc

[role="xpack"]
[[indices-downsample-data-stream]]
=== Downsample index API
++++
<titleabbrev>Downsample</titleabbrev>
++++

preview::[]

Aggregates a time series (TSDS) index and stores
pre-computed statistical summaries (`min`, `max`, `sum`, `value_count` and
`avg`) for each metric field grouped by a configured time interval. For example,
a TSDS index that contains metrics sampled every 10 seconds can be downsampled
to an hourly index. All documents within an hour interval are summarized and
stored as a single document in the downsample index.

////
[source,console]
----
PUT /my-time-series-index
{
    "settings": {
        "index": {
            "mode": "time_series",
            "time_series": {
                "start_time": "2022-06-10T00:00:00Z",
                "end_time": "2022-06-30T23:59:59Z"
            },
            "routing_path": [
                "test.namespace"
            ],
            "number_of_replicas": 0,
            "number_of_shards": 2
        }
    },
    "mappings": {
        "properties": {
            "@timestamp": {
                "type": "date"
            },
            "metric": {
                "type": "long",
                "time_series_metric": "gauge"
            },
            "dimension": {
                "type": "keyword",
                "time_series_dimension": true
            }
        }
    }
}

PUT /my-time-series-index/_block/write

----
// TEST
////

[source,console]
----
POST /my-time-series-index/_downsample/my-downsampled-time-series-index
{
    "fixed_interval": "1d"
}
----
// TEST[continued]

////
[source,console]
----
DELETE /my-time-series-index*
DELETE _data_stream/*
DELETE _index_template/*
----
// TEST[continued]
////

[[downsample-api-request]]
==== {api-request-title}

`POST /<source-index>/_downsample/<output-downsampled-index>`

[[downsample-api-prereqs]]
==== {api-prereq-title}

* Only indices in a <<tsds,time series data stream>> are supported.

* If the {es} {security-features} are enabled, you must have the `all`
or `manage` <<privileges-list-indices,index privilege>> for the data stream.

* Neither <<field-and-document-access-control,field nor document level security>> can be defined on the source index.

* The source index must be read only (`index.blocks.write: true`).

[[downsample-api-path-params]]
==== {api-path-parms-title}

`<source-index>`::
(Optional, string) Name of the time series index to downsample.

`<output-downsampled_index>`::
+
--
(Required, string) Name of the index to create. 

include::{es-repo-dir}/indices/create-index.asciidoc[tag=index-name-reqs]
--

[role="child_attributes"]
[[downsample-api-query-parms]]
==== {api-query-parms-title}

`fixed_interval`:: (Required, <<time-units,time units>>) The interval at which
to aggregate the original time series index. For example, `60m` produces a
document for each 60 minute (hourly) interval. This follows standard time
formatting syntax as used elsewhere in {es}.
+
NOTE: Smaller, more granular intervals take up proportionally more space.

[[downsample-api-process]]
==== The downsampling process

The downsampling operation traverses the source TSDS index and performs the
following steps:

. Creates a new document for each value of the `_tsid` field and each
`@timestamp` value, rounded to the `fixed_interval` defined in the downsample
configuration.
. For each new document, copies all <<time-series-dimension,time
series dimensions>> from the source index to the target index. Dimensions in a
TSDS are constant, so this is done only once per bucket.
. For each <<time-series-metric,time series metric>> field, computes aggregations
for all documents in the bucket. Depending on the metric type of each metric
field a different set of pre-aggregated results is stored:

** `gauge`: The `min`, `max`, `sum`, and `value_count` are stored; `value_count`
is stored as type `aggregate_metric_double`.
** `counter`: The `last_value` is stored.
. For all other fields, the most recent value is copied to the target index.

[[downsample-api-mappings]]
==== Source and target index field mappings

Fields in the target, downsampled index are created based on fields in the
original source index, as follows:

. All fields mapped with the `time-series-dimension` parameter are created in
the target downsample index with the same mapping as in the source index.
. All fields mapped with the `time_series_metric` parameter are created
in the target downsample index with the same mapping as in the source
index. An exception is that for fields mapped as `time_series_metric: gauge`
the field type is changed to `aggregate_metric_double`.
. All other fields that are neither dimensions nor metrics (that is, label
fields), are created in the target downsample index with the same mapping
that they had in the source index.

Check the <<downsampling,Downsampling>> documentation for an overview and
examples of running downsampling manually and as part of an ILM policy.