elasticsearch/docs/reference/rollup/apis/rollup-search.asciidoc

[role="xpack"]
[[rollup-search]]
=== Rollup search
++++
<titleabbrev>Rollup search</titleabbrev>
++++

Enables searching rolled-up data using the standard Query DSL.

experimental[]

NOTE: For version 8.5 and above we recommend <<downsampling,downsampling>> over
rollups as a way to reduce your storage costs for time series data.

[[rollup-search-request]]
==== {api-request-title}

`GET <target>/_rollup_search`

[[rollup-search-desc]]
==== {api-description-title}

The rollup search endpoint is needed because, internally, rolled-up documents
utilize a different document structure than the original data. The rollup search
endpoint rewrites standard Query DSL into a format that matches the rollup
documents, then takes the response and rewrites it back to what a client would
expect given the original query.

[[rollup-search-path-params]]
==== {api-path-parms-title}

`<target>`::
+
--
(Required, string)
Comma-separated list of data streams and indices used to limit
the request. Wildcard expressions (`*`) are supported.

This target can include both rollup and non-rollup indices.

Rules for the `<target>` parameter:

- At least one data stream, index, or wildcard expression must be specified.
This target can include a rollup or non-rollup index. For data streams, the
stream's backing indices can only serve as non-rollup indices. Omitting the
`<target>` parameter or using `_all` is not permitted.
- Multiple non-rollup indices may be specified.
- Only one rollup index may be specified. If more than one are supplied, an
exception occurs.
- Wildcard expressions may be used, but, if they match more than one rollup index, an
exception occurs. However, you can use an expression to match multiple non-rollup
indices or data streams.
--

[[rollup-search-request-body]]
==== {api-request-body-title}

The request body supports a subset of features from the regular Search API. It
supports:

- `query` param for specifying an DSL query, subject to some limitations
(see <<rollup-search-limitations>> and <<rollup-agg-limitations>>
- `aggregations` param for specifying aggregations

Functionality that is not available:

- `size`: Because rollups work on pre-aggregated data, no search hits can be
returned and so size must be set to zero or omitted entirely.
- `highlighter`, `suggestors`, `post_filter`, `profile`, `explain`: These are
similarly disallowed.

[[rollup-search-example]]
==== {api-examples-title}

===== Historical-only search example

Imagine we have an index named `sensor-1` full of raw data, and we have created
a {rollup-job} with the following configuration:

[source,console]
--------------------------------------------------
PUT _rollup/job/sensor
{
  "index_pattern": "sensor-*",
  "rollup_index": "sensor_rollup",
  "cron": "*/30 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "fixed_interval": "1h",
      "delay": "7d"
    },
    "terms": {
      "fields": [ "node" ]
    }
  },
  "metrics": [
    {
      "field": "temperature",
      "metrics": [ "min", "max", "sum" ]
    },
    {
      "field": "voltage",
      "metrics": [ "avg" ]
    }
  ]
}
--------------------------------------------------
// TEST[setup:sensor_index]

This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`.
To search this rolled up data, we need to use the `_rollup_search` endpoint.
However, you'll notice that we can use regular Query DSL to search the rolled-up
data:

[source,console]
--------------------------------------------------
GET /sensor_rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "max_temperature": {
      "max": {
        "field": "temperature"
      }
    }
  }
}
--------------------------------------------------
// TEST[setup:sensor_prefab_data]
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]

The query is targeting the `sensor_rollup` data, since this contains the rollup
data as configured in the job. A `max` aggregation has been used on the
`temperature` field, yielding the following response:

[source,console-result]
----
{
  "took" : 102,
  "timed_out" : false,
  "terminated_early" : false,
  "_shards" : ... ,
  "hits" : {
    "total" : {
        "value": 0,
        "relation": "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "max_temperature" : {
      "value" : 202.0
    }
  }
}
----
// TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/]
// TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/]

The response is exactly as you'd expect from a regular query + aggregation; it
provides some metadata about the request (`took`, `_shards`, etc), the search
hits (which is always empty for rollup searches), and the aggregation response.

Rollup searches are limited to functionality that was configured in the
{rollup-job}. For example, we are not able to calculate the average temperature
because `avg` was not one of the configured metrics for the `temperature` field.
If we try to execute that search:

[source,console]
--------------------------------------------------
GET sensor_rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "avg_temperature": {
      "avg": {
        "field": "temperature"
      }
    }
  }
}
--------------------------------------------------
// TEST[continued]
// TEST[catch:/illegal_argument_exception/]

[source,console-result]
----
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
        "stack_trace": ...
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
    "stack_trace": ...
  },
  "status": 400
}
----
// TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/]

===== Searching both historical rollup and non-rollup data

The rollup search API has the capability to search across both "live"
non-rollup data and the aggregated rollup data. This is done by simply adding
the live indices to the URI:

[source,console]
--------------------------------------------------
GET sensor-1,sensor_rollup/_rollup_search <1>
{
  "size": 0,
  "aggregations": {
    "max_temperature": {
      "max": {
        "field": "temperature"
      }
    }
  }
}
--------------------------------------------------
// TEST[continued]
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]
<1> Note the URI now searches `sensor-1` and `sensor_rollup` at the same time

When the search is executed, the rollup search endpoint does two things:

1. The original request is sent to the non-rollup index unaltered.
2. A rewritten version of the original request is sent to the rollup index.

When the two responses are received, the endpoint rewrites the rollup response
and merges the two together. During the merging process, if there is any overlap
in buckets between the two responses, the buckets from the non-rollup index are
used.

The response to the above query looks as expected, despite spanning rollup and
non-rollup indices:

[source,console-result]
----
{
  "took" : 102,
  "timed_out" : false,
  "terminated_early" : false,
  "_shards" : ... ,
  "hits" : {
    "total" : {
        "value": 0,
        "relation": "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "max_temperature" : {
      "value" : 202.0
    }
  }
}
----
// TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/]
// TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/]