boxplot-aggregation.asciidoc 5.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[search-aggregations-metrics-boxplot-aggregation]]
  4. === Boxplot aggregation
  5. ++++
  6. <titleabbrev>Boxplot</titleabbrev>
  7. ++++
  8. A `boxplot` metrics aggregation that computes boxplot of numeric values extracted from the aggregated documents.
  9. These values can be generated by a provided script or extracted from specific numeric or
  10. <<histogram,histogram fields>> in the documents.
  11. The `boxplot` aggregation returns essential information for making a {wikipedia}/Box_plot[box plot]: minimum, maximum,
  12. median, first quartile (25th percentile) and third quartile (75th percentile) values.
  13. ==== Syntax
  14. A `boxplot` aggregation looks like this in isolation:
  15. [source,js]
  16. --------------------------------------------------
  17. {
  18. "boxplot": {
  19. "field": "load_time"
  20. }
  21. }
  22. --------------------------------------------------
  23. // NOTCONSOLE
  24. Let's look at a boxplot representing load time:
  25. [source,console]
  26. --------------------------------------------------
  27. GET latency/_search
  28. {
  29. "size": 0,
  30. "aggs": {
  31. "load_time_boxplot": {
  32. "boxplot": {
  33. "field": "load_time" <1>
  34. }
  35. }
  36. }
  37. }
  38. --------------------------------------------------
  39. // TEST[setup:latency]
  40. <1> The field `load_time` must be a numeric field
  41. The response will look like this:
  42. [source,console-result]
  43. --------------------------------------------------
  44. {
  45. ...
  46. "aggregations": {
  47. "load_time_boxplot": {
  48. "min": 0.0,
  49. "max": 990.0,
  50. "q1": 165.0,
  51. "q2": 445.0,
  52. "q3": 725.0,
  53. "lower": 0.0,
  54. "upper": 990.0
  55. }
  56. }
  57. }
  58. --------------------------------------------------
  59. // TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
  60. In this case, the lower and upper whisker values are equal to the min and max. In general, these values are the 1.5 *
  61. IQR range, which is to say the nearest values to `q1 - (1.5 * IQR)` and `q3 + (1.5 * IQR)`. Since this is an approximation, the given values
  62. may not actually be observed values from the data, but should be within a reasonable error bound of them. While the Boxplot aggregation
  63. doesn't directly return outlier points, you can check if `lower > min` or `upper < max` to see if outliers exist on either side, and then
  64. query for them directly.
  65. ==== Script
  66. The boxplot metric supports scripting. For example, if our load times
  67. are in milliseconds but we want values calculated in seconds, we could use
  68. a script to convert them on-the-fly:
  69. [source,console]
  70. --------------------------------------------------
  71. GET latency/_search
  72. {
  73. "size": 0,
  74. "aggs": {
  75. "load_time_boxplot": {
  76. "boxplot": {
  77. "script": {
  78. "lang": "painless",
  79. "source": "doc['load_time'].value / params.timeUnit", <1>
  80. "params": {
  81. "timeUnit": 1000 <2>
  82. }
  83. }
  84. }
  85. }
  86. }
  87. }
  88. --------------------------------------------------
  89. // TEST[setup:latency]
  90. <1> The `field` parameter is replaced with a `script` parameter, which uses the
  91. script to generate values which percentiles are calculated on
  92. <2> Scripting supports parameterized input just like any other script
  93. This will interpret the `script` parameter as an `inline` script with the `painless` script language and no script parameters. To use a
  94. stored script use the following syntax:
  95. [source,console]
  96. --------------------------------------------------
  97. GET latency/_search
  98. {
  99. "size": 0,
  100. "aggs": {
  101. "load_time_boxplot": {
  102. "boxplot": {
  103. "script": {
  104. "id": "my_script",
  105. "params": {
  106. "field": "load_time"
  107. }
  108. }
  109. }
  110. }
  111. }
  112. }
  113. --------------------------------------------------
  114. // TEST[setup:latency,stored_example_script]
  115. [[search-aggregations-metrics-boxplot-aggregation-approximation]]
  116. ==== Boxplot values are (usually) approximate
  117. The algorithm used by the `boxplot` metric is called TDigest (introduced by
  118. Ted Dunning in
  119. https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf[Computing Accurate Quantiles using T-Digests]).
  120. [WARNING]
  121. ====
  122. Boxplot as other percentile aggregations are also
  123. {wikipedia}/Nondeterministic_algorithm[non-deterministic].
  124. This means you can get slightly different results using the same data.
  125. ====
  126. [[search-aggregations-metrics-boxplot-aggregation-compression]]
  127. ==== Compression
  128. Approximate algorithms must balance memory utilization with estimation accuracy.
  129. This balance can be controlled using a `compression` parameter:
  130. [source,console]
  131. --------------------------------------------------
  132. GET latency/_search
  133. {
  134. "size": 0,
  135. "aggs": {
  136. "load_time_boxplot": {
  137. "boxplot": {
  138. "field": "load_time",
  139. "compression": 200 <1>
  140. }
  141. }
  142. }
  143. }
  144. --------------------------------------------------
  145. // TEST[setup:latency]
  146. <1> Compression controls memory usage and approximation error
  147. include::percentile-aggregation.asciidoc[tags=t-digest]
  148. ==== Missing value
  149. The `missing` parameter defines how documents that are missing a value should be treated.
  150. By default they will be ignored but it is also possible to treat them as if they
  151. had a value.
  152. [source,console]
  153. --------------------------------------------------
  154. GET latency/_search
  155. {
  156. "size": 0,
  157. "aggs": {
  158. "grade_boxplot": {
  159. "boxplot": {
  160. "field": "grade",
  161. "missing": 10 <1>
  162. }
  163. }
  164. }
  165. }
  166. --------------------------------------------------
  167. // TEST[setup:latency]
  168. <1> Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`.