boxplot-aggregation.asciidoc 5.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[search-aggregations-metrics-boxplot-aggregation]]
  4. === Boxplot aggregation
  5. ++++
  6. <titleabbrev>Boxplot</titleabbrev>
  7. ++++
  8. A `boxplot` metrics aggregation that computes boxplot of numeric values extracted from the aggregated documents.
  9. These values can be generated from specific numeric or <<histogram,histogram fields>> in the documents.
  10. The `boxplot` aggregation returns essential information for making a {wikipedia}/Box_plot[box plot]: minimum, maximum,
  11. median, first quartile (25th percentile) and third quartile (75th percentile) values.
  12. ==== Syntax
  13. A `boxplot` aggregation looks like this in isolation:
  14. [source,js]
  15. --------------------------------------------------
  16. {
  17. "boxplot": {
  18. "field": "load_time"
  19. }
  20. }
  21. --------------------------------------------------
  22. // NOTCONSOLE
  23. Let's look at a boxplot representing load time:
  24. [source,console]
  25. --------------------------------------------------
  26. GET latency/_search
  27. {
  28. "size": 0,
  29. "aggs": {
  30. "load_time_boxplot": {
  31. "boxplot": {
  32. "field": "load_time" <1>
  33. }
  34. }
  35. }
  36. }
  37. --------------------------------------------------
  38. // TEST[setup:latency]
  39. <1> The field `load_time` must be a numeric field
  40. The response will look like this:
  41. [source,console-result]
  42. --------------------------------------------------
  43. {
  44. ...
  45. "aggregations": {
  46. "load_time_boxplot": {
  47. "min": 0.0,
  48. "max": 990.0,
  49. "q1": 165.0,
  50. "q2": 445.0,
  51. "q3": 725.0,
  52. "lower": 0.0,
  53. "upper": 990.0
  54. }
  55. }
  56. }
  57. --------------------------------------------------
  58. // TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
  59. In this case, the lower and upper whisker values are equal to the min and max. In general, these values are the 1.5 *
  60. IQR range, which is to say the nearest values to `q1 - (1.5 * IQR)` and `q3 + (1.5 * IQR)`. Since this is an approximation, the given values
  61. may not actually be observed values from the data, but should be within a reasonable error bound of them. While the Boxplot aggregation
  62. doesn't directly return outlier points, you can check if `lower > min` or `upper < max` to see if outliers exist on either side, and then
  63. query for them directly.
  64. ==== Script
  65. If you need to create a boxplot for values that aren't indexed exactly you
  66. should create a <<runtime,runtime field>> and get the boxplot of that. For
  67. example, if your load times are in milliseconds but you want values calculated
  68. in seconds, use a runtime field to convert them:
  69. [source,console]
  70. ----
  71. GET latency/_search
  72. {
  73. "size": 0,
  74. "runtime_mappings": {
  75. "load_time.seconds": {
  76. "type": "long",
  77. "script": {
  78. "source": "emit(doc['load_time'].value / params.timeUnit)",
  79. "params": {
  80. "timeUnit": 1000
  81. }
  82. }
  83. }
  84. },
  85. "aggs": {
  86. "load_time_boxplot": {
  87. "boxplot": { "field": "load_time.seconds" }
  88. }
  89. }
  90. }
  91. ----
  92. // TEST[setup:latency]
  93. // TEST[s/_search/_search?filter_path=aggregations/]
  94. // TEST[s/"timeUnit": 1000/"timeUnit": 10/]
  95. ////
  96. [source,console-result]
  97. --------------------------------------------------
  98. {
  99. "aggregations": {
  100. "load_time_boxplot": {
  101. "min": 0.0,
  102. "max": 99.0,
  103. "q1": 16.5,
  104. "q2": 44.5,
  105. "q3": 72.5,
  106. "lower": 0.0,
  107. "upper": 99.0
  108. }
  109. }
  110. }
  111. --------------------------------------------------
  112. ////
  113. [[search-aggregations-metrics-boxplot-aggregation-approximation]]
  114. ==== Boxplot values are (usually) approximate
  115. The algorithm used by the `boxplot` metric is called TDigest (introduced by
  116. Ted Dunning in
  117. https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf[Computing Accurate Quantiles using T-Digests]).
  118. [WARNING]
  119. ====
  120. Boxplot as other percentile aggregations are also
  121. {wikipedia}/Nondeterministic_algorithm[non-deterministic].
  122. This means you can get slightly different results using the same data.
  123. ====
  124. [[search-aggregations-metrics-boxplot-aggregation-compression]]
  125. ==== Compression
  126. Approximate algorithms must balance memory utilization with estimation accuracy.
  127. This balance can be controlled using a `compression` parameter:
  128. [source,console]
  129. --------------------------------------------------
  130. GET latency/_search
  131. {
  132. "size": 0,
  133. "aggs": {
  134. "load_time_boxplot": {
  135. "boxplot": {
  136. "field": "load_time",
  137. "compression": 200 <1>
  138. }
  139. }
  140. }
  141. }
  142. --------------------------------------------------
  143. // TEST[setup:latency]
  144. <1> Compression controls memory usage and approximation error
  145. include::percentile-aggregation.asciidoc[tags=t-digest]
  146. ==== Missing value
  147. The `missing` parameter defines how documents that are missing a value should be treated.
  148. By default they will be ignored but it is also possible to treat them as if they
  149. had a value.
  150. [source,console]
  151. --------------------------------------------------
  152. GET latency/_search
  153. {
  154. "size": 0,
  155. "aggs": {
  156. "grade_boxplot": {
  157. "boxplot": {
  158. "field": "grade",
  159. "missing": 10 <1>
  160. }
  161. }
  162. }
  163. }
  164. --------------------------------------------------
  165. // TEST[setup:latency]
  166. <1> Documents without a value in the `grade` field will fall into the same bucket as documents that have the value `10`.