Re-enable doc testing for Pipeline Aggregations (#24374)

* Re-enable doc testing for Pipeline Aggregations

Also adds a response + test for movavg pipeline

docs/build.gradle

@@ -39,20 +39,6 @@ buildRestTests.expectedUnconvertedCandidates = [
   'reference/aggregations/metrics/scripted-metric-aggregation.asciidoc',
   'reference/aggregations/metrics/stats-aggregation.asciidoc',
   'reference/aggregations/metrics/tophits-aggregation.asciidoc',
-  'reference/aggregations/pipeline.asciidoc',
-  'reference/aggregations/pipeline/avg-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/bucket-script-aggregation.asciidoc',
-  'reference/aggregations/pipeline/bucket-selector-aggregation.asciidoc',
-  'reference/aggregations/pipeline/cumulative-sum-aggregation.asciidoc',
-  'reference/aggregations/pipeline/derivative-aggregation.asciidoc',
-  'reference/aggregations/pipeline/extended-stats-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/max-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/min-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/movavg-aggregation.asciidoc',
-  'reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/serial-diff-aggregation.asciidoc',
-  'reference/aggregations/pipeline/stats-bucket-aggregation.asciidoc',
-  'reference/aggregations/pipeline/sum-bucket-aggregation.asciidoc',
   'reference/cat/snapshots.asciidoc',
   'reference/cat/templates.asciidoc',
   'reference/cat/thread_pool.asciidoc',

docs/reference/aggregations/pipeline.asciidoc

@@ -194,6 +194,7 @@ may be referred to as:
 ---------------
 "buckets_path": "my_percentile[99.9]"
 ---------------
+// NOTCONSOLE
 
 [[gap-policy]]
 [float]

docs/reference/aggregations/pipeline/avg-bucket-aggregation.asciidoc

@@ -18,6 +18,7 @@ An `avg_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`avg_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/bucket-script-aggregation.asciidoc

@@ -22,6 +22,7 @@ A `bucket_script` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 <1> Here, `my_var1` is the name of the variable for this buckets path to use in the script, `the_sum` is the path to
 the metrics to use for that variable.
 

docs/reference/aggregations/pipeline/bucket-selector-aggregation.asciidoc

@@ -27,6 +27,7 @@ A `bucket_selector` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 <1> Here, `my_var1` is the name of the variable for this buckets path to use in the script, `the_sum` is the path to
 the metrics to use for that variable.
 

docs/reference/aggregations/pipeline/cumulative-sum-aggregation.asciidoc

@@ -19,6 +19,7 @@ A `cumulative_sum` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`cumulative_sum` Parameters
 |===

docs/reference/aggregations/pipeline/derivative-aggregation.asciidoc

@@ -17,6 +17,7 @@ A `derivative` aggregation looks like this in isolation:
   "buckets_path": "the_sum"
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`derivative` Parameters
 |===

docs/reference/aggregations/pipeline/extended-stats-bucket-aggregation.asciidoc

@@ -20,6 +20,7 @@ A `extended_stats_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`extended_stats_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/max-bucket-aggregation.asciidoc

@@ -19,6 +19,7 @@ A `max_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`max_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/min-bucket-aggregation.asciidoc

@@ -19,6 +19,7 @@ A `min_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`min_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/movavg-aggregation.asciidoc

@@ -34,6 +34,7 @@ A `moving_avg` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`moving_avg` Parameters
 |===
@@ -58,12 +59,12 @@ POST /_search
     "aggs": {
         "my_date_histo":{                <1>
             "date_histogram":{
-                "field":"timestamp",
-                "interval":"day"
+                "field":"date",
+                "interval":"1M"
             },
             "aggs":{
                 "the_sum":{
-                    "sum":{ "field": "lemmings" } <2>
+                    "sum":{ "field": "price" } <2>
                 },
                 "the_movavg":{
                     "moving_avg":{ "buckets_path": "the_sum" } <3>
@@ -74,6 +75,7 @@ POST /_search
 }
 --------------------------------------------------
 // CONSOLE
+// TEST[setup:sales]
 
 <1> A `date_histogram` named "my_date_histo" is constructed on the "timestamp" field, with one-day intervals
 <2> A `sum` metric is used to calculate the sum of a field.  This could be any metric (sum, min, max, etc)
@@ -84,6 +86,57 @@ add normal metrics, such as a `sum`, inside of that histogram.  Finally, the `mo
 The `buckets_path` parameter is then used to "point" at one of the sibling metrics inside of the histogram (see
 <<buckets-path-syntax>> for a description of the syntax for `buckets_path`.
 
+An example response from the above aggregation may look like:
+
+[source,js]
+--------------------------------------------------
+{
+   "took": 11,
+   "timed_out": false,
+   "_shards": ...,
+   "hits": ...,
+   "aggregations": {
+      "my_date_histo": {
+         "buckets": [
+             {
+                 "key_as_string": "2015/01/01 00:00:00",
+                 "key": 1420070400000,
+                 "doc_count": 3,
+                 "the_sum": {
+                    "value": 550.0
+                 }
+             },
+             {
+                 "key_as_string": "2015/02/01 00:00:00",
+                 "key": 1422748800000,
+                 "doc_count": 2,
+                 "the_sum": {
+                    "value": 60.0
+                 },
+                 "the_movavg": {
+                    "value": 550.0
+                 }
+             },
+             {
+                 "key_as_string": "2015/03/01 00:00:00",
+                 "key": 1425168000000,
+                 "doc_count": 2,
+                 "the_sum": {
+                    "value": 375.0
+                 },
+                 "the_movavg": {
+                    "value": 305.0
+                 }
+             }
+         ]
+      }
+   }
+}
+--------------------------------------------------
+// TESTRESPONSE[s/"took": 11/"took": $body.took/]
+// TESTRESPONSE[s/"_shards": \.\.\./"_shards": $body._shards/]
+// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
+
 
 ==== Models
 
@@ -102,16 +155,33 @@ the values from a `simple` moving average tend to "lag" behind the real data.
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "simple"
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg":{
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "simple"
+                    }
+                }
+            }
         }
     }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 A `simple` model has no special settings to configure
 
@@ -138,15 +208,33 @@ the "lag" behind the data's mean, since older points have less influence.
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "linear"
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "linear"
+                    }
+                }
+            }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 A `linear` model has no special settings to configure
 
@@ -179,19 +267,36 @@ The EWMA model can be <<movavg-minimizer, Minimized>>
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "ewma",
-            "settings" : {
-                "alpha" : 0.5
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "ewma",
+                        "settings" : {
+                            "alpha" : 0.5
+                        }
+                    }
+                }
             }
         }
+    }
 }
 --------------------------------------------------
-
+// CONSOLE
+// TEST[setup:sales]
 
 
 [[single_0.2alpha]]
@@ -221,19 +326,37 @@ The Holt-Linear model can be <<movavg-minimizer, Minimized>>
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "holt",
-            "settings" : {
-                "alpha" : 0.5,
-                "beta" : 0.5
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "holt",
+                        "settings" : {
+                            "alpha" : 0.5,
+                            "beta" : 0.5
+                        }
+                    }
+                }
             }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 In practice, the `alpha` value behaves very similarly in `holt` as `ewma`: small values produce more smoothing
 and more lag, while larger values produce closer tracking and less lag.  The value of `beta` is often difficult
@@ -291,22 +414,40 @@ The additive Holt-Winters model can be <<movavg-minimizer, Minimized>>
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "holt_winters",
-            "settings" : {
-                "type" : "add",
-                "alpha" : 0.5,
-                "beta" : 0.5,
-                "gamma" : 0.5,
-                "period" : 7
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "holt_winters",
+                        "settings" : {
+                            "type" : "add",
+                            "alpha" : 0.5,
+                            "beta" : 0.5,
+                            "gamma" : 0.5,
+                            "period" : 7
+                        }
+                    }
+                }
             }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 
 [[holt_winters_add]]
@@ -334,23 +475,41 @@ you can disable this behavior with `pad: false`
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "holt_winters",
-            "settings" : {
-                "type" : "mult",
-                "alpha" : 0.5,
-                "beta" : 0.5,
-                "gamma" : 0.5,
-                "period" : 7,
-                "pad" : true
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "holt_winters",
+                        "settings" : {
+                            "type" : "mult",
+                            "alpha" : 0.5,
+                            "beta" : 0.5,
+                            "gamma" : 0.5,
+                            "period" : 7,
+                            "pad" : true
+                        }
+                    }
+                }
             }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 ==== Prediction
 
@@ -363,16 +522,34 @@ as your buckets:
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "window" : 30,
-            "model" : "simple",
-            "predict" : 10
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "window" : 30,
+                        "model" : "simple",
+                        "predict" : 10
+                    }
+                }
+            }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
 
 The `simple`, `linear` and `ewma` models all produce "flat" predictions: they essentially converge on the mean
 of the last value in the series, producing a flat:
@@ -423,19 +600,38 @@ Minimization is enabled/disabled via the `minimize` parameter:
 
 [source,js]
 --------------------------------------------------
+POST /_search
 {
-    "the_movavg":{
-        "moving_avg":{
-            "buckets_path": "the_sum",
-            "model" : "holt_winters",
-            "window" : 30,
-            "minimize" : true,  <1>
-            "settings" : {
-                "period" : 7
+    "size": 0,
+    "aggs": {
+        "my_date_histo":{
+            "date_histogram":{
+                "field":"date",
+                "interval":"1M"
+            },
+            "aggs":{
+                "the_sum":{
+                    "sum":{ "field": "price" }
+                },
+                "the_movavg": {
+                    "moving_avg":{
+                        "buckets_path": "the_sum",
+                        "model" : "holt_winters",
+                        "window" : 30,
+                        "minimize" : true,  <1>
+                        "settings" : {
+                            "period" : 7
+                        }
+                    }
+                }
             }
         }
+    }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[setup:sales]
+
 <1> Minimization is enabled with the `minimize` parameter
 
 When enabled, minimization will find the optimal values for `alpha`, `beta` and `gamma`.  The user should still provide

docs/reference/aggregations/pipeline/percentiles-bucket-aggregation.asciidoc

@@ -18,6 +18,7 @@ A `percentiles_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`sum_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/serial-diff-aggregation.asciidoc

@@ -46,6 +46,7 @@ A `serial_diff` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`serial_diff` Parameters
 |===

docs/reference/aggregations/pipeline/stats-bucket-aggregation.asciidoc

@@ -18,6 +18,7 @@ A `stats_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`stats_bucket` Parameters
 |===

docs/reference/aggregations/pipeline/sum-bucket-aggregation.asciidoc

@@ -18,6 +18,7 @@ A `sum_bucket` aggregation looks like this in isolation:
     }
 }
 --------------------------------------------------
+// NOTCONSOLE
 
 .`sum_bucket` Parameters
 |===