Switch TDigestState to use HybridDigest by default (#96904)

* Initial import for TDigest forking.

* Fix MedianTest.

More work needed for TDigestPercentile*Tests and the TDigestTest (and
the rest of the tests) in the tdigest lib to pass.

* Fix Dist.

* Fix AVLTreeDigest.quantile to match Dist for uniform centroids.

* Update docs/changelog/96086.yaml

* Fix `MergingDigest.quantile` to match `Dist` on uniform distribution.

* Add merging to TDigestState.hashCode and .equals.

Remove wrong asserts from tests and MergingDigest.

* Fix style violations for tdigest library.

* Fix typo.

* Fix more style violations.

* Fix more style violations.

* Fix remaining style violations in tdigest library.

* Update results in docs based on the forked tdigest.

* Fix YAML tests in aggs module.

* Fix YAML tests in x-pack/plugin.

* Skip failing V7 compat tests in modules/aggregations.

* Fix TDigest library unittests.

Remove redundant serializing interfaces from the library.

* Remove YAML test versions for older releases.

These tests don't address compatibility issues in mixed cluster tests as
the latter contain a mix of older and newer nodes, so the output depends
on which node is picked as a data node since the forked TDigest library
is not backwards compatible (produces slightly different results).

* Fix test failures in docs and mixed cluster.

* Reduce buffer sizes in MergingDigest to avoid oom.

* Exclude more failing V7 compatibility tests.

* Update results for JdbcCsvSpecIT tests.

* Update results for JdbcDocCsvSpecIT tests.

* Revert unrelated change.

* More test fixes.

* Use version skips instead of blacklisting in mixed cluster tests.

* Switch TDigestState back to AVLTreeDigest.

* Update docs and tests with AVLTreeDigest output.

* Update flaky test.

* Remove dead code, esp around tracking of incoming data.

* Update docs/changelog/96086.yaml

* Delete docs/changelog/96086.yaml

* Remove explicit compression calls.

This was added to prevent concurrency tests from failing, but it leads
to reduces precision. Submit this to see if the concurrency tests are
still failing.

* Revert "Remove explicit compression calls."

This reverts commit 5352c96f656b9cdc63f0f673c983150bea35c7ed.

* Remove explicit compression calls to MedianAbsoluteDeviation input.

* Add unittests for AVL and merging digest accuracy.

* Fix spotless violations.

* Delete redundant tests and benchmarks.

* Fix spotless violation.

* Use the old implementation of AVLTreeDigest.

The latest library version is 50% slower and less accurate, as verified
by ComparisonTests.

* Update docs with latest percentile results.

* Update docs with latest percentile results.

* Remove repeated compression calls.

* Update more percentile results.

* Use approximate percentile values in integration tests.

This helps with mixed cluster tests, where some of the tests where
blocked.

* Fix expected percentile value in test.

* Revert in-place node updates in AVL tree.

Update quantile calculations between centroids and min/max values to
match v.3.2.

* Add SortingDigest and HybridDigest.

The SortingDigest tracks all samples in an ArrayList that
gets sorted for quantile calculations. This approach
provides perfectly accurate results and is the most
efficient implementation for up to millions of samples,
at the cost of bloated memory footprint.

The HybridDigest uses a SortingDigest for small sample
populations, then switches to a MergingDigest. This
approach combines to the best performance and results for
small sample counts with very good performance and
acceptable accuracy for effectively unbounded sample
counts.

* Remove deps to the 3.2 library.

* Remove unused licenses for tdigest.

* Revert changes for SortingDigest and HybridDigest.

These will be submitted in a follow-up PR for enabling MergingDigest.

* Remove unused Histogram classes and unit tests.

Delete dead and commented out code, make the remaining tests run
reasonably fast. Remove unused annotations, esp. SuppressWarnings.

* Remove Comparison class, not used.

* Revert "Revert changes for SortingDigest and HybridDigest."

This reverts commit 2336b11598a5b2e4ef284560db2469cb0a054849.

* Use HybridDigest as default tdigest implementation

Add SortingDigest as a simple structure for percentile calculations that
tracks all data points in a sorted array. This is a fast and perfectly
accurate solution that leads to bloated memory allocation.

Add HybridDigest that uses SortingDigest for small sample counts, then
switches to MergingDigest. This approach delivers extreme
performance and accuracy for small populations while scaling
indefinitely and maintaining acceptable performance and accuracy with
constant memory allocation (15kB by default).

Provide knobs to switch back to AVLTreeDigest, either per query or
through ClusterSettings.

* Small fixes.

* Add javadoc and tests.

* Add javadoc and tests.

* Remove special logic for singletons in the boundaries.

While this helps with the case where the digest contains only
singletons (perfect accuracy), it has a major issue problem
(non-monotonic quantile function) when the first singleton is followed
by a non-singleton centroid. It's preferable to revert to the old
version from 3.2; inaccuracies in a singleton-only digest should be
mitigated by using a sorted array for small sample counts.

* Revert changes to expected values in tests.

This is due to restoring quantile functions to match head.

* Revert changes to expected values in tests.

This is due to restoring quantile functions to match head.

* Tentatively restore percentile rank expected results.

* Use cdf version from 3.2

Update Dist.cdf to use interpolation, use the same cdf
version in AVLTreeDigest and MergingDigest.

* Revert "Tentatively restore percentile rank expected results."

This reverts commit 7718dbba594f068913dfb7b85f488ca25e5e47bf.

* Revert remaining changes compared to main.

* Revert excluded V7 compat tests.

* Exclude V7 compat tests still failing.

* Exclude V7 compat tests still failing.

* Remove ClusterSettings tentatively.

* Initial import for TDigest forking.

* Fix MedianTest.

More work needed for TDigestPercentile*Tests and the TDigestTest (and
the rest of the tests) in the tdigest lib to pass.

* Fix Dist.

* Fix AVLTreeDigest.quantile to match Dist for uniform centroids.

* Update docs/changelog/96086.yaml

* Fix `MergingDigest.quantile` to match `Dist` on uniform distribution.

* Add merging to TDigestState.hashCode and .equals.

Remove wrong asserts from tests and MergingDigest.

* Fix style violations for tdigest library.

* Fix typo.

* Fix more style violations.

* Fix more style violations.

* Fix remaining style violations in tdigest library.

* Update results in docs based on the forked tdigest.

* Fix YAML tests in aggs module.

* Fix YAML tests in x-pack/plugin.

* Skip failing V7 compat tests in modules/aggregations.

* Fix TDigest library unittests.

Remove redundant serializing interfaces from the library.

* Remove YAML test versions for older releases.

These tests don't address compatibility issues in mixed cluster tests as
the latter contain a mix of older and newer nodes, so the output depends
on which node is picked as a data node since the forked TDigest library
is not backwards compatible (produces slightly different results).

* Fix test failures in docs and mixed cluster.

* Reduce buffer sizes in MergingDigest to avoid oom.

* Exclude more failing V7 compatibility tests.

* Update results for JdbcCsvSpecIT tests.

* Update results for JdbcDocCsvSpecIT tests.

* Revert unrelated change.

* More test fixes.

* Use version skips instead of blacklisting in mixed cluster tests.

* Switch TDigestState back to AVLTreeDigest.

* Update docs and tests with AVLTreeDigest output.

* Update flaky test.

* Remove dead code, esp around tracking of incoming data.

* Remove explicit compression calls.

This was added to prevent concurrency tests from failing, but it leads
to reduces precision. Submit this to see if the concurrency tests are
still failing.

* Update docs/changelog/96086.yaml

* Delete docs/changelog/96086.yaml

* Revert "Remove explicit compression calls."

This reverts commit 5352c96f656b9cdc63f0f673c983150bea35c7ed.

* Remove explicit compression calls to MedianAbsoluteDeviation input.

* Add unittests for AVL and merging digest accuracy.

* Fix spotless violations.

* Delete redundant tests and benchmarks.

* Fix spotless violation.

* Use the old implementation of AVLTreeDigest.

The latest library version is 50% slower and less accurate, as verified
by ComparisonTests.

* Update docs with latest percentile results.

* Update docs with latest percentile results.

* Remove repeated compression calls.

* Update more percentile results.

* Use approximate percentile values in integration tests.

This helps with mixed cluster tests, where some of the tests where
blocked.

* Fix expected percentile value in test.

* Revert in-place node updates in AVL tree.

Update quantile calculations between centroids and min/max values to
match v.3.2.

* Add SortingDigest and HybridDigest.

The SortingDigest tracks all samples in an ArrayList that
gets sorted for quantile calculations. This approach
provides perfectly accurate results and is the most
efficient implementation for up to millions of samples,
at the cost of bloated memory footprint.

The HybridDigest uses a SortingDigest for small sample
populations, then switches to a MergingDigest. This
approach combines to the best performance and results for
small sample counts with very good performance and
acceptable accuracy for effectively unbounded sample
counts.

* Remove deps to the 3.2 library.

* Remove unused licenses for tdigest.

* Revert changes for SortingDigest and HybridDigest.

These will be submitted in a follow-up PR for enabling MergingDigest.

* Remove unused Histogram classes and unit tests.

Delete dead and commented out code, make the remaining tests run
reasonably fast. Remove unused annotations, esp. SuppressWarnings.

* Remove Comparison class, not used.

* Revert "Revert changes for SortingDigest and HybridDigest."

This reverts commit 2336b11598a5b2e4ef284560db2469cb0a054849.

* Use HybridDigest as default tdigest implementation

Add SortingDigest as a simple structure for percentile calculations that
tracks all data points in a sorted array. This is a fast and perfectly
accurate solution that leads to bloated memory allocation.

Add HybridDigest that uses SortingDigest for small sample counts, then
switches to MergingDigest. This approach delivers extreme
performance and accuracy for small populations while scaling
indefinitely and maintaining acceptable performance and accuracy with
constant memory allocation (15kB by default).

Provide knobs to switch back to AVLTreeDigest, either per query or
through ClusterSettings.

* Add javadoc and tests.

* Remove ClusterSettings tentatively.

* Restore bySize function in TDigest and subclasses.

* Update Dist.cdf to match the rest.

Update tests.

* Revert outdated test changes.

* Revert outdated changes.

* Small fixes.

* Update docs/changelog/96794.yaml

* TDigestState uses MergingDigest by default.

* Make HybridDigest the default implementation.

* Update boxplot documentation.

* Use HybridDigest for real.

* Restore AVLTreeDigest as the default in TDigestState.

TDigest.createHybridDigest nw returns the right type.
The switch in TDigestState will happen in a separate PR
as it requires many test updates.

* Use execution_hint in tdigest spec.

* Restore expected test values.

* Fix Dist.cdf for empty digest.

* Bump up TransportVersion.

* More test updates.

* Bump up TransportVersion for real.

* Restore V7 compat blacklisting.

* HybridDigest uses its final implementation during deserialization.

* Restore the right TransportVersion in TDigestState.read

* More test fixes.

* More test updates.

* Use TDigestExecutionHint instead of strings.

* Add link to TDigest javadoc.

* Spotless fix.

* Small fixes.

* Bump up TransportVersion.

* Bump up the TransportVersion, again.

* Update docs/changelog/96904.yaml

* Delete 96794.yaml

Delete existing changelog to get a new one.

* Restore previous changelog.

* Rename  96794.yaml to 96794.yaml

* Update breaking change notes in changelog.

* Remove mapping value from changelog.

* Set a valid breaking area.

* Use HybridDigest as default TDigest impl.

* Update docs/changelog/96904.yaml

* Use TDigestExecutionHint in MedianAbsoluteDeviationAggregator.

* Update changelog and comment in blacklisted V7 compat tests.

* Update breaking area in changelog.

docs/changelog/96904.yaml

@@ -0,0 +1,18 @@
+pr: 96904
+summary: Switch TDigestState to use `HybridDigest` by default
+area: Aggregations
+type: breaking
+issues: []
+breaking:
+  title: Switch TDigestState to use `HybridDigest` by default
+  area: REST API
+  details: "The default implementation for TDigest in percentile calculations switches\
+    \ to a new internal implementation offering superior performance (2x-10x speedup),\
+    \ at a very small accuracy penalty for very large sample populations."
+  impact: "This change leads to generating slightly different results in percentile\
+    \ calculations. If the highest possible accuracy is desired, or it's crucial to\
+    \ produce exactly the same results as in previous versions, one can either set\
+    \ `execution_hint` to `high_accuracy` in the `tdigest` spec of a given percentile\
+    \ calculation, or set `search.aggs.tdigest_execution_hint` to `high_accuracy`\
+    \ in cluster settings to apply to all percentile queries."
+  notable: true

docs/reference/aggregations/metrics/boxplot-aggregation.asciidoc

@@ -55,9 +55,9 @@ The response will look like this:
     "load_time_boxplot": {
       "min": 0.0,
       "max": 990.0,
-      "q1": 165.0,
+      "q1": 167.5,
       "q2": 445.0,
-      "q3": 725.0,
+      "q3": 722.5,
       "lower": 0.0,
       "upper": 990.0
     }
@@ -114,9 +114,9 @@ GET latency/_search
     "load_time_boxplot": {
       "min": 0.0,
       "max": 99.0,
-      "q1": 16.5,
+      "q1": 16.75,
       "q2": 44.5,
-      "q3": 72.5,
+      "q3": 72.25,
       "lower": 0.0,
       "upper": 99.0
     }

docs/reference/aggregations/metrics/percentile-aggregation.asciidoc

@@ -54,19 +54,26 @@ percentiles: `[ 1, 5, 25, 50, 75, 95, 99 ]`. The response will look like this:
  "aggregations": {
     "load_time_outlier": {
       "values": {
-        "1.0": 5.0,
-        "5.0": 25.0,
-        "25.0": 165.0,
+        "1.0": 10.0,
+        "5.0": 30.0,
+        "25.0": 170.0,
         "50.0": 445.0,
-        "75.0": 725.0,
-        "95.0": 945.0,
-        "99.0": 985.0
+        "75.0": 720.0,
+        "95.0": 940.0,
+        "99.0": 980.0
       }
     }
   }
 }
 --------------------------------------------------
 // TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
+// TESTRESPONSE[s/"1.0": 10.0/"1.0": 9.9/]
+// TESTRESPONSE[s/"5.0": 30.0/"5.0": 29.5/]
+// TESTRESPONSE[s/"25.0": 170.0/"25.0": 167.5/]
+// TESTRESPONSE[s/"50.0": 445.0/"50.0": 445.0/]
+// TESTRESPONSE[s/"75.0": 720.0/"75.0": 722.5/]
+// TESTRESPONSE[s/"95.0": 940.0/"95.0": 940.5/]
+// TESTRESPONSE[s/"99.0": 980.0/"99.0": 980.1/]
 
 As you can see, the aggregation will return a calculated value for each percentile
 in the default range. If we assume response times are in milliseconds, it is
@@ -128,15 +135,15 @@ Response:
       "values": [
         {
           "key": 1.0,
-          "value": 5.0
+          "value": 10.0
         },
         {
           "key": 5.0,
-          "value": 25.0
+          "value": 30.0
         },
         {
           "key": 25.0,
-          "value": 165.0
+          "value": 170.0
         },
         {
           "key": 50.0,
@@ -144,15 +151,15 @@ Response:
         },
         {
           "key": 75.0,
-          "value": 725.0
+          "value": 720.0
         },
         {
           "key": 95.0,
-          "value": 945.0
+          "value": 940.0
         },
         {
           "key": 99.0,
-          "value": 985.0
+          "value": 980.0
         }
       ]
     }
@@ -160,6 +167,13 @@ Response:
 }
 --------------------------------------------------
 // TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
+// TESTRESPONSE[s/"value": 10.0/"value": 9.9/]
+// TESTRESPONSE[s/"value": 30.0/"value": 29.5/]
+// TESTRESPONSE[s/"value": 170.0/"value": 167.5/]
+// TESTRESPONSE[s/"value": 445.0/"value": 445.0/]
+// TESTRESPONSE[s/"value": 720.0/"value": 722.5/]
+// TESTRESPONSE[s/"value": 940.0/"value": 940.5/]
+// TESTRESPONSE[s/"value": 980.0/"value": 980.1/]
 
 ==== Script
 
@@ -203,13 +217,13 @@ GET latency/_search
  "aggregations": {
     "load_time_outlier": {
       "values": {
-        "1.0": 0.5,
-        "5.0": 2.5,
-        "25.0": 16.5,
+        "1.0": 0.99,
+        "5.0": 2.95,
+        "25.0": 16.75,
         "50.0": 44.5,
-        "75.0": 72.5,
-        "95.0": 94.5,
-        "99.0": 98.5
+        "75.0": 72.25,
+        "95.0": 94.05,
+        "99.0": 98.01
       }
     }
   }

docs/reference/aggregations/pipeline/moving-percentiles-aggregation.asciidoc

@@ -97,7 +97,7 @@ And the following may be the response:
                  "doc_count": 3,
                  "the_percentile": {
                      "values": {
-                       "1.0": 150.0,
+                       "1.0": 151.0,
                        "99.0": 200.0
                      }
                  }
@@ -108,13 +108,13 @@ And the following may be the response:
                  "doc_count": 2,
                  "the_percentile": {
                      "values": {
-                       "1.0": 10.0,
-                       "99.0": 50.0
+                       "1.0": 10.4,
+                       "99.0": 49.6
                      }
                  },
                  "the_movperc": {
                    "values": {
-                     "1.0": 150.0,
+                     "1.0": 151.0,
                      "99.0": 200.0
                    }
                  }
@@ -125,13 +125,13 @@ And the following may be the response:
                  "doc_count": 2,
                  "the_percentile": {
                     "values": {
-                      "1.0": 175.0,
-                      "99.0": 200.0
+                      "1.0": 175.25,
+                      "99.0": 199.75
                     }
                  },
                  "the_movperc": {
                     "values": {
-                      "1.0": 10.0,
+                      "1.0": 11.6,
                       "99.0": 200.0
                     }
                  }

modules/aggregations/build.gradle

@@ -43,7 +43,17 @@ tasks.named("yamlRestTestV7CompatTransform").configure { task ->
   task.skipTest("search.aggregation/20_terms/numeric profiler", "The profiler results aren't backwards compatible.")
   task.skipTest("search.aggregation/210_top_hits_nested_metric/top_hits aggregation with sequence numbers", "#42809 the use nested path and filter sort throws an exception")
   task.skipTest("search.aggregation/370_doc_count_field/Test filters agg with doc_count", "Uses profiler for assertions which is not backwards compatible")
-  task.skipTest("search.aggregation/420_percentile_ranks_tdigest_metric/filtered", "Uses t-digest library which is not backwards compatible")
+
+  // In 8.9.0, the default t-digest algorithm changed from AVL-tree-based to hybrid, combining a sorted array of samples with a merging
+  // implementation. This change leads to slight different percentile results, compared to previous versions.
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Basic test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Non-keyed test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Only aggs test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Explicit Percents test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Metadata test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/180_percentiles_tdigest_metric/Filtered test", "Hybrid t-digest produces different results.")
+  task.skipTest("search.aggregation/420_percentile_ranks_tdigest_metric/filtered", "Hybrid t-digest produces different results.")
+
   task.addAllowedWarningRegex("\\[types removal\\].*")
 }
 

modules/aggregations/src/yamlRestTest/resources/rest-api-spec/test/aggregations/percentiles_tdigest_metric.yml

@@ -46,6 +46,9 @@ setup:
 
 ---
 "Basic test":
+  - skip:
+      version: " - 8.8.99"
+      reason: t-digest forked in 8.9.0
   - do:
       search:
         rest_total_hits_as_int: true
@@ -61,21 +64,21 @@ setup:
   - match:  { hits.total: 4 }
   - length: { hits.hits: 4 }
 
-  - match:  { aggregations.percentiles_int.values.1\.0: 1.0 }
-  - match:  { aggregations.percentiles_int.values.5\.0: 1.0 }
-  - match:  { aggregations.percentiles_int.values.25\.0: 26.0 }
+  - match:  { aggregations.percentiles_int.values.1\.0: 2.5 }
+  - match:  { aggregations.percentiles_int.values.5\.0: 8.5 }
+  - match:  { aggregations.percentiles_int.values.25\.0: 38.5 }
   - match:  { aggregations.percentiles_int.values.50\.0: 76.0 }
-  - match:  { aggregations.percentiles_int.values.75\.0: 126.0 }
-  - match:  { aggregations.percentiles_int.values.95\.0: 151.0 }
-  - match:  { aggregations.percentiles_int.values.99\.0: 151.0 }
+  - match:  { aggregations.percentiles_int.values.75\.0: 113.5 }
+  - match:  { aggregations.percentiles_int.values.95\.0: 143.5 }
+  - match:  { aggregations.percentiles_int.values.99\.0: 149.5 }
 
-  - match:  { aggregations.percentiles_double.values.1\.0: 1.0 }
-  - match:  { aggregations.percentiles_double.values.5\.0: 1.0 }
-  - match:  { aggregations.percentiles_double.values.25\.0: 26.0 }
+  - match:  { aggregations.percentiles_double.values.1\.0: 2.5 }
+  - match:  { aggregations.percentiles_double.values.5\.0: 8.5 }
+  - match:  { aggregations.percentiles_double.values.25\.0: 38.5 }
   - match:  { aggregations.percentiles_double.values.50\.0: 76.0 }
-  - match:  { aggregations.percentiles_double.values.75\.0: 126.0 }
-  - match:  { aggregations.percentiles_double.values.95\.0: 151.0 }
-  - match:  { aggregations.percentiles_double.values.99\.0: 151.0 }
+  - match:  { aggregations.percentiles_double.values.75\.0: 113.5 }
+  - match:  { aggregations.percentiles_double.values.95\.0: 143.5 }
+  - match:  { aggregations.percentiles_double.values.99\.0: 149.5 }
 
   - do:
       search:
@@ -97,21 +100,21 @@ setup:
   - match: { hits.total: 4 }
   - length: { hits.hits: 4 }
 
-  - match:  { aggregations.percentiles_int.values.1\.0: 1.0 }
-  - match:  { aggregations.percentiles_int.values.5\.0: 1.0 }
-  - match:  { aggregations.percentiles_int.values.25\.0: 26.0 }
+  - match:  { aggregations.percentiles_int.values.1\.0: 2.5 }
+  - match:  { aggregations.percentiles_int.values.5\.0: 8.5 }
+  - match:  { aggregations.percentiles_int.values.25\.0: 38.5 }
   - match:  { aggregations.percentiles_int.values.50\.0: 76.0 }
-  - match:  { aggregations.percentiles_int.values.75\.0: 126.0 }
-  - match:  { aggregations.percentiles_int.values.95\.0: 151.0 }
-  - match:  { aggregations.percentiles_int.values.99\.0: 151.0 }
+  - match:  { aggregations.percentiles_int.values.75\.0: 113.5 }
+  - match:  { aggregations.percentiles_int.values.95\.0: 143.5 }
+  - match:  { aggregations.percentiles_int.values.99\.0: 149.5 }
 
-  - match:  { aggregations.percentiles_double.values.1\.0: 1.0 }
-  - match:  { aggregations.percentiles_double.values.5\.0: 1.0 }
-  - match:  { aggregations.percentiles_double.values.25\.0: 26.0 }
+  - match:  { aggregations.percentiles_double.values.1\.0: 2.5 }
+  - match:  { aggregations.percentiles_double.values.5\.0: 8.5 }
+  - match:  { aggregations.percentiles_double.values.25\.0: 38.5 }
   - match:  { aggregations.percentiles_double.values.50\.0: 76.0 }
-  - match:  { aggregations.percentiles_double.values.75\.0: 126.0 }
-  - match:  { aggregations.percentiles_double.values.95\.0: 151.0 }
-  - match:  { aggregations.percentiles_double.values.99\.0: 151.0 }
+  - match:  { aggregations.percentiles_double.values.75\.0: 113.5 }
+  - match:  { aggregations.percentiles_double.values.95\.0: 143.5 }
+  - match:  { aggregations.percentiles_double.values.99\.0: 149.5 }
 
 
 ---

server/src/main/java/org/elasticsearch/search/aggregations/metrics/MedianAbsoluteDeviationAggregator.java

@@ -92,7 +92,7 @@ public class MedianAbsoluteDeviationAggregator extends NumericMetricsAggregator.
 
                 TDigestState valueSketch = valueSketches.get(bucket);
                 if (valueSketch == null) {
-                    valueSketch = TDigestState.create(compression);
+                    valueSketch = TDigestState.create(compression, executionHint);
                     valueSketches.set(bucket, valueSketch);
                 }
 

server/src/main/java/org/elasticsearch/search/aggregations/metrics/TDigestState.java

@@ -38,7 +38,7 @@ public class TDigestState {
         SORTING;
 
         static Type defaultValue() {
-            return AVL_TREE;
+            return HYBRID;
         }
 
         static Type valueForHighAccuracy() {

server/src/test/java/org/elasticsearch/search/aggregations/metrics/TDigestPercentilesAggregatorTests.java

@@ -83,8 +83,8 @@ public class TDigestPercentilesAggregatorTests extends AggregatorTestCase {
         }, tdigest -> {
             assertEquals(7L, tdigest.state.size());
             assertEquals(7L, tdigest.state.centroidCount());
-            assertEquals(4.5d, tdigest.percentile(75), 0.0d);
-            assertEquals("4.5", tdigest.percentileAsString(75));
+            assertEquals(4.0d, tdigest.percentile(75), 0.0d);
+            assertEquals("4.0", tdigest.percentileAsString(75));
             assertEquals(2.0d, tdigest.percentile(50), 0.0d);
             assertEquals("2.0", tdigest.percentileAsString(50));
             assertEquals(1.0d, tdigest.percentile(22), 0.0d);
@@ -107,14 +107,14 @@ public class TDigestPercentilesAggregatorTests extends AggregatorTestCase {
             assertEquals(tdigest.state.centroidCount(), 7L);
             assertEquals(8.0d, tdigest.percentile(100), 0.0d);
             assertEquals("8.0", tdigest.percentileAsString(100));
-            assertEquals(6.98d, tdigest.percentile(88), 0.0d);
-            assertEquals("6.98", tdigest.percentileAsString(88));
+            assertEquals(4.0d, tdigest.percentile(75), 0.0d);
+            assertEquals("4.0", tdigest.percentileAsString(75));
             assertEquals(1.0d, tdigest.percentile(33), 0.0d);
             assertEquals("1.0", tdigest.percentileAsString(33));
             assertEquals(1.0d, tdigest.percentile(25), 0.0d);
             assertEquals("1.0", tdigest.percentileAsString(25));
-            assertEquals(0.0d, tdigest.percentile(1), 0.0d);
-            assertEquals("0.0", tdigest.percentileAsString(1));
+            assertEquals(0.06d, tdigest.percentile(1), 0.0d);
+            assertEquals("0.06", tdigest.percentileAsString(1));
             assertTrue(AggregationInspectionHelper.hasValue(tdigest));
         });
     }
@@ -135,7 +135,7 @@ public class TDigestPercentilesAggregatorTests extends AggregatorTestCase {
             assertEquals(4L, tdigest.state.centroidCount());
             assertEquals(2.0d, tdigest.percentile(100), 0.0d);
             assertEquals(1.0d, tdigest.percentile(50), 0.0d);
-            assertEquals(0.5d, tdigest.percentile(25), 0.0d);
+            assertEquals(0.75d, tdigest.percentile(25), 0.0d);
             assertTrue(AggregationInspectionHelper.hasValue(tdigest));
         });
 

server/src/test/java/org/elasticsearch/search/aggregations/metrics/TDigestStateTests.java

@@ -158,6 +158,8 @@ public class TDigestStateTests extends ESTestCase {
 
         assertEquals(fast, anotherFast);
         assertEquals(accurate, anotherAccurate);
+        assertNotEquals(fast, accurate);
+        assertNotEquals(anotherFast, anotherAccurate);
     }
 
     private static TDigestState writeToAndReadFrom(TDigestState state, TransportVersion version) throws IOException {
@@ -195,6 +197,7 @@ public class TDigestStateTests extends ESTestCase {
         assertEquals(serialized, state);
 
         TDigestState serializedBackwardsCompatible = writeToAndReadFrom(state, TransportVersion.V_8_500_000);
+        assertNotEquals(serializedBackwardsCompatible, state);
         assertEquals(serializedBackwardsCompatible, backwardsCompatible);
     }
 }

x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/aggregations/metrics/TDigestPreAggregatedPercentilesAggregatorTests.java

@@ -88,10 +88,10 @@ public class TDigestPreAggregatedPercentilesAggregatorTests extends AggregatorTe
         }, hdr -> {
             // assertEquals(4L, hdr.state.getTotalCount());
             double approximation = 0.05d;
-            assertEquals(15.0d, hdr.percentile(25), approximation);
+            assertEquals(17.5d, hdr.percentile(25), approximation);
             assertEquals(30.0d, hdr.percentile(50), approximation);
-            assertEquals(50.0d, hdr.percentile(75), approximation);
-            assertEquals(60.0d, hdr.percentile(99), approximation);
+            assertEquals(45.0d, hdr.percentile(75), approximation);
+            assertEquals(59.4d, hdr.percentile(99), approximation);
             assertTrue(AggregationInspectionHelper.hasValue(hdr));
         });
     }
@@ -105,9 +105,9 @@ public class TDigestPreAggregatedPercentilesAggregatorTests extends AggregatorTe
         }, hdr -> {
             // assertEquals(16L, hdr.state.getTotalCount());
             double approximation = 0.05d;
-            assertEquals(15.0d, hdr.percentile(25), approximation);
+            assertEquals(17.5d, hdr.percentile(25), approximation);
             assertEquals(30.0d, hdr.percentile(50), approximation);
-            assertEquals(50.0d, hdr.percentile(75), approximation);
+            assertEquals(45.0d, hdr.percentile(75), approximation);
             assertEquals(60.0d, hdr.percentile(99), approximation);
             assertTrue(AggregationInspectionHelper.hasValue(hdr));
         });

x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/boxplot/BoxplotAggregatorTests.java

@@ -106,9 +106,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, boxplot -> {
             assertEquals(2, boxplot.getMin(), 0);
             assertEquals(10, boxplot.getMax(), 0);
-            assertEquals(2, boxplot.getQ1(), 0);
+            assertEquals(2.25, boxplot.getQ1(), 0);
             assertEquals(3.5, boxplot.getQ2(), 0);
-            assertEquals(5, boxplot.getQ3(), 0);
+            assertEquals(4.75, boxplot.getQ3(), 0);
         });
     }
 
@@ -123,9 +123,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, boxplot -> {
             assertEquals(2, boxplot.getMin(), 0);
             assertEquals(10, boxplot.getMax(), 0);
-            assertEquals(2, boxplot.getQ1(), 0);
+            assertEquals(2.25, boxplot.getQ1(), 0);
             assertEquals(3.5, boxplot.getQ2(), 0);
-            assertEquals(5, boxplot.getQ3(), 0);
+            assertEquals(4.75, boxplot.getQ3(), 0);
         });
     }
 
@@ -141,9 +141,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, boxplot -> {
             assertEquals(2, boxplot.getMin(), 0);
             assertEquals(10, boxplot.getMax(), 0);
-            assertEquals(2, boxplot.getQ1(), 0);
+            assertEquals(2.25, boxplot.getQ1(), 0);
             assertEquals(3.5, boxplot.getQ2(), 0);
-            assertEquals(5, boxplot.getQ3(), 0);
+            assertEquals(4.75, boxplot.getQ3(), 0);
         });
     }
 
@@ -159,9 +159,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, boxplot -> {
             assertEquals(2, boxplot.getMin(), 0);
             assertEquals(10, boxplot.getMax(), 0);
-            assertEquals(2, boxplot.getQ1(), 0);
+            assertEquals(2.25, boxplot.getQ1(), 0);
             assertEquals(3.5, boxplot.getQ2(), 0);
-            assertEquals(5, boxplot.getQ3(), 0);
+            assertEquals(4.75, boxplot.getQ3(), 0);
         });
     }
 
@@ -274,9 +274,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
             InternalBoxplot boxplot = (InternalBoxplot) histo.getBuckets().get(0).getAggregations().asMap().get("boxplot");
             assertEquals(1, boxplot.getMin(), 0);
             assertEquals(3, boxplot.getMax(), 0);
-            assertEquals(1, boxplot.getQ1(), 0);
+            assertEquals(1.5, boxplot.getQ1(), 0);
             assertEquals(2, boxplot.getQ2(), 0);
-            assertEquals(3, boxplot.getQ3(), 0);
+            assertEquals(2.5, boxplot.getQ3(), 0);
 
             assertNotNull(histo.getBuckets().get(1).getAggregations().asMap().get("boxplot"));
             boxplot = (InternalBoxplot) histo.getBuckets().get(1).getAggregations().asMap().get("boxplot");
@@ -290,9 +290,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
             boxplot = (InternalBoxplot) histo.getBuckets().get(2).getAggregations().asMap().get("boxplot");
             assertEquals(21, boxplot.getMin(), 0);
             assertEquals(23, boxplot.getMax(), 0);
-            assertEquals(21, boxplot.getQ1(), 0);
+            assertEquals(21.5, boxplot.getQ1(), 0);
             assertEquals(22, boxplot.getQ2(), 0);
-            assertEquals(23, boxplot.getQ3(), 0);
+            assertEquals(22.5, boxplot.getQ3(), 0);
         }, new AggTestConfig(histogram, fieldType));
     }
 
@@ -310,14 +310,14 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, (Consumer<InternalBoxplot>) boxplot -> {
             assertEquals(1, boxplot.getMin(), 0);
             assertEquals(5, boxplot.getMax(), 0);
-            assertEquals(1.75, boxplot.getQ1(), 0);
+            assertEquals(2, boxplot.getQ1(), 0);
             assertEquals(3, boxplot.getQ2(), 0);
-            assertEquals(4.25, boxplot.getQ3(), 0);
+            assertEquals(4, boxplot.getQ3(), 0);
             assertEquals("0001.0", boxplot.getMinAsString());
             assertEquals("0005.0", boxplot.getMaxAsString());
-            assertEquals("0001.8", boxplot.getQ1AsString());
+            assertEquals("0002.0", boxplot.getQ1AsString());
             assertEquals("0003.0", boxplot.getQ2AsString());
-            assertEquals("0004.2", boxplot.getQ3AsString());
+            assertEquals("0004.0", boxplot.getQ3AsString());
         }, new AggTestConfig(aggregationBuilder, fieldType));
     }
 
@@ -359,9 +359,9 @@ public class BoxplotAggregatorTests extends AggregatorTestCase {
         }, (Consumer<InternalBoxplot>) boxplot -> {
             assertEquals(2, boxplot.getMin(), 0);
             assertEquals(8, boxplot.getMax(), 0);
-            assertEquals(2, boxplot.getQ1(), 0);
+            assertEquals(3.5, boxplot.getQ1(), 0);
             assertEquals(5, boxplot.getQ2(), 0);
-            assertEquals(8, boxplot.getQ3(), 0);
+            assertEquals(6.5, boxplot.getQ3(), 0);
         }, new AggTestConfig(aggregationBuilder, fieldType));
     }
 

x-pack/plugin/build.gradle

@@ -89,6 +89,13 @@ tasks.named("yamlRestTestV7CompatTest").configure {
     'aggregate-metrics/90_tsdb_mappings/aggregate_double_metric with wrong time series mappings',
     'analytics/histogram/histogram with wrong time series mappings',
     'analytics/histogram/histogram with time series mappings',
+    'analytics/boxplot/Basic Search',
+    'analytics/boxplot/Search With Runtime Field',
+    'analytics/boxplot/Search With Missing',
+    'analytics/moving_percentile/Basic Search TDigest',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc given actual_field is int',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc include curve',
     'ml/evaluate_data_frame/Test classification auc_roc',
     'ml/evaluate_data_frame/Test classification auc_roc with default top_classes_field',
   ].join(',')

x-pack/plugin/sql/qa/server/src/main/resources/agg.csv-spec

@@ -63,7 +63,7 @@ SELECT gender, PERCENTILE(emp_no, 92.45) p1, PERCENTILE(emp_no, 91) p2 FROM test
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.745             |10018.599999999999
-F                    |10098.0085            |10096.119999999999
+F                    |10096.336             |10094.48
 M                    |10091.393             |10090.37       
 ;
 
@@ -72,8 +72,8 @@ SELECT gender, PERCENTILE(emp_no, 91) p1, PERCENTILE(emp_no, 89) p2 FROM test_em
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.599999999999    |10018.4        
-F                    |10096.119999999999    |10093.74       
-M                    |10090.37              |10086.92       
+F                    |10094.48              |10093.74       
+M                    |10090.37              |10085.84       
 ;
 
 multiplePercentilesWithComma
@@ -81,7 +81,7 @@ SELECT gender, PERCENTILE(emp_no, 85.7) p1, PERCENTILE(emp_no, 94.3) p2 FROM tes
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.070000000002    |10018.929999999998
-F                    |10091.343             |10098.619         
+F                    |10090.272             |10098.619         
 M                    |10084.349             |10093.502         
 ;
 
@@ -1208,9 +1208,9 @@ GROUP BY gender ORDER BY gender;
 
    percentile    | percentile_rank  |    gender
 -----------------+------------------+---------------
-86857.79999999999|32.69659025378865 |null
-94042.92000000001|37.03569653103581 |F
-87348.36         |44.337514210592246|M
+85906.31999999999|32.69659025378865 |null
+93503.28         |37.03569653103581 |F
+86754.72         |44.337514210592246|M
 ;
 
 extendedStatsAggregateFunctionsWithScalars

x-pack/plugin/sql/qa/server/src/main/resources/columns.csv-spec

@@ -12,7 +12,7 @@ columnDetectionOverride
 SELECT gender, FLOOR(PERCENTILE(emp_no, 97.76)) p1 FROM test_emp GROUP BY gender ORDER BY gender DESC;
 
 gender:s             | p1:l
-M                    | 10096
+M                    | 10095
 F                    | 10099
-null                 | 10019
+null                 | 10018
 ;

x-pack/plugin/sql/qa/server/src/main/resources/docs/docs.csv-spec

@@ -1514,12 +1514,12 @@ SELECT languages, PERCENTILE(salary, 95) AS "95th" FROM emp
 
    languages   |      95th       
 ---------------+-----------------
-null           |74999.0          
-1              |72790.5          
-2              |71924.70000000001
-3              |73638.25         
-4              |72115.59999999999
-5              |61071.7       
+null           |74482.4          
+1              |71122.8          
+2              |70271.4
+3              |71926.0          
+4              |69352.15
+5              |56371.0       
 // end::aggPercentile
 ;
 
@@ -1531,12 +1531,12 @@ SELECT languages, PERCENTILE(salary / 12.0, 95) AS "95th" FROM emp
 
    languages   |       95th
 ---------------+------------------
-null           |6249.916666666667
-1              |6065.875
-2              |5993.725
-3              |6136.520833333332
-4              |6009.633333333332
-5              |5089.3083333333325
+null           |6206.866666666667
+1              |5926.9
+2              |5855.949999999999
+3              |5993.833333333333
+4              |5779.345833333333
+5              |4697.583333333333
 // end::aggPercentileScalars
 ;
 
@@ -1549,14 +1549,14 @@ SELECT
 FROM emp 
 GROUP BY languages;
 
-   languages   | 97.3_TDigest  |   97.3_HDR    
----------------+---------------+---------------
-null           |74999.0        |74992.0        
-1              |73717.0        |73712.0        
-2              |73530.238      |69936.0        
-3              |74970.0        |74992.0        
-4              |74572.0        |74608.0        
-5              |66117.118      |56368.0        
+   languages   | 97.3_TDigest    |   97.3_HDR    
+---------------+-----------------+---------------
+null           |74720.036        |74992.0        
+1              |72316.132        |73712.0        
+2              |71792.436        |69936.0        
+3              |73326.23999999999|74992.0        
+4              |71753.281        |74608.0        
+5              |61176.16000000001|56368.0        
 // end::aggPercentileWithPercentileConfig
 ;
 
@@ -1571,7 +1571,7 @@ null           |73.65766569962062
 2              |88.88005607010643
 3              |79.43662623295829
 4              |85.70446389643493
-5              |100.0      
+5              |96.79075152940749      
 // end::aggPercentileRank
 ;
 
@@ -1607,7 +1607,7 @@ null           |73.66          |80.0
 2              |88.88          |89.47          
 3              |79.44          |76.47          
 4              |85.7           |83.33          
-5              |100.0          |95.24          
+5              |96.79          |95.24          
 // end::aggPercentileRankWithPercentileConfig
 ;
 

x-pack/plugin/sql/qa/server/src/main/resources/field-alias.csv-spec

@@ -71,7 +71,7 @@ SELECT extra.info.gender AS gender, PERCENTILE(emp_no, 92.45) p1, PERCENTILE(emp
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.745             |10018.599999999999
-F                    |10098.0085            |10096.119999999999
+F                    |10096.336             |10094.48
 M                    |10091.393             |10090.37       
 ;
 
@@ -80,8 +80,8 @@ SELECT extra.info.gender AS gender, PERCENTILE(emp_no, 91) p1, PERCENTILE(emp_no
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.599999999999    |10018.4        
-F                    |10096.119999999999    |10093.74       
-M                    |10090.37              |10086.92       
+F                    |10094.48              |10093.74       
+M                    |10090.37              |10085.84       
 ;
 
 multiplePercentilesWithComma
@@ -89,7 +89,7 @@ SELECT extra.info.gender AS gender, PERCENTILE(emp_no, 85.7) p1, PERCENTILE(emp_
 
 gender:s             | p1:d                 | p2:d
 null                 |10018.070000000002    |10018.929999999998
-F                    |10091.343             |10098.619         
+F                    |10090.272             |10098.619         
 M                    |10084.349             |10093.502         
 ;
 

x-pack/plugin/sql/qa/server/src/main/resources/fulltext.csv-spec

@@ -184,12 +184,12 @@ SELECT MIN(salary) min, MAX(salary) max, ROUND(AVG(salary)) avg, gender g, COUNT
 multiAggWithPercentileAndMultiQuery
 SELECT languages, PERCENTILE(salary, 95) "95th", ROUND(PERCENTILE_RANK(salary, 65000)) AS rank, MAX(salary), MIN(salary), COUNT(*) c FROM test_emp WHERE QUERY('A*','default_field=first_name') OR QUERY('B*', 'default_field=first_name') OR languages IS NULL GROUP BY languages;
 
-  languages:bt |    95th:d     |    rank:d     | MAX(salary):i | MIN(salary):i |      c:l
----------------+---------------+---------------+---------------+---------------+---------------
-null           |74999          |74             |74999          |28336          |10
-2              |44307          |100            |44307          |29175          |3
-3              |65030          |100            |65030          |38376          |4
-5              |66817          |100            |66817          |37137          |4
+  languages:bt |    95th:d        |    rank:d     | MAX(salary):i | MIN(salary):i |      c:l
+---------------+------------------+---------------+---------------+---------------+---------------
+null           |74482.4           |74             |74999          |28336          |10
+2              |43213.299999999996|100            |44307          |29175          |3
+3              |64325.75          |87.0            |65030          |38376          |4
+5              |65250.09999999999 |83.0            |66817          |37137          |4
 ;
 
 multiAggWithStatsAndMatrixStatsAndMultiQuery

x-pack/plugin/sql/qa/server/src/main/resources/slow/frozen.csv-spec

@@ -36,7 +36,7 @@ SELECT gender, PERCENTILE(emp_no, 92.45) p1 FROM FROZEN frozen_emp GROUP BY gend
 
 gender:s             | p1:d
 null                 |10018.745
-F                    |10098.0085
+F                    |10096.336
 M                    |10091.393
 ;
 

x-pack/plugin/sql/qa/server/src/main/resources/unsigned-long.csv-spec

@@ -255,9 +255,9 @@ GROUP BY gender ORDER BY gender;
 
    percentile    | percentile_rank  |    gender
 -----------------+------------------+---------------
-86857.79999999999|32.69659025378865 |null
-94042.92000000001|37.03569653103581 |F
-87348.36         |44.337514210592246|M
+85906.32         |32.69659025378865 |null
+93503.28         |37.03569653103581 |F
+86754.72         |44.337514210592246|M
 ;
 
 extendedStatsAggregateFunctionsWithScalars
@@ -348,12 +348,12 @@ SELECT languages, PERCENTILE(salary_ul, 95) AS "95th" FROM test_emp_copy GROUP B
 
    languages:bt|      95th:d
 ---------------+-----------------
-null           |74999.0
-1              |72790.5
-2              |71924.70000000001
-3              |73638.25
-4              |72115.59999999999
-5              |61071.7
+null           |74482.4
+1              |71122.8
+2              |70271.4
+3              |71926.0
+4              |69352.15
+5              |56371.0
 ;
 
 mathPowerNegative
@@ -567,8 +567,8 @@ ORDER BY status;
 
      percentile:d    | percentile_rank:d |    status:s
 ---------------------+-------------------+---------------
-1.8836190713044468E19|0.0                |Error
-1.7957483822449326E19|26.644793296251386 |OK
+1.8618876151530934E19|0.0                |Error
+1.7923220123812004E19|26.644793296251386 |OK
 ;
 
 extendedStatsAggregateFunctionsWithScalarsLargeInput
@@ -639,8 +639,8 @@ SELECT status, PERCENTILE(bytes_in, 95) AS "95th" FROM logs_unsigned_long GROUP
 
     status:s   |        95th:d
 ---------------+---------------------
-Error          |1.8345360876889252E19
-OK             |1.8295214210102768E19
+Error          |1.666616785254379E19
+OK             |1.8196840183187124E19
 ;
 
 mathPowerNegativeLargeInput

x-pack/plugin/sql/qa/server/src/main/resources/wildcard.csv-spec

@@ -80,15 +80,15 @@ SELECT CONCAT(SUBSTRING(wildcard_name, 0, 1), SUBSTRING(wildcard_name, LOCATE('
 ---------------+---------------+---------------
 SP             |10024.0        |2              
 SF             |10089.0        |3              
-PB             |10080.0        |3              
+PB             |10075.92       |3              
 MS             |10054.0        |4              
 MB             |10074.0        |2              
-KM             |10085.0        |2              
+KM             |10082.6        |2              
 GD             |10075.0        |2              
-CB             |10068.0        |2              
-BM             |10058.0        |2              
+CB             |10066.47       |2              
+BM             |10056.95       |2              
 BB             |10060.0        |2              
-AP             |10062.0        |2              
+AP             |10060.32       |2              
                |10039.0        |10             
 ;
 

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/analytics/boxplot.yml

@@ -55,9 +55,9 @@ setup:
 
   - match: { aggregations.plot.min: 2.0 }
   - match: { aggregations.plot.max: 15.0 }
-  - match: { aggregations.plot.q1: 2.75 }
+  - match: { aggregations.plot.q1: 3.0 }
   - match: { aggregations.plot.q2: 5.0 }
-  - match: { aggregations.plot.q3: 11.25 }
+  - match: { aggregations.plot.q3: 10.0 }
 
 ---
 "Search With Runtime Field":
@@ -84,9 +84,9 @@ setup:
 
   - match: { aggregations.plot.min: 20.0 }
   - match: { aggregations.plot.max: 10000.0 }
-  - match: { aggregations.plot.q1: 27.5 }
+  - match: { aggregations.plot.q1: 30.0 }
   - match: { aggregations.plot.q2: 50.0 }
-  - match: { aggregations.plot.q3: 2612.5 }
+  - match: { aggregations.plot.q3: 150.0 }
   - match: { aggregations.plot.lower: 20.0 }
   - match: { aggregations.plot.upper: 150.0 }
 
@@ -106,8 +106,8 @@ setup:
 
   - match: { aggregations.plot.min: 2.0 }
   - match: { aggregations.plot.max: 10000.0 }
-  - match: { aggregations.plot.q1: 3 }
+  - match: { aggregations.plot.q1: 3.5 }
   - match: { aggregations.plot.q2: 7.5 }
-  - match: { aggregations.plot.q3: 15.0 }
+  - match: { aggregations.plot.q3: 13.75 }
   - match: { aggregations.plot.lower: 2.0 }
   - match: { aggregations.plot.upper: 15.0 }

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/analytics/moving_percentiles.yml

@@ -60,6 +60,7 @@ setup:
               values: [0.1, 0.5, 1, 2, 4, 10]
               counts: [1, 4, 5, 4, 5, 1]
 
+
 ---
 shift:
   - do:
@@ -91,13 +92,14 @@ shift:
   - match: { aggregations.histo.buckets.0.moving_percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.1.key_as_string: "2017-01-02T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.1.doc_count: 2 }
-  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 5.0 }
+  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.1.moving_percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.2.key_as_string: "2017-01-03T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.2.doc_count: 1 }
-  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  7.0 }
+  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  4.0 }
   - match: { aggregations.histo.buckets.2.moving_percentiles.values.0.value: 4.0 }
 
+
 ---
 default choses tdigest:
   - do:
@@ -128,15 +130,16 @@ default choses tdigest:
   - is_false: aggregations.histo.buckets.0.moving_percentiles
   - match: { aggregations.histo.buckets.1.key_as_string: "2017-01-02T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.1.doc_count: 2 }
-  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 5.0 }
+  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.1.moving_percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.2.key_as_string: "2017-01-03T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.2.doc_count: 1 }
-  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  7.0 }
+  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  4.0 }
   - match: { aggregations.histo.buckets.2.moving_percentiles.values.0.value: 4.0 }
 
+
 ---
-tdest explicitly:
+tdigest explicitly:
   - do:
       search:
         index: "foo"
@@ -166,11 +169,11 @@ tdest explicitly:
   - is_false: aggregations.histo.buckets.0.moving_percentiles
   - match: { aggregations.histo.buckets.1.key_as_string: "2017-01-02T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.1.doc_count: 2 }
-  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 5.0 }
+  - match: { aggregations.histo.buckets.1.percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.1.moving_percentiles.values.0.value: 4.0 }
   - match: { aggregations.histo.buckets.2.key_as_string: "2017-01-03T00:00:00.000Z" }
   - match: { aggregations.histo.buckets.2.doc_count: 1 }
-  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  7.0 }
+  - match: { aggregations.histo.buckets.2.percentiles.values.0.value:  4.0 }
   - match: { aggregations.histo.buckets.2.moving_percentiles.values.0.value: 4.0 }
 
 ---

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/ml/evaluate_data_frame.yml

@@ -207,7 +207,7 @@ setup:
               }
             }
           }
-  - match: { outlier_detection.auc_roc.value: 0.9919403846153846 }
+  - match: { outlier_detection.auc_roc.value: 0.99995 }
   - is_false: outlier_detection.auc_roc.curve
 
 ---
@@ -227,7 +227,7 @@ setup:
               }
             }
           }
-  - match: { outlier_detection.auc_roc.value: 0.9919403846153846 }
+  - match: { outlier_detection.auc_roc.value: 0.99995 }
   - is_false: outlier_detection.auc_roc.curve
 
 ---
@@ -247,7 +247,7 @@ setup:
               }
             }
           }
-  - match: { outlier_detection.auc_roc.value: 0.9919403846153846 }
+  - match: { outlier_detection.auc_roc.value: 0.99995 }
   - is_true: outlier_detection.auc_roc.curve
 
 ---
@@ -722,7 +722,7 @@ setup:
               }
             }
           }
-  - match: { classification.auc_roc.value: 0.77541527618109091 }
+  - match: { classification.auc_roc.value: 0.8612749970197677 }
   - is_false: classification.auc_roc.curve
 ---
 "Test classification auc_roc with default top_classes_field":
@@ -742,7 +742,7 @@ setup:
               }
             }
           }
-  - match: { classification.auc_roc.value: 0.77541527618109091 }
+  - match: { classification.auc_roc.value: 0.8612749970197677 }
   - is_false: classification.auc_roc.curve
 ---
 "Test classification accuracy with missing predicted_field":

x-pack/plugin/transform/qa/single-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/transform/integration/TransformPivotRestSpecialCasesIT.java

@@ -226,7 +226,7 @@ public class TransformPivotRestSpecialCasesIT extends TransformRestTestCase {
         )).get(0);
 
         assertEquals(28.0, (double) percentiles.get("50"), 0.000001);
-        assertEquals(77.0, (double) percentiles.get("99"), 0.000001);
+        assertEquals(76.12, (double) percentiles.get("99"), 0.000001);
 
         searchResult = getAsMap(transformIndex + "/_search?q=host:host-3");
         assertEquals(1, XContentMapValues.extractValue("hits.total.value", searchResult));

x-pack/qa/xpack-prefix-rest-compat/build.gradle

@@ -9,9 +9,6 @@
 
 import org.elasticsearch.gradle.Version
 import org.elasticsearch.gradle.VersionProperties
-import org.elasticsearch.gradle.internal.info.BuildParams
-import org.elasticsearch.gradle.internal.test.rest.CopyRestTestsTask
-import org.elasticsearch.gradle.util.GradleUtils
 
 apply plugin: 'elasticsearch.legacy-yaml-rest-compat-test'
 
@@ -190,5 +187,8 @@ tasks.named("yamlRestTestV7CompatTest").configure {
   systemProperty 'tests.rest.blacklist', [
     'ml/evaluate_data_frame/Test classification auc_roc',
     'ml/evaluate_data_frame/Test classification auc_roc with default top_classes_field',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc include curve',
+    'ml/evaluate_data_frame/Test outlier_detection auc_roc given actual_field is int',
   ].join(',')
 }