Browse Source

Support requests_per_second=-1 to mean no throttling in reindex

This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes #19089
Nik Everett 9 năm trước cách đây
mục cha
commit
d573541f66

+ 16 - 15
docs/reference/docs/delete-by-query.asciidoc

@@ -40,7 +40,7 @@ That will return something like this:
     "search": 0
   },
   "throttled_millis": 0,
-  "requests_per_second": "unlimited",
+  "requests_per_second": -1.0,
   "throttled_until_millis": 0,
   "total": 119,
   "failures" : [ ]
@@ -164,15 +164,15 @@ request. `timeout` controls how long each write request waits for unavailable
 shards to become available. Both work exactly how they work in the
 <<docs-bulk,Bulk API>>.
 
-`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
-and throttles the number of requests per second that the delete by query issues.
-The throttling is done waiting between bulk batches so that it can manipulate
-the scroll timeout. The wait time is the difference between the time it took the
-batch to complete and the time `requests_per_second * requests_in_the_batch`.
-Since the batch isn't broken into multiple bulk requests large batch sizes will
-cause Elasticsearch to create many requests and then wait for a while before
-starting the next set. This is "bursty" instead of "smooth". The default is
-`unlimited` which is also the only non-number value that it accepts.
+`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
+`1000`, etc) and throttles the number of requests per second that the delete-by-query
+issues or it can be set to `-1` to disabled throttling. The throttling is done
+waiting between bulk batches so that it can manipulate the scroll timeout. The
+wait time is the difference between the time it took the batch to complete and
+the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
+broken into multiple bulk requests large batch sizes will cause Elasticsearch
+to create many requests and then wait for a while before starting the next set.
+This is "bursty" instead of "smooth". The default is `-1`.
 
 [float]
 === Response body
@@ -326,14 +326,15 @@ using the `_rethrottle` API:
 
 [source,js]
 --------------------------------------------------
-POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=unlimited
+POST _delete_by_query/taskid:1/_rethrottle?requests_per_second=-1
 --------------------------------------------------
 // CONSOLE
 
 The `task_id` can be found using the tasks API above.
 
 Just like when setting it on the `_delete_by_query` API `requests_per_second`
-can be either `unlimited` to disable throttling or any decimal number like `1.7`
-or `12` to throttle to that level. Rethrottling that speeds up the query takes
-effect immediately but rethrotting that slows down the query will take effect
-on after completing the current batch. This prevents scroll timeouts.
+can be either `-1` to disable throttling or any decimal number
+like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
+query takes effect immediately but rethrotting that slows down the query will
+take effect on after completing the current batch. This prevents scroll
+timeouts.

+ 17 - 17
docs/reference/docs/reindex.asciidoc

@@ -39,7 +39,7 @@ That will return something like this:
     "search": 0
   },
   "throttled_millis": 0,
-  "requests_per_second": "unlimited",
+  "requests_per_second": -1.0,
   "throttled_until_millis": 0,
   "total": 120,
   "failures" : [ ]
@@ -439,15 +439,15 @@ request. `timeout` controls how long each write request waits for unavailable
 shards to become available. Both work exactly how they work in the
 <<docs-bulk,Bulk API>>.
 
-`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
-and throttles the number of requests per second that the reindex issues. The
-throttling is done waiting between bulk batches so that it can manipulate the
-scroll timeout. The wait time is the difference between the time it took the
-batch to complete and the time `requests_per_second * requests_in_the_batch`.
-Since the batch isn't broken into multiple bulk requests large batch sizes will
-cause Elasticsearch to create many requests and then wait for a while before
-starting the next set. This is "bursty" instead of "smooth". The default is
-`unlimited` which is also the only non-number value that it accepts.
+`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
+`1000`, etc) and throttles the number of requests per second that the reindex
+issues or it can be set to `-1` to disabled throttling. The throttling is done
+waiting between bulk batches so that it can manipulate the scroll timeout. The
+wait time is the difference between the time it took the batch to complete and
+the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
+broken into multiple bulk requests large batch sizes will cause Elasticsearch
+to create many requests and then wait for a while before starting the next set.
+This is "bursty" instead of "smooth". The default is `-1`.
 
 [float]
 [[docs-reindex-response-body]]
@@ -613,18 +613,18 @@ the `_rethrottle` API:
 
 [source,js]
 --------------------------------------------------
-POST _reindex/taskid:1/_rethrottle?requests_per_second=unlimited
+POST _reindex/taskid:1/_rethrottle?requests_per_second=-1
 --------------------------------------------------
 // CONSOLE
 
 The `task_id` can be found using the tasks API above.
 
-Just like when setting it on the `_reindex` API `requests_per_second` can be
-either `unlimited` to disable throttling or any decimal number like `1.7` or
-`12` to throttle to that level. Rethrottling that speeds up the query takes
-effect immediately but rethrotting that slows down the query will take effect
-on after completing the current batch. This prevents scroll timeouts.
-
+Just like when setting it on the `_reindex` API `requests_per_second`
+can be either `-1` to disable throttling or any decimal number
+like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
+query takes effect immediately but rethrotting that slows down the query will
+take effect on after completing the current batch. This prevents scroll
+timeouts.
 
 [float]
 === Reindex to change the name of a field

+ 16 - 16
docs/reference/docs/update-by-query.asciidoc

@@ -32,7 +32,7 @@ That will return something like this:
     "search": 0
   },
   "throttled_millis": 0,
-  "requests_per_second": "unlimited",
+  "requests_per_second": -1.0,
   "throttled_until_millis": 0,
   "total": 120,
   "failures" : [ ]
@@ -221,15 +221,15 @@ request. `timeout` controls how long each write request waits for unavailable
 shards to become available. Both work exactly how they work in the
 <<docs-bulk,Bulk API>>.
 
-`requests_per_second` can be set to any decimal number (`1.4`, `6`, `1000`, etc)
-and throttles the number of requests per second that the update by query issues.
-The throttling is done waiting between bulk batches so that it can manipulate
-the scroll timeout. The wait time is the difference between the time it took the
-batch to complete and the time `requests_per_second * requests_in_the_batch`.
-Since the batch isn't broken into multiple bulk requests large batch sizes will
-cause Elasticsearch to create many requests and then wait for a while before
-starting the next set. This is "bursty" instead of "smooth". The default is
-`unlimited` which is also the only non-number value that it accepts.
+`requests_per_second` can be set to any positive decimal number (`1.4`, `6`,
+`1000`, etc) and throttles the number of requests per second that the update-by-query
+issues or it can be set to `-1` to disabled throttling. The throttling is done
+waiting between bulk batches so that it can manipulate the scroll timeout. The
+wait time is the difference between the time it took the batch to complete and
+the time `requests_per_second * requests_in_the_batch`. Since the batch isn't
+broken into multiple bulk requests large batch sizes will cause Elasticsearch
+to create many requests and then wait for a while before starting the next set.
+This is "bursty" instead of "smooth". The default is `-1`.
 
 [float]
 [[docs-update-by-query-response-body]]
@@ -391,18 +391,18 @@ using the `_rethrottle` API:
 
 [source,js]
 --------------------------------------------------
-POST _update_by_query/taskid:1/_rethrottle?requests_per_second=unlimited
+POST _update_by_query/taskid:1/_rethrottle?requests_per_second=-1
 --------------------------------------------------
 // CONSOLE
 
 The `task_id` can be found using the tasks API above.
 
 Just like when setting it on the `_update_by_query` API `requests_per_second`
-can be either `unlimited` to disable throttling or any decimal number like `1.7`
-or `12` to throttle to that level. Rethrottling that speeds up the query takes
-effect immediately but rethrotting that slows down the query will take effect
-on after completing the current batch. This prevents scroll timeouts.
-
+can be either `-1` to disable throttling or any decimal number
+like `1.7` or `12` to throttle to that level. Rethrottling that speeds up the
+query takes effect immediately but rethrotting that slows down the query will
+take effect on after completing the current batch. This prevents scroll
+timeouts.
 
 [float]
 [[picking-up-a-new-property]]

+ 6 - 6
modules/reindex/src/main/java/org/elasticsearch/index/reindex/AbstractBaseReindexRestHandler.java

@@ -137,20 +137,20 @@ public abstract class AbstractBaseReindexRestHandler<
         if (requestsPerSecondString == null) {
             return null;
         }
-        if ("unlimited".equals(requestsPerSecondString)) {
-            return  Float.POSITIVE_INFINITY;
-        }
         float requestsPerSecond;
         try {
             requestsPerSecond = Float.parseFloat(requestsPerSecondString);
         } catch (NumberFormatException e) {
             throw new IllegalArgumentException(
-                    "[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling.", e);
+                    "[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.", e);
+        }
+        if (requestsPerSecond == -1) {
+            return Float.POSITIVE_INFINITY;
         }
         if (requestsPerSecond <= 0) {
-            // We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of "unlimited"
+            // We validate here and in the setters because the setters use "Float.POSITIVE_INFINITY" instead of -1
             throw new IllegalArgumentException(
-                    "[requests_per_second] must be a float greater than 0. Use \"unlimited\" to disable throttling.");
+                    "[requests_per_second] must be a float greater than 0. Use -1 to disable throttling.");
         }
         return requestsPerSecond;
     }

+ 2 - 2
modules/reindex/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

@@ -217,7 +217,7 @@ public class BulkByScrollTask extends CancellableTask {
             }
             builder.endObject();
             builder.timeValueField("throttled_millis", "throttled", throttled);
-            builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? "unlimited" : requestsPerSecond);
+            builder.field("requests_per_second", requestsPerSecond == Float.POSITIVE_INFINITY ? -1 : requestsPerSecond);
             if (reasonCancelled != null) {
                 builder.field("canceled", reasonCancelled);
             }
@@ -324,7 +324,7 @@ public class BulkByScrollTask extends CancellableTask {
         }
 
         /**
-         * The number of requests per second to which to throttle the request. 0 means unlimited.
+         * The number of requests per second to which to throttle the request. Float.POSITIVE_INFINITY means unlimited.
          */
         public float getRequestsPerSecond() {
             return requestsPerSecond;

+ 1 - 1
modules/reindex/src/test/java/org/elasticsearch/index/reindex/BulkByScrollTaskTests.java

@@ -285,7 +285,7 @@ public class BulkByScrollTaskTests extends ESTestCase {
     public void testXContentRepresentationOfUnlimitedRequestsPerSecon() throws IOException {
         XContentBuilder builder = JsonXContent.contentBuilder();
         task.getStatus().toXContent(builder, ToXContent.EMPTY_PARAMS);
-        assertThat(builder.string(), containsString("\"requests_per_second\":\"unlimited\""));
+        assertThat(builder.string(), containsString("\"requests_per_second\":-1"));
     }
 
     public void testPerfectlyThrottledBatchTime() {

+ 15 - 3
modules/reindex/src/test/resources/rest-api-spec/test/delete_by_query/20_validation.yaml

@@ -68,7 +68,7 @@
 ---
 "requests_per_second cannot be an empty string":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       delete_by_query:
         requests_per_second: ""
         index: test
@@ -79,7 +79,7 @@
 ---
 "requests_per_second cannot be negative":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       delete_by_query:
         requests_per_second: -12
         index: test
@@ -90,10 +90,22 @@
 ---
 "requests_per_second cannot be zero":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       delete_by_query:
         requests_per_second: 0
         index: test
         body:
           query:
             match_all: {}
+
+
+---
+"requests_per_second cannot be unlimited":
+  - do:
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
+      delete_by_query:
+        requests_per_second: unlimited
+        index: test
+        body:
+          query:
+            match_all: {}

+ 4 - 4
modules/reindex/src/test/resources/rest-api-spec/test/delete_by_query/70_throttle.yaml

@@ -40,7 +40,7 @@
   - lt: {throttled_millis: 4000}
 
 ---
-"requests_per_second supports unlimited which turns off throttling":
+"requests_per_second supports -1 which turns off throttling":
   - do:
       indices.create:
         index: test
@@ -69,7 +69,7 @@
       delete_by_query:
         index: test
         scroll_size: 1
-        requests_per_second: unlimited
+        requests_per_second: -1
         body:
           query:
             match_all: {}
@@ -79,7 +79,7 @@
   - match: {throttled_millis: 0}
 
 ---
-"Rethrottle":
+"Rethrottle to -1 which turns off throttling":
   # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
   # and a small batch size on the request
   - do:
@@ -121,7 +121,7 @@
 
   - do:
       reindex.rethrottle:
-        requests_per_second: unlimited
+        requests_per_second: -1
         task_id: $task
 
   - do:

+ 15 - 3
modules/reindex/src/test/resources/rest-api-spec/test/reindex/20_validation.yaml

@@ -152,7 +152,7 @@
 ---
 "requests_per_second cannot be an empty string":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       reindex:
         requests_per_second: ""
         body:
@@ -164,7 +164,7 @@
 ---
 "requests_per_second cannot be negative":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       reindex:
         requests_per_second: -12
         body:
@@ -176,7 +176,7 @@
 ---
 "requests_per_second cannot be zero":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       reindex:
         requests_per_second: 0
         body:
@@ -185,6 +185,18 @@
           dest:
             index: dest
 
+---
+"requests_per_second cannot be unlimited":
+  - do:
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
+      reindex:
+        requests_per_second: unlimited
+        body:
+          source:
+            index: test
+          dest:
+            index: dest
+
 ---
 "reindex without source gives useful error message":
   - do:

+ 4 - 4
modules/reindex/src/test/resources/rest-api-spec/test/reindex/70_throttle.yaml

@@ -50,7 +50,7 @@
   - is_false: task
 
 ---
-"requests_per_second supports unlimited to turn off throttling":
+"requests_per_second supports -1 to turn off throttling":
   - do:
       indices.create:
         index: source
@@ -81,7 +81,7 @@
 
   - do:
       reindex:
-        requests_per_second: unlimited
+        requests_per_second: -1
         body:
           source:
             index: source
@@ -97,7 +97,7 @@
   - is_false: task
 
 ---
-"Rethrottle":
+"Rethrottle to -1 which turns off throttling":
   # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
   # and a small batch size on the request
   - do:
@@ -143,7 +143,7 @@
 
   - do:
       reindex.rethrottle:
-        requests_per_second: unlimited
+        requests_per_second: -1
         task_id: $task
 
   - do:

+ 11 - 3
modules/reindex/src/test/resources/rest-api-spec/test/update_by_query/20_validation.yaml

@@ -76,7 +76,7 @@
 ---
 "requests_per_second cannot be an empty string":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       update_by_query:
         requests_per_second: ""
         index: test
@@ -84,7 +84,7 @@
 ---
 "requests_per_second cannot be negative":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       update_by_query:
         requests_per_second: -12
         index: test
@@ -92,7 +92,15 @@
 ---
 "requests_per_second cannot be zero":
   - do:
-      catch: /\[requests_per_second\] must be a float greater than 0. Use "unlimited" to disable throttling./
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
+      update_by_query:
+        requests_per_second: 0
+        index: test
+
+---
+"requests_per_second cannot be unlimited":
+  - do:
+      catch: /\[requests_per_second\] must be a float greater than 0. Use -1 to disable throttling./
       update_by_query:
         requests_per_second: 0
         index: test

+ 4 - 4
modules/reindex/src/test/resources/rest-api-spec/test/update_by_query/60_throttle.yaml

@@ -36,7 +36,7 @@
   - lt: {throttled_millis: 4000}
 
 ---
-"requests_per_second supports unlimited which turns off throttling":
+"requests_per_second supports -1 which turns off throttling":
   - do:
       indices.create:
         index: test
@@ -65,13 +65,13 @@
       update_by_query:
         index: test
         scroll_size: 1
-        requests_per_second: unlimited
+        requests_per_second: -1
   - match: {batches: 3}
   - match: {updated: 3}
   - match: {throttled_millis: 0}
 
 ---
-"Rethrottle":
+"Rethrottle to -1 which turns off throttling":
   # Throttling happens between each scroll batch so we need to control the size of the batch by using a single shard
   # and a small batch size on the request
   - do:
@@ -109,7 +109,7 @@
 
   - do:
       reindex.rethrottle:
-        requests_per_second: unlimited
+        requests_per_second: -1
         task_id: $task
 
   - do:

+ 1 - 1
rest-api-spec/src/main/resources/rest-api-spec/api/delete_by_query.json

@@ -195,7 +195,7 @@
         "requests_per_second": {
           "type": "float",
           "default": 0,
-          "description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
+          "description": "The throttle for this request in sub-requests per second. -1 means set no throttle."
         }
       }
     },

+ 1 - 1
rest-api-spec/src/main/resources/rest-api-spec/api/reindex.json

@@ -29,7 +29,7 @@
         "requests_per_second": {
           "type": "float",
           "default": 0,
-          "description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
+          "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
         }
       }
     },

+ 1 - 1
rest-api-spec/src/main/resources/rest-api-spec/api/reindex.rethrottle.json

@@ -15,7 +15,7 @@
         "requests_per_second": {
           "type": "float",
           "required": true,
-          "description": "The throttle to set on this request in sub-requests per second. 0 means set no throttle. As does \"unlimited\". Otherwise it must be a float."
+          "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
         }
       }
     },

+ 1 - 1
rest-api-spec/src/main/resources/rest-api-spec/api/update_by_query.json

@@ -203,7 +203,7 @@
         "requests_per_second": {
           "type": "float",
           "default": 0,
-          "description": "The throttle for this request in sub-requests per second. 0 means set no throttle."
+          "description": "The throttle to set on this request in sub-requests per second. -1 means set no throttle as does \"unlimited\" which is the only non-float this accepts."
         }
       }
     },