Browse Source

[DOCS] Add throttling based on configuration parameter (#56653)

Lisa Cawley 5 years ago
parent
commit
d7735d9ef2

+ 4 - 6
docs/reference/rest-api/common-parms.asciidoc

@@ -918,15 +918,13 @@ default value is `60s`.
 end::sync-time-delay[]
 
 tag::transform-settings[]
-Defines the settings of the {transform}.
+Defines optional {transform} settings.
 end::transform-settings[]
 
 tag::transform-settings-docs-per-second[]
-Set `docs-per-second` to throttle the rate transform is issuing search and
-index requests. Throttling slows down search requests by adding a wait time
-between searches. For the number of docs, the number of input documents is taken.
-Throttling can be disabled by setting it to `null`.
-The default is no throttling, value `null`.
+Specifies a limit on the number of input documents per second. This setting
+throttles the {transform} by adding a wait time between search requests. The 
+default value is `null`, which disables throttling.
 end::transform-settings-docs-per-second[]
 
 tag::transform-settings-max-page-search-size[]

+ 17 - 17
docs/reference/transform/apis/put-transform.asciidoc

@@ -128,6 +128,23 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=pivot-group-by]
 ====
 //End pivot
 
+//Begin settings
+`settings`::
+(Optional, object)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
++
+.Properties of `settings`
+[%collapsible%open]
+====
+`docs_per_second`:::
+(Optional, float)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
+`max_page_search_size`:::
+(Optional, integer)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
+====
+//End settings
+
 //Begin source
 `source`::
 (Required, object)
@@ -185,23 +202,6 @@ delays.
 ====
 //End sync
 
-//Begin settings
-`settings`::
-(Optional, object)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
-+
-.Properties of `settings`
-[%collapsible%open]
-====
-`docs_per_second`:::
-(Optional, float)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
-`max_page_search_size`:::
-(Optional, integer)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
-====
-//End settings
-
 
 [[put-transform-example]]
 ==== {api-examples-title}

+ 17 - 17
docs/reference/transform/apis/update-transform.asciidoc

@@ -102,6 +102,23 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=dest-pipeline]
 (Optional, <<time-units, time units>>)
 include::{docdir}/rest-api/common-parms.asciidoc[tag=frequency]
 
+//Begin settings
+`settings`::
+(Optional, object)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
++
+.Properties of `settings`
+[%collapsible%open]
+====
+`docs_per_second`:::
+(Optional, float)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
+`max_page_search_size`:::
+(Optional, integer)
+include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
+====
+//End settings
+
 //Begin source
 `source`::
 (Optional, object)
@@ -159,23 +176,6 @@ delays.
 ====
 //End sync
 
-//Begin settings
-`settings`::
-(Optional, object)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings]
-+
-.Properties of `settings`
-[%collapsible%open]
-====
-`docs_per_second`:::
-(Optional, float)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-docs-per-second]
-`max_page_search_size`:::
-(Optional, integer)
-include::{docdir}/rest-api/common-parms.asciidoc[tag=transform-settings-max-page-search-size]
-====
-//End settings
-
 [[update-transform-example]]
 ==== {api-examples-title}
 

+ 1 - 5
docs/reference/transform/checkpoints.asciidoc

@@ -49,11 +49,7 @@ requires enough resources to support both the composite aggregation search and
 the indexing of its results. 
 
 TIP: If the cluster experiences unsuitable performance degradation due to the
-{transform}, stop the {transform}. Consider whether you can apply a
-source query to the {transform} to reduce the scope of data it
-processes. Also consider whether the cluster has sufficient resources in place
-to support both the composite aggregation search and the indexing of its
-results.
+{transform}, stop the {transform} and refer to <<transform-performance>>.
 
 [discrete]
 [[ml-transform-checkpoint-errors]]

+ 16 - 1
docs/reference/transform/overview.asciidoc

@@ -72,4 +72,19 @@ If your {transform} must process a lot of historic data, it has high resource
 usage initially--particularly during the first checkpoint.
 
 For better performance, make sure that your search aggregations and queries are 
-optimized and that your {transform} is processing only necessary data.
+optimized and that your {transform} is processing only necessary data. Consider 
+whether you can apply a source query to the {transform} to reduce the scope of 
+data it processes. Also consider whether the cluster has sufficient resources in 
+place to support both the composite aggregation search and the indexing of its
+results.
+
+If you prefer to spread out the impact on your cluster (at the cost of a slower
+{transform}), you can throttle the rate at which it performs search and index
+requests. Set the `docs_per_second` limit when you <<put-transform,create>> or
+<<update-transform,update>> your {transform}. If you want to calculate the
+current rate, use the following information from the
+{ref}/get-transform-stats.html[get {transform} stats API]:
+```
+documents_processed / search_time_in_ms * 1000
+```
+