|
|
@@ -56,18 +56,20 @@ coordinating nodes.
|
|
|
(<<static-cluster-setting,Static>>) The maximum inference cache size allowed.
|
|
|
The inference cache exists in the JVM heap on each ingest node. The cache
|
|
|
affords faster processing times for the `inference` processor. The value can be
|
|
|
-a static byte sized value (i.e. "2gb") or a percentage of total allocated heap.
|
|
|
-The default is "40%". See also <<model-inference-circuit-breaker>>.
|
|
|
+a static byte sized value (such as `2gb`) or a percentage of total allocated
|
|
|
+heap. Defaults to `40%`. See also <<model-inference-circuit-breaker>>.
|
|
|
|
|
|
[[xpack-interference-model-ttl]]
|
|
|
// tag::interference-model-ttl-tag[]
|
|
|
`xpack.ml.inference_model.time_to_live` {ess-icon}::
|
|
|
-(<<static-cluster-setting,Static>>) The time to live (TTL) for models in the
|
|
|
-inference model cache. The TTL is calculated from last access. The `inference`
|
|
|
-processor attempts to load the model from cache. If the `inference` processor
|
|
|
-does not receive any documents for the duration of the TTL, the referenced model
|
|
|
-is flagged for eviction from the cache. If a document is processed later, the
|
|
|
-model is again loaded into the cache. Defaults to `5m`.
|
|
|
+(<<static-cluster-setting,Static>>) The time to live (TTL) for trained models in
|
|
|
+the inference model cache. The TTL is calculated from last access. Users of the
|
|
|
+cache (such as the inference processor or inference aggregator) cache a model on
|
|
|
+its first use and reset the TTL on every use. If a cached model is not accessed
|
|
|
+for the duration of the TTL, it is flagged for eviction from the cache. If a
|
|
|
+document is processed later, the model is again loaded into the cache. To update
|
|
|
+this setting in {ess}, see
|
|
|
+{cloud}/ec-add-user-settings.html[Add {es} user settings]. Defaults to `5m`.
|
|
|
// end::interference-model-ttl-tag[]
|
|
|
|
|
|
`xpack.ml.max_inference_processors`::
|
|
|
@@ -77,40 +79,54 @@ adding an `inference` processor to a pipeline is disallowed. Defaults to `50`.
|
|
|
|
|
|
`xpack.ml.max_machine_memory_percent`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The maximum percentage of the machine's
|
|
|
-memory that {ml} may use for running analytics processes. (These processes are
|
|
|
-separate to the {es} JVM.) Defaults to `30` percent. The limit is based on the
|
|
|
-total memory of the machine, not current free memory. Jobs are not allocated to
|
|
|
-a node if doing so would cause the estimated memory use of {ml} jobs to exceed
|
|
|
-the limit. When the {operator-feature} is enabled, this setting can be updated
|
|
|
-only by operator users.
|
|
|
+memory that {ml} may use for running analytics processes. These processes are
|
|
|
+separate to the {es} JVM. The limit is based on the total memory of the machine,
|
|
|
+not current free memory. Jobs are not allocated to a node if doing so would
|
|
|
+cause the estimated memory use of {ml} jobs to exceed the limit. When the
|
|
|
+{operator-feature} is enabled, this setting can be updated only by operator
|
|
|
+users. The minimum value is `5`; the maximum value is `200`. Defaults to `30`.
|
|
|
++
|
|
|
+--
|
|
|
+TIP: Do not configure this setting to a value higher than the amount of memory
|
|
|
+left over after running the {es} JVM unless you have enough swap space to
|
|
|
+accommodate it and have determined this is an appropriate configuration for a
|
|
|
+specialist use case. The maximum setting value is for the special case where it
|
|
|
+has been determined that using swap space for {ml} jobs is acceptable. The
|
|
|
+general best practice is to not use swap on {es} nodes.
|
|
|
+
|
|
|
+--
|
|
|
|
|
|
`xpack.ml.max_model_memory_limit`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The maximum `model_memory_limit` property
|
|
|
-value that can be set for any job on this node. If you try to create a job with
|
|
|
-a `model_memory_limit` property value that is greater than this setting value,
|
|
|
-an error occurs. Existing jobs are not affected when you update this setting.
|
|
|
-For more information about the `model_memory_limit` property, see
|
|
|
-<<put-analysislimits>>.
|
|
|
+value that can be set for any {ml} jobs in this cluster. If you try to create a
|
|
|
+job with a `model_memory_limit` property value that is greater than this setting
|
|
|
+value, an error occurs. Existing jobs are not affected when you update this
|
|
|
+setting. If this setting is `0` or unset, there is no maximum
|
|
|
+`model_memory_limit` value. If there are no nodes that meet the memory
|
|
|
+requirements for a job, this lack of a maximum memory limit means it's possible
|
|
|
+to create jobs that cannot be assigned to any available nodes. For more
|
|
|
+information about the `model_memory_limit` property, see
|
|
|
+<<ml-put-job,Create {anomaly-jobs}>> or <<put-dfanalytics>>. Defaults to `0`.
|
|
|
|
|
|
[[xpack.ml.max_open_jobs]]
|
|
|
`xpack.ml.max_open_jobs`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The maximum number of jobs that can run
|
|
|
-simultaneously on a node. Defaults to `20`. In this context, jobs include both
|
|
|
-{anomaly-jobs} and {dfanalytics-jobs}. The maximum number of jobs is also
|
|
|
-constrained by memory usage. Thus if the estimated memory usage of the jobs
|
|
|
-would be higher than allowed, fewer jobs will run on a node. Prior to version
|
|
|
-7.1, this setting was a per-node non-dynamic setting. It became a cluster-wide
|
|
|
-dynamic setting in version 7.1. As a result, changes to its value after node
|
|
|
-startup are used only after every node in the cluster is running version 7.1 or
|
|
|
-higher. The maximum permitted value is `512`.
|
|
|
+simultaneously on a node. In this context, jobs include both {anomaly-jobs} and
|
|
|
+{dfanalytics-jobs}. The maximum number of jobs is also constrained by memory
|
|
|
+usage. Thus if the estimated memory usage of the jobs would be higher than
|
|
|
+allowed, fewer jobs will run on a node. Prior to version 7.1, this setting was a
|
|
|
+per-node non-dynamic setting. It became a cluster-wide dynamic setting in
|
|
|
+version 7.1. As a result, changes to its value after node startup are used only
|
|
|
+after every node in the cluster is running version 7.1 or higher. The minimum
|
|
|
+value is `1`; the maximum value is `512`. Defaults to `20`.
|
|
|
|
|
|
`xpack.ml.nightly_maintenance_requests_per_second`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The rate at which the nightly maintenance
|
|
|
task deletes expired model snapshots and results. The setting is a proxy to the
|
|
|
-<<docs-delete-by-query-throttle,requests_per_second>> parameter used in the
|
|
|
+<<docs-delete-by-query-throttle,`requests_per_second`>> parameter used in the
|
|
|
delete by query requests and controls throttling. When the {operator-feature} is
|
|
|
enabled, this setting can be updated only by operator users. Valid values must
|
|
|
-be greater than `0.0` or equal to `-1.0` where `-1.0` means a default value is
|
|
|
+be greater than `0.0` or equal to `-1.0`, where `-1.0` means a default value is
|
|
|
used. Defaults to `-1.0`
|
|
|
|
|
|
`xpack.ml.node_concurrent_job_allocations`::
|
|
|
@@ -134,19 +150,19 @@ enabled, this setting can be updated only by operator users.
|
|
|
|
|
|
`xpack.ml.max_anomaly_records`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The maximum number of records that are
|
|
|
-output per bucket. The default value is `500`.
|
|
|
+output per bucket. Defaults to `500`.
|
|
|
|
|
|
`xpack.ml.max_lazy_ml_nodes`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The number of lazily spun up {ml} nodes.
|
|
|
Useful in situations where {ml} nodes are not desired until the first {ml} job
|
|
|
-opens. It defaults to `0` and has a maximum acceptable value of `3`. If the
|
|
|
-current number of {ml} nodes is greater than or equal to this setting, it is
|
|
|
-assumed that there are no more lazy nodes available as the desired number
|
|
|
-of nodes have already been provisioned. If a job is opened and this setting has
|
|
|
-a value greater than zero and there are no nodes that can accept the job, the
|
|
|
-job stays in the `OPENING` state until a new {ml} node is added to the cluster
|
|
|
-and the job is assigned to run on that node. When the {operator-feature} is
|
|
|
-enabled, this setting can be updated only by operator users.
|
|
|
+opens. If the current number of {ml} nodes is greater than or equal to this
|
|
|
+setting, it is assumed that there are no more lazy nodes available as the
|
|
|
+desired number of nodes have already been provisioned. If a job is opened and
|
|
|
+this setting has a value greater than zero and there are no nodes that can
|
|
|
+accept the job, the job stays in the `OPENING` state until a new {ml} node is
|
|
|
+added to the cluster and the job is assigned to run on that node. When the
|
|
|
+{operator-feature} is enabled, this setting can be updated only by operator
|
|
|
+users. Defaults to `0`.
|
|
|
+
|
|
|
IMPORTANT: This setting assumes some external process is capable of adding {ml}
|
|
|
nodes to the cluster. This setting is only useful when used in conjunction with
|
|
|
@@ -155,65 +171,67 @@ such an external process.
|
|
|
`xpack.ml.max_ml_node_size`::
|
|
|
(<<cluster-update-settings,Dynamic>>)
|
|
|
The maximum node size for {ml} nodes in a deployment that supports automatic
|
|
|
-cluster scaling. Defaults to `0b`, which means this value is ignored. If you set
|
|
|
-it to the maximum possible size of future {ml} nodes, when a {ml} job is
|
|
|
-assigned to a lazy node it can check (and fail quickly) when scaling cannot
|
|
|
-support the size of the job. When the {operator-feature} is enabled, this
|
|
|
-setting can be updated only by operator users.
|
|
|
+cluster scaling. If you set it to the maximum possible size of future {ml} nodes,
|
|
|
+when a {ml} job is assigned to a lazy node it can check (and fail quickly) when
|
|
|
+scaling cannot support the size of the job. When the {operator-feature} is
|
|
|
+enabled, this setting can be updated only by operator users. Defaults to `0b`,
|
|
|
+which means it will be assumed that automatic cluster scaling can add arbitrarily large nodes to the cluster.
|
|
|
|
|
|
`xpack.ml.persist_results_max_retries`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The maximum number of times to retry bulk
|
|
|
indexing requests that fail while processing {ml} results. If the limit is
|
|
|
reached, the {ml} job stops processing data and its status is `failed`. When the
|
|
|
{operator-feature} is enabled, this setting can be updated only by operator
|
|
|
-users. Defaults to `20`. The maximum value for this setting is `50`.
|
|
|
+users. The minimum value is `0`; the maximum value is `50`. Defaults to `20`.
|
|
|
|
|
|
`xpack.ml.process_connect_timeout`::
|
|
|
(<<cluster-update-settings,Dynamic>>) The connection timeout for {ml} processes
|
|
|
-that run separately from the {es} JVM. Defaults to `10s`. Some {ml} processing
|
|
|
-is done by processes that run separately to the {es} JVM. When such processes
|
|
|
-are started they must connect to the {es} JVM. If such a process does not
|
|
|
-connect within the time period specified by this setting then the process is
|
|
|
-assumed to have failed. When the {operator-feature} is enabled, this setting can
|
|
|
-be updated only by operator users. Defaults to `10s`. The minimum value for this
|
|
|
-setting is `5s`.
|
|
|
+that run separately from the {es} JVM. When such processes are started they must
|
|
|
+connect to the {es} JVM. If the process does not connect within the time period
|
|
|
+specified by this setting then the process is assumed to have failed. When the
|
|
|
+{operator-feature} is enabled, this setting can be updated only by operator
|
|
|
+users. The minimum value is `5s`. Defaults to `10s`.
|
|
|
|
|
|
xpack.ml.use_auto_machine_memory_percent::
|
|
|
(<<cluster-update-settings,Dynamic>>) If this setting is `true`, the
|
|
|
`xpack.ml.max_machine_memory_percent` setting is ignored. Instead, the maximum
|
|
|
percentage of the machine's memory that can be used for running {ml} analytics
|
|
|
processes is calculated automatically and takes into account the total node size
|
|
|
-and the size of the JVM on the node. The default value is `false`. If this
|
|
|
-setting differs between nodes, the value on the current master node is heeded.
|
|
|
-When the {operator-feature} is enabled, this setting can be updated only by
|
|
|
-operator users.
|
|
|
+and the size of the JVM on the node. If this setting differs between nodes, the
|
|
|
+value on the current master node is heeded. When the {operator-feature} is
|
|
|
+enabled, this setting can be updated only by operator users. The default value
|
|
|
+is `false`.
|
|
|
+
|
|
|
-TIP: If you do not have dedicated {ml} nodes (that is to say, the node has
|
|
|
+--
|
|
|
+[IMPORTANT]
|
|
|
+====
|
|
|
+* If you do not have dedicated {ml} nodes (that is to say, the node has
|
|
|
multiple roles), do not enable this setting. Its calculations assume that {ml}
|
|
|
analytics are the main purpose of the node.
|
|
|
-+
|
|
|
-IMPORTANT: The calculation assumes that dedicated {ml} nodes have at least
|
|
|
+* The calculation assumes that dedicated {ml} nodes have at least
|
|
|
`256MB` memory reserved outside of the JVM. If you have tiny {ml}
|
|
|
nodes in your cluster, you shouldn't use this setting.
|
|
|
+====
|
|
|
+--
|
|
|
|
|
|
[discrete]
|
|
|
[[model-inference-circuit-breaker]]
|
|
|
==== {ml-cap} circuit breaker settings
|
|
|
|
|
|
`breaker.model_inference.limit`::
|
|
|
-(<<cluster-update-settings,Dynamic>>) Limit for the model inference breaker,
|
|
|
-which defaults to 50% of the JVM heap. If the parent circuit breaker is less
|
|
|
-than 50% of the JVM heap, it is bound to that limit instead. See
|
|
|
-<<circuit-breaker>>.
|
|
|
+(<<cluster-update-settings,Dynamic>>) The limit for the trained model circuit
|
|
|
+breaker. This value is defined as a percentage of the JVM heap. Defaults to
|
|
|
+`50%`. If the <<parent-circuit-breaker,parent circuit breaker>> is set to a
|
|
|
+value less than `50%`, this setting uses that value as its default instead.
|
|
|
|
|
|
`breaker.model_inference.overhead`::
|
|
|
-(<<cluster-update-settings,Dynamic>>) A constant that all accounting estimations
|
|
|
-are multiplied by to determine a final estimation. Defaults to 1. See
|
|
|
-<<circuit-breaker>>.
|
|
|
+(<<cluster-update-settings,Dynamic>>) A constant that all trained model
|
|
|
+estimations are multiplied by to determine a final estimation. See
|
|
|
+<<circuit-breaker>>. Defaults to `1`.
|
|
|
|
|
|
`breaker.model_inference.type`::
|
|
|
(<<static-cluster-setting,Static>>) The underlying type of the circuit breaker.
|
|
|
There are two valid options: `noop` and `memory`. `noop` means the circuit
|
|
|
breaker does nothing to prevent too much memory usage. `memory` means the
|
|
|
-circuit breaker tracks the memory used by inference models and can potentially
|
|
|
-break and prevent `OutOfMemory` errors. The default is `memory`.
|
|
|
+circuit breaker tracks the memory used by trained models and can potentially
|
|
|
+break and prevent `OutOfMemory` errors. The default value is `memory`.
|
Lisa Cawley