Autoscaling APIs not experimental (#67202)

We no longer regard the autoscaling APIs experimental though they are
only intended for use by ESS/ECE/ECK. This commit updates the docs
to reflect this and adds a minimal set of documentation for the
feature.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

docs/reference/autoscaling/apis/autoscaling-apis.asciidoc

@@ -1,9 +1,9 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[autoscaling-apis]]
 == Autoscaling APIs
 
-experimental[]
+include::../autoscaling-designed-for-note.asciidoc[]
 
 You can use the following APIs to perform autoscaling operations.
 

docs/reference/autoscaling/apis/delete-autoscaling-policy.asciidoc

@@ -1,12 +1,12 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[autoscaling-delete-autoscaling-policy]]
 === Delete autoscaling policy API
 ++++
 <titleabbrev>Delete autoscaling policy</titleabbrev>
 ++++
 
-experimental[]
+include::../autoscaling-designed-for-note.asciidoc[]
 
 Delete autoscaling policy.
 

docs/reference/autoscaling/apis/get-autoscaling-capacity.asciidoc

@@ -1,12 +1,12 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[autoscaling-get-autoscaling-capacity]]
 === Get autoscaling capacity API
 ++++
 <titleabbrev>Get autoscaling capacity</titleabbrev>
 ++++
 
-experimental[]
+include::../autoscaling-designed-for-note.asciidoc[]
 
 Get autoscaling capacity.
 
@@ -33,6 +33,196 @@ This API gets the current autoscaling capacity based on the configured
 autoscaling policy. This API will return information to size the cluster
 appropriately to the current workload.
 
+The `required_capacity` is calculated as the max of the `required_capacity`
+result of all individual deciders that are enabled for the policy.
+
+The operator should verify that the `current_nodes` match
+the operator's knowledge of the cluster to avoid making autoscaling decisions
+based on stale or incomplete information.
+
+The response contains decider-specific information you can use to diagnose how
+and why autoscaling determined a certain capacity was required. This information
+is provided for diagnosis only. Do not use this information to make autoscaling
+decisions.
+
+[role="child_attributes"]
+[[autoscaling-get-autoscaling-capacity-api-response-body]]
+==== {api-response-body-title}
+
+`policies`::
+(object)
+Contains the map of policy name to capacity result
++
+.Properties of `policies`
+[%collapsible%open]
+====
+`<policy_name>`::
+(object)
+Contains capacity information for the policy
++
+.Properties of `<policy_name>`
+[%collapsible%open]
+=====
+`required_capacity`::
+(object)
+Contains the required capacity for the policy.
++
+.Properties of `required_capacity`
+[%collapsible%open]
+======
+`node`::
+(object)
+Contains the minimum node sizes required per node, ensuring that individual
+shards or ML jobs can fit into a single node.
++
+.Properties of `node`
+[%collapsible%open]
+=======
+`storage`::
+(integer)
+Bytes of storage required per node.
+
+`memory`::
+(integer)
+Bytes of memory required per node.
+
+=======
+`total`::
+(object)
+Contains the total size required for the policy.
++
+.Properties of `total`
+[%collapsible%open]
+=======
+`storage`::
+(integer)
+Total bytes of storage required for the policy.
+
+`memory`::
+(integer)
+Total bytes of memory required for the policy.
+=======
+======
+`current_capacity`::
+(object)
+Contains the current capacity for nodes governed by the policy, i.e. the nodes
+that {es} bases its calculation on.
++
+.Properties of `current_capacity`
+[%collapsible%open]
+======
+`node`::
+(object)
+Contains the maximum sizes of nodes governed by the policy.
++
+.Properties of `node`
+[%collapsible%open]
+=======
+`storage`::
+(integer)
+Maximum bytes of storage of a node.
+
+`memory`::
+(integer)
+Maximum bytes of memory of a node.
+
+=======
+`total`::
+(object)
+Contains the current total storage and memory sizes for nodes governed by the policy.
++
+.Properties of `total`
+[%collapsible%open]
+=======
+`storage`::
+(integer)
+Current bytes of storage available for the policy.
+
+`memory`::
+(integer)
+Current bytes of memory available for the policy.
+=======
+======
+`current_nodes`::
+(array of objects)
+List of nodes used for capacity calculation.
++
+.Properties of elements in `current_nodes`
+[%collapsible%open]
+======
+`name`::
+(string)
+Name of the node.
+======
+`deciders`::
+(object)
+The capacity results from individual deciders, allowing insight into how the
+outer level `required_capacity` was calculated.
++
+.Properties of `deciders`
+[%collapsible%open]
+=======
+`<decider_name>`::
+(object)
+The capacity result for a specific decider enabled for the policy.
++
+.Properties of `<decider_name>`
+[%collapsible%open]
+========
+`required_capacity`::
+(object)
+Required capacity determined by the decider.
++
+.Properties of `required_capacity`
+[%collapsible%open]
+=========
+`node`::
+(object)
+Contains the minimum node sizes required per node, ensuring that individual
+shards or {ml} jobs can fit into a single node.
++
+.Properties of `node`
+[%collapsible%open]
+==========
+`storage`::
+(integer)
+Bytes of storage required per node.
+
+`memory`::
+(integer)
+Bytes of memory required per node.
+
+==========
+`total`::
+(object)
+Contains the total size required for the policy.
++
+.Properties of `total`
+[%collapsible%open]
+==========
+`storage`::
+(integer)
+Total bytes of storage required for the policy.
+
+`memory`::
+(integer)
+Total bytes of memory required for the policy.
+==========
+=========
+`reason_summary`::
+(string)
+Description of the basis for the decider's result.
+
+`reason_details`::
+(object)
+A per-decider structure containing details about the basis for the deciders' result.
+The contents should not be relied on for application purposes and are not subject
+to backwards compatibility guarantees.
+========
+=======
+=====
+====
+
 [[autoscaling-get-autoscaling-capacity-examples]]
 ==== {api-examples-title}
 

docs/reference/autoscaling/apis/get-autoscaling-policy.asciidoc

@@ -1,12 +1,12 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[autoscaling-get-autoscaling-policy]]
 === Get autoscaling policy API
 ++++
 <titleabbrev>Get autoscaling policy</titleabbrev>
 ++++
 
-experimental[]
+include::../autoscaling-designed-for-note.asciidoc[]
 
 Get autoscaling policy.
 

docs/reference/autoscaling/apis/put-autoscaling-policy.asciidoc

@@ -1,12 +1,12 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[autoscaling-put-autoscaling-policy]]
 === Put autoscaling policy API
 ++++
 <titleabbrev>Put autoscaling policy</titleabbrev>
 ++++
 
-experimental[]
+include::../autoscaling-designed-for-note.asciidoc[]
 
 Put autoscaling policy.
 
@@ -47,6 +47,7 @@ DELETE /_autoscaling/policy/name
 ==== {api-description-title}
 
 This API puts an autoscaling policy with the provided name.
+See <<autoscaling-deciders,Autoscaling Deciders>> for available deciders.
 
 [[autoscaling-put-autoscaling-policy-examples]]
 ==== {api-examples-title}

docs/reference/autoscaling/autoscaling-deciders.asciidoc

@@ -0,0 +1,24 @@
+[role="xpack"]
+[testenv="enterprise"]
+[[autoscaling-deciders]]
+== Autoscaling deciders
+
+<<autoscaling-reactive-storage-decider,Reactive storage decider>>::
+Estimates required storage capacity of current data set. Available for policies
+governing data nodes.
+
+<<autoscaling-proactive-storage-decider,Proactive storage decider>>::
+Estimates required storage capacity based on current ingestion into hot nodes.
+Available for policies governing hot data nodes.
+
+<<autoscaling-machine-learning-decider,Machine learning decider>>::
+Estimates required memory capacity based on machine learning jobs.
+Available for policies governing machine learning nodes.
+
+<<autoscaling-fixed-decider,Fixed decider>>::
+Responds with a fixed required capacity. This decider is intended for testing only.
+
+include::deciders/reactive-storage-decider.asciidoc[]
+include::deciders/proactive-storage-decider.asciidoc[]
+include::deciders/machine-learning-decider.asciidoc[]
+include::deciders/fixed-decider.asciidoc[]

docs/reference/autoscaling/autoscaling-designed-for-note.asciidoc

@@ -0,0 +1,4 @@
+[NOTE]
+Autoscaling is designed for indirect use by {ess-trial}[{ess}],
+{ece-ref}[{ece}], and {eck-ref}[Elastic Cloud on Kubernetes]. Direct use is not
+supported.

docs/reference/autoscaling/deciders/fixed-decider.asciidoc

@@ -0,0 +1,67 @@
+[role="xpack"]
+[testenv="enterprise"]
+[[autoscaling-fixed-decider]]
+=== Fixed decider
+
+experimental[]
+[WARNING]
+The fixed decider is intended for testing only. Do not use this decider in production.
+
+The `fixed` decider responds with a fixed required capacity. It is not enabled
+by default but can be enabled for any policy by explicitly configuring it.
+
+==== Configuration settings
+
+`storage`::
+(Optional, <<byte-units,byte value>>)
+Required amount of node-level storage. Defaults to `-1` (disabled).
+
+`memory`::
+(Optional, <<byte-units,byte value>>)
+Required amount of node-level memory. Defaults to `-1` (disabled).
+
+`nodes`::
+(Optional, integer)
+Number of nodes to use when calculating capacity. Defaults to `1`.
+
+[[autoscaling-fixed-decider-examples]]
+==== {api-examples-title}
+
+This example puts an autoscaling policy named `my_autoscaling_policy`, enabling
+and configuring the fixed decider.
+
+[source,console]
+--------------------------------------------------
+PUT /_autoscaling/policy/my_autoscaling_policy
+{
+  "roles" : [ "data_hot" ],
+  "deciders": {
+    "fixed": {
+      "storage": "1tb",
+      "memory": "32gb",
+      "nodes": 8
+    }
+  }
+}
+--------------------------------------------------
+// TEST
+
+The API returns the following result:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true
+}
+--------------------------------------------------
+
+//////////////////////////
+
+[source,console]
+--------------------------------------------------
+DELETE /_autoscaling/policy/my_autoscaling_policy
+--------------------------------------------------
+// TEST[continued]
+
+//////////////////////////
+

docs/reference/autoscaling/deciders/machine-learning-decider.asciidoc

@@ -0,0 +1,65 @@
+[role="xpack"]
+[testenv="enterprise"]
+[[autoscaling-machine-learning-decider]]
+=== Machine learning decider
+
+The {ml} decider (`ml`) calculates the memory required to run
+{ml} jobs created by users.
+
+The {ml} decider is enabled for policies governing `ml` nodes.
+
+[[autoscaling-machine-learning-decider-settings]]
+==== Configuration settings
+
+`num_anomaly_jobs_in_queue`::
+(Optional, integer)
+Number of queued anomaly jobs to allow. Defaults to `0`.
+
+`num_analytics_jobs_in_queue`::
+(Optional, integer)
+Number of queued analytics jobs to allow. Defaults to `0`.
+
+`down_scale_delay`::
+(Optional, <<time-units,time value>>)
+Delay before scaling down. Defaults to 1 hour.
+
+[[autoscaling-machine-learning-decider-examples]]
+==== {api-examples-title}
+
+This example puts an autoscaling policy named `my_autoscaling_policy`,
+overriding the machine learning decider's configuration.
+
+[source,console]
+--------------------------------------------------
+PUT /_autoscaling/policy/my_autoscaling_policy
+{
+  "roles" : [ "ml" ],
+  "deciders": {
+    "ml": {
+      "num_anomaly_jobs_in_queue": 5,
+      "num_analytics_jobs_in_queue": 3,
+      "down_scale_delay": "30m"
+    }
+  }
+}
+--------------------------------------------------
+// TEST
+
+The API returns the following result:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true
+}
+--------------------------------------------------
+
+//////////////////////////
+
+[source,console]
+--------------------------------------------------
+DELETE /_autoscaling/policy/my_autoscaling_policy
+--------------------------------------------------
+// TEST[continued]
+
+//////////////////////////

docs/reference/autoscaling/deciders/proactive-storage-decider.asciidoc

@@ -0,0 +1,59 @@
+[role="xpack"]
+[testenv="enterprise"]
+[[autoscaling-proactive-storage-decider]]
+=== Proactive storage decider
+
+The proactive storage decider (`proactive_storage`) calculates the storage required to contain
+the current data set plus an estimated amount of expected additional data.
+
+The proactive storage decider is enabled for all policies governing nodes with the `data_hot` role.
+
+The estimation of expected additional data is based on past indexing that
+occurred within the `forecast_window`.
+Only indexing into data streams contributes to the estimate.
+
+[[autoscaling-proactive-storage-decider-settings]]
+==== Configuration settings
+
+`forecast_window`::
+(Optional, <<time-units,time value>>)
+The window of time to use for forecasting. Defaults to 30 minutes.
+
+[[autoscaling-proactive-storage-decider-examples]]
+==== {api-examples-title}
+
+This example puts an autoscaling policy named `my_autoscaling_policy`, overriding
+the proactive decider's `forecast_window` to be 10 minutes.
+
+[source,console]
+--------------------------------------------------
+PUT /_autoscaling/policy/my_autoscaling_policy
+{
+  "roles" : [ "data_hot" ],
+  "deciders": {
+    "proactive_storage": {
+      "forecast_window": "10m"
+    }
+  }
+}
+--------------------------------------------------
+// TEST
+
+The API returns the following result:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true
+}
+--------------------------------------------------
+
+//////////////////////////
+
+[source,console]
+--------------------------------------------------
+DELETE /_autoscaling/policy/my_autoscaling_policy
+--------------------------------------------------
+// TEST[continued]
+
+//////////////////////////

docs/reference/autoscaling/deciders/reactive-storage-decider.asciidoc

@@ -0,0 +1,9 @@
+[role="xpack"]
+[[autoscaling-reactive-storage-decider]]
+=== Reactive storage decider
+
+The reactive storage decider (`reactive_storage`) calculates the storage required to contain
+the current data set. It signals that additional storage capacity is necessary
+when existing capacity has been exceeded (reactively).
+
+The reactive storage decider is enabled for all policies governing data nodes and has no configuration options.

docs/reference/autoscaling/index.asciidoc

@@ -1,10 +1,9 @@
 [role="xpack"]
-[testenv="platinum"]
+[testenv="enterprise"]
 [[xpack-autoscaling]]
-[chapter]
 = Autoscaling
 
-experimental[]
+include::autoscaling-designed-for-note.asciidoc[]
 
 The autoscaling feature enables an operator to configure tiers of nodes that
 self-monitor whether or not they need to scale based on an operator-defined
@@ -17,3 +16,15 @@ allocate existing and future shard copies due to disk space, then the
 autoscaling API would report that the cluster needs to scale due to disk space.
 It remains the responsibility of the operator to add the additional resources
 that the cluster signals it requires.
+
+A policy is composed of a list of roles and a list of deciders. Nodes matching
+the roles are governed by the policy. The deciders provide independent estimates
+of the capacity required. See <<autoscaling-deciders,Autoscaling deciders>> for
+more information on the deciders available.
+
+Autoscaling supports the scale-up and scale-down of dedicated {ml}
+nodes. Autoscaling also supports the scale-up of data nodes based on storage.
+[NOTE]
+Autoscaling is not supported on Debian 8.
+
+include::autoscaling-deciders.asciidoc[]