Node shutdown API docs (#74505)

* Put Shutdown API docs

* Get Shutdown API docs

* Properly escape regex

* Doc tests build now! (but don't pass)

* Doc tests pass now! (with a code fix that will go in a different PR)

* DELETE shutdown docs

* Edit for language & consistency

* Fix doctest

* Add cloud-only banner

* Add allocation_delay docs

* Restore file that got deleted somehow?

* Use `restart` type in example to demonstrate `allocation_delay` param

* Fix typo per review

Co-authored-by: debadair <debadair@elastic.co>

* Vastly improve wording per review

Co-authored-by: debadair <debadair@elastic.co>

* Adjust test request & response so it passes

Co-authored-by: Deb Adair <debadair@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

docs/reference/rest-api/index.asciidoc

@@ -33,6 +33,7 @@ not be included yet.
 * <<ml-apis,{ml-cap} {anomaly-detect} APIs>>
 * <<ml-df-analytics-apis,{ml-cap} {dfanalytics} APIs>>
 * <<migration-api,Migration APIs>>
+* <<node-lifecycle-api,Node lifecycle API>>
 * <<indices-reload-analyzers,Reload search analyzers API>>
 * <<repositories-metering-apis,Repositories metering APIs>>
 * <<rollup-apis,Rollup APIs>>
@@ -71,6 +72,7 @@ include::{xes-repo-dir}/rest-api/logstash/index.asciidoc[]
 include::{es-repo-dir}/ml/anomaly-detection/apis/index.asciidoc[]
 include::{es-repo-dir}/ml/df-analytics/apis/index.asciidoc[]
 include::{es-repo-dir}/migration/migration.asciidoc[]
+include::{es-repo-dir}/shutdown/apis/shutdown-api.asciidoc[]
 include::{es-repo-dir}/indices/apis/reload-analyzers.asciidoc[]
 include::{es-repo-dir}/repositories-metering-api/repositories-metering-apis.asciidoc[]
 include::{es-repo-dir}/rollup/rollup-apis.asciidoc[]

docs/reference/shutdown/apis/shutdown-api.asciidoc

@@ -0,0 +1,19 @@
+[role="xpack"]
+[testenv="basic"]
+[[node-lifecycle-api]]
+== Node lifecycle APIs
+
+NOTE: {cloud-only}
+
+You use the shutdown APIs to prepare nodes for temporary or permanent shutdown, monitor the shutdown status, and enable a previously shut-down node to resume normal operations.
+
+[discrete]
+=== Node shutdown APIs
+
+* <<put-shutdown,Put shutdown API>>
+* <<get-shutdown,Get shutdown API>>
+* <<delete-shutdown,Delete shutdown API>>
+
+include::shutdown-put.asciidoc[]
+include::shutdown-get.asciidoc[]
+include::shutdown-delete.asciidoc[]

docs/reference/shutdown/apis/shutdown-delete.asciidoc

@@ -0,0 +1,77 @@
+[[delete-shutdown]]
+=== Delete shutdown API
+
+NOTE: {cloud-only}
+
+Cancels shutdown preparations or clears a shutdown request
+so a node can resume normal operations.
+
+[[delete-shutdown-api-request]]
+==== {api-request-title}
+
+`DELETE _nodes/<node-id>/shutdown`
+
+[[delete-shutdown-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API.
+
+* If the <<operator-privileges,{operator-feature}>> is enabled,
+you must be an operator to use this API.
+
+[[delete-shutdown-api-desc]]
+==== {api-description-title}
+
+Enables a node to resume normal operations following a
+<<put-shutdown, put shutdown>> request.
+You must explicitly clear the shutdown request when a node rejoins the cluster,
+regardless of the shutdown type.
+
+
+[[delete-shutdown-api-path-params]]
+==== {api-path-parms-title}
+
+`<node-id>`::
+(Optional, string)
+The ID of a node that you prepared for shut down.
+
+[[delete-shutdown-api-params]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+[[delete-shutdown-api-example]]
+==== {api-examples-title}
+
+Prepare a node to be restarted:
+
+[source,console]
+--------------------------------------------------
+PUT /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
+{
+  "type": "restart",
+  "reason": "Demonstrating how the node shutdown API works"
+}
+--------------------------------------------------
+// TEST[setup:node]
+// TEST[s/USpTGYaBSIKbgSUJR2Z9lg/\$node_name/]
+
+Cancel the shutdown preparations or clear the shutdown request
+after restarting:
+
+[source,console]
+--------------------------------------------------
+DELETE /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
+--------------------------------------------------
+// TEST[continued]
+// TEST[s/USpTGYaBSIKbgSUJR2Z9lg/\$node_name/]
+
+Which returns the following response:
+
+[source,console-result]
+--------------------------------------------------
+{
+    "acknowledged": true
+}
+--------------------------------------------------

docs/reference/shutdown/apis/shutdown-get.asciidoc

@@ -0,0 +1,98 @@
+[[get-shutdown]]
+=== Get shutdown API
+
+NOTE: {cloud-only}
+
+Retrieves the status of a node that's being prepared for shutdown.
+
+[[get-shutdown-api-request]]
+==== {api-request-title}
+
+`GET _nodes/shutdown`
+
+`GET _nodes/<node-id>/shutdown`
+
+[[get-shutdown-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API.
+
+* If the <<operator-privileges,{operator-feature}>> is enabled,
+you must be an operator to use this API.
+
+[[get-shutdown-api-desc]]
+==== {api-description-title}
+
+Indicates whether a node is ready to be shut down, or if shut down preparations
+are still in progress or have stalled.
+Returns status information for each part of the shut down process.
+Use to monitor the shut down process after calling <<put-shutdown, put shutdown>>.
+
+[[get-shutdown-api-path-params]]
+==== {api-path-parms-title}
+
+`<node-id>`::
+(Optional, string)
+The ID of a node that is being prepared for shutdown.
+If no ID is specified, returns the status of all nodes being prepared for shutdown.
+
+[[get-shutdown-api-params]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+[[get-shutdown-api-example]]
+==== {api-examples-title}
+
+Prepare a node to be restarted:
+
+[source,console]
+--------------------------------------------------
+PUT /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
+{
+  "type": "restart",
+  "reason": "Demonstrating how the node shutdown API works",
+  "allocation_delay": "10m"
+}
+--------------------------------------------------
+
+Get the status of the shutdown preparations:
+
+[source,console]
+--------------------------------------------------
+GET /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
+--------------------------------------------------
+// TEST[continued]
+
+The response shows information about the shutdown preparations,
+including the status of shard migration, task migration, and plugin cleanup:
+
+[source,console-result]
+--------------------------------------------------
+{
+    "nodes": [
+        {
+            "node_id": "USpTGYaBSIKbgSUJR2Z9lg",
+            "type": "RESTART",
+            "reason": "Demonstrating how the node shutdown API works",
+            "shutdown_startedmillis": 1624406108685,
+            "allocation_delay": "10m",
+            "status": "COMPLETE",
+            "shard_migration": {
+                "status": "COMPLETE",
+                "shard_migrations_remaining": 0,
+                "explanation": "no shard relocation is necessary for a node restart"
+            },
+            "persistent_tasks": {
+                "status": "COMPLETE"
+            },
+            "plugins": {
+                "status": "COMPLETE"
+            }
+        }
+    ]
+}
+--------------------------------------------------
+// TESTRESPONSE[s/1624406108685/$body.nodes.0.shutdown_startedmillis/]
+// TESTRESPONSE[s/USpTGYaBSIKbgSUJR2Z9lg/$body.nodes.0.node_id/]

docs/reference/shutdown/apis/shutdown-put.asciidoc

@@ -0,0 +1,104 @@
+[[put-shutdown]]
+=== Put shutdown API
+
+NOTE: {cloud-only}
+
+Prepares a node to be shut down.
+
+[[put-shutdown-api-request]]
+==== {api-request-title}
+
+`PUT _nodes/<node-id>/shutdown`
+
+[[put-shutdown-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API.
+
+* If the <<operator-privileges,{operator-feature}>> is enabled, you must be an operator
+to use this API.
+
+[[put-shutdown-api-desc]]
+==== {api-description-title}
+
+Migrates ongoing tasks and index shards to other nodes as needed
+to prepare a node to be restarted or shut down and removed from the cluster.
+This ensures that {es} can be stopped safely with minimal disruption to the cluster.
+
+You must specify the type of shutdown: `restart` or `remove`.
+If a node is already being prepared for shutdown,
+you can use this API to change the shutdown type.
+
+IMPORTANT: This API does *NOT* terminate the {es} process.
+Monitor the <<get-shutdown,node shutdown status>> to determine
+when it  is safe to stop {es}.
+
+[[put-shutdown-api-path-params]]
+==== {api-path-parms-title}
+
+`<node-id>`::
+(Required, string)
+The ID of the node you want to prepare for shutdown.
+If you specify a node that is offline,
+it will be prepared for shut down when it rejoins the cluster.
+
+IMPORTANT: This parameter is *NOT* validated against the cluster's active nodes.
+This enables you to register a node for shut down while it is offline.
+No error is thrown if you specify an invalid node ID.
+
+[[put-shutdown-api-params]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+[role="child_attributes"]
+[[put-shutdown-api-request-body]]
+==== {api-request-body-title}
+
+`type`::
+(Required, string)
+Valid values are `restart` and `remove`.
+Use `restart` when you need to temporarily shut down a node to perform an upgrade,
+make configuration changes, or perform other maintenance.
+Because the node is expected to rejoin the cluster, data is not migrated off of the node.
+Use `remove` when you need to permanently remove a node from the cluster.
+The node is not marked ready for shutdown until data is migrated off of the node
+
+`reason`::
+(Required, string)
+A human-readable reason that the node is being shut down.
+This field provides information for other cluster operators;
+it does not affect the shut down process.
+
+`allocation_delay`::
+(Optional, string)
+Only valid if `type` is `restart`. Controls how long {es} will wait for the node to restart and join the cluster before reassigning its shards to other nodes. This works the same as
+<<delayed-allocation,delaying allocation>> with the `index.unassigned.node_left.delayed_timeout` setting. If you specify both a restart allocation delay and an index-level allocation delay, the longer of the two is used.
+
+[[put-shutdown-api-example]]
+==== {api-examples-title}
+
+Register a node for shutdown:
+
+[source,console]
+--------------------------------------------------
+PUT /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
+{
+  "type": "restart", <1>
+  "reason": "Demonstrating how the node shutdown API works",
+  "allocation_delay": "20m"
+}
+--------------------------------------------------
+
+////
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true
+}
+--------------------------------------------------
+////
+
+<1> Prepares the node to be restarted.
+Use `remove` for nodes that will be permanently removed from the cluster.