Dangling indices documentation (#58751)

Part of #48366. Add documentation for the dangling indices
API added in #58176.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>

docs/reference/indices.asciidoc

@@ -76,6 +76,14 @@ index settings, aliases, mappings, and index templates.
 * <<indices-flush>>
 * <<indices-forcemerge>>
 
+[float]
+[[dangling-indices-api]]
+=== Dangling indices:
+* <<dangling-indices-list>>
+* <<dangling-index-import>>
+* <<dangling-index-delete>>
+
+
 
 include::indices/add-alias.asciidoc[]
 
@@ -156,3 +164,9 @@ include::indices/aliases.asciidoc[]
 include::indices/update-settings.asciidoc[]
 
 include::indices/resolve.asciidoc[]
+
+include::indices/dangling-indices-list.asciidoc[]
+
+include::indices/dangling-index-import.asciidoc[]
+
+include::indices/dangling-index-delete.asciidoc[]

docs/reference/indices/dangling-index-delete.asciidoc

@@ -0,0 +1,44 @@
+[[dangling-index-delete]]
+=== Delete dangling index API
+++++
+<titleabbrev>Delete dangling index</titleabbrev>
+++++
+
+Deletes a dangling index.
+
+[[dangling-index-delete-api-request]]
+==== {api-request-title}
+
+[source,console]
+--------------------------------------------------
+DELETE /_dangling/<index-uuid>?accept_data_loss=true
+--------------------------------------------------
+// TEST[skip:Difficult to set up]
+
+[[dangling-index-delete-api-desc]]
+==== {api-description-title}
+
+include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
+
+
+Deletes a dangling index by referencing its UUID. Use the
+<<dangling-indices-list,List dangling indices API>> to locate the UUID of an index.
+
+
+[[dangling-index-delete-api-path-params]]
+==== {api-path-parms-title}
+
+`<index-uuid>`::
+(Required, string)
+UUID of the index to delete. You can find this using the
+<<dangling-indices-list,List dangling indices API>>.
+
+[[dangling-index-delete-api-query-params]]
+==== {api-query-parms-title}
+
+`accept_data_loss`::
+(Optional, boolean)
+This field must be set to `true` in order to carry out the import, since it will
+no longer be possible to recover the data from the dangling index.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]

docs/reference/indices/dangling-index-import.asciidoc

@@ -0,0 +1,65 @@
+[[dangling-index-import]]
+=== Import dangling index API
+++++
+<titleabbrev>Import dangling index</titleabbrev>
+++++
+
+Imports a dangling index.
+
+[[dangling-index-import-api-request]]
+==== {api-request-title}
+
+[source,console]
+--------------------------------------------------
+POST /_dangling/<index-uuid>?accept_data_loss=true
+--------------------------------------------------
+// TEST[skip:Difficult to set up]
+
+[[dangling-index-import-api-desc]]
+==== {api-description-title}
+
+include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
+
+Import a single index into the cluster by referencing its UUID. Use the
+<<dangling-indices-list,List dangling indices API>> to locate the UUID of an index.
+
+
+[[dangling-index-import-api-path-params]]
+==== {api-path-parms-title}
+
+`<index-uuid>`::
+(Required, string)
+UUID of the index to import, which you can find using the
+<<dangling-indices-list,List dangling indices API>>.
+
+[[dangling-index-import-api-query-params]]
+==== {api-query-parms-title}
+
+`accept_data_loss`::
+(Required, boolean)
+This field must be set to `true` to import a dangling index. Because {es}
+cannot know where the dangling index data came from or determine which shard
+copies are fresh and which are stale, it cannot guarantee that the imported data
+represents the latest state of the index when it was last in the cluster.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]
+
+[[dangling-index-import-api-example]]
+==== {api-examples-title}
+
+The following example shows how to import a dangling index:
+
+[source,console]
+--------------------------------------------------
+POST /_dangling/zmM4e0JtBkeUjiHD-MihPQ?accept_data_loss=true
+--------------------------------------------------
+// TEST[skip:Difficult to set up]
+
+The API returns following response:
+
+[source,console-result]
+--------------------------------------------------
+{
+    "acknowledged" : true
+}
+--------------------------------------------------

docs/reference/indices/dangling-indices-list.asciidoc

@@ -0,0 +1,49 @@
+[[dangling-indices-list]]
+=== List dangling indices API
+++++
+<titleabbrev>List dangling indices</titleabbrev>
+++++
+
+Lists dangling indices.
+
+[[dangling-indices-list-api-request]]
+==== {api-request-title}
+
+[source,console]
+--------------------------------------------------
+GET /_dangling
+--------------------------------------------------
+// TEST[skip:TBD]
+
+[[dangling-indices-list-api-desc]]
+==== {api-description-title}
+
+// tag::dangling-index-description[]
+If {es} encounters index data that is absent from the current cluster
+state, those indices are considered to be dangling. For example,
+this can happen if you delete more than
+`cluster.indices.tombstones.size` indices while an {es} node is offline.
+// end::dangling-index-description[]
+
+Use this API to list dangling indices, which you can then
+<<dangling-index-import,import>> or <<dangling-index-delete,delete>>.
+
+
+[[dangling-indices-list-api-example]]
+==== {api-examples-title}
+
+The API returns the following response:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "dangling_indices": [
+    "index_name": "twitter",
+    "index_uuid": "zmM4e0JtBkeUjiHD-MihPQ",
+    "creation_date_millis": 1589414451372,
+    "node_ids": [
+      "pL47UN3dAb2d5RCWP6lQ3e"
+    ]
+  ]
+}
+--------------------------------------------------

docs/reference/modules/cluster/misc.asciidoc

@@ -107,6 +107,10 @@ increase it if you expect nodes to be absent from the cluster and miss more
 than 500 deletes. We think that is rare, thus the default. Tombstones don't take
 up much space, but we also think that a number like 50,000 is probably too big.
 
+include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index-description]
+You can use the <<dangling-indices-api,Dangling indices API>> to manage
+this situation.
+
 [[cluster-logger]]
 ===== Logger
 

docs/reference/modules/discovery.asciidoc

@@ -59,7 +59,7 @@ formation:
 <<modules-discovery-settings,Settings>>::
 
     There are settings that enable users to influence the discovery, cluster
-    formation, master election and fault detection processes.    
+    formation, master election and fault detection processes.
 
 include::discovery/discovery.asciidoc[]
 

docs/reference/modules/gateway.asciidoc

@@ -56,9 +56,29 @@ NOTE: These settings only take effect on a full cluster restart.
 [[dangling-indices]]
 ==== Dangling indices
 
-When a node joins the cluster, any shards stored in its local data
-directory which do not already exist in the cluster will be imported into the
-cluster. This functionality is intended as a best effort to help users who
-lose all master nodes. If a new master node is started which is unaware of
-the other indices in the cluster, adding the old nodes will cause the old
-indices to be imported, instead of being deleted.
+When a node joins the cluster, if it finds any shards stored in its local data
+directory that do not already exist in the cluster, it will consider those
+shards to be "dangling". Importing dangling indices
+into the cluster using `gateway.auto_import_dangling_indices` is not safe.
+Instead, use the <<dangling-indices-api,Dangling indices API>>. Neither
+mechanism provides any guarantees as to whether the imported data truly
+represents the latest state of the data when the index was still part of
+the cluster.
+
+`gateway.auto_import_dangling_indices`::
+
+    deprecated:[7.9.0, This setting will be removed in 8.0. You should use the dedicated dangling indices API instead.]
+    Whether to automatically import dangling indices into the cluster
+    state, provided no indices already exist with the same name. Defaults
+    to `false`.
+
+WARNING: The auto-import functionality was intended as a best effort to help users
+who lose all master nodes. For example, if a new master node were to be
+started which was unaware of the other indices in the cluster, adding the
+old nodes would cause the old indices to be imported, instead of being
+deleted. However there are several issues with automatic importing, and
+its use is strongly discouraged in favour of the
+<<dangling-indices-api,dedicated API>.
+
+WARNING: Losing all master nodes is a situation that should be avoided at
+all costs, as it puts your cluster's metadata and data at risk.