Troubleshooting docs for ACTION_RESTORE_FROM_SNAPSHOT (#87692)

Troubleshooting guide to restore indices and data streams that have
missing data from a snapshot.

This will be associated with the user action
`ACTION_RESTORE_FROM_SNAPSHOT`.

Preview link:
https://elasticsearch_87692.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/restore-from-snapshot.html

docs/reference/tab-widgets/troubleshooting/data/restore-from-snapshot-widget.asciidoc

@@ -0,0 +1,40 @@
+++++
+<div class="tabs" data-tab-group="host">
+  <div role="tablist" aria-label="Restore from snapshot">
+    <button role="tab"
+            aria-selected="true"
+            aria-controls="cloud-tab-restore-from-snapshot"
+            id="cloud-restore-from-snapshot">
+      Elasticsearch Service
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="self-managed-tab-restore-from-snapshot"
+            id="self-managed-restore-from-snapshot"
+            tabindex="-1">
+      Self-managed
+    </button>
+  </div>
+  <div tabindex="0"
+       role="tabpanel"
+       id="cloud-tab-restore-from-snapshot"
+       aria-labelledby="cloud-restore-from-snapshot">
+++++
+
+include::restore-from-snapshot.asciidoc[tag=cloud]
+
+++++
+  </div>
+  <div tabindex="0"
+       role="tabpanel"
+       id="self-managed-tab-restore-from-snapshot"
+       aria-labelledby="self-managed-restore-from-snapshot"
+       hidden="">
+++++
+
+include::restore-from-snapshot.asciidoc[tag=self-managed]
+
+++++
+  </div>
+</div>
+++++

docs/reference/tab-widgets/troubleshooting/data/restore-from-snapshot.asciidoc

@@ -0,0 +1,456 @@
+// tag::cloud[]
+In order to restore the indices and data streams that are missing data:
+
+**Use {kib}**
+
+//tag::kibana-api-ex[]
+. Log in to the {ess-console}[{ecloud} console].
++
+
+. On the **Elasticsearch Service** panel, click the name of your deployment.
++
+
+NOTE: If the name of your deployment is disabled your {kib} instances might be
+unhealthy, in which case please contact https://support.elastic.co[Elastic Support].
+If your deployment doesn't include {kib}, all you need to do is
+{cloud}/ec-access-kibana.html[enable it first].
++
+. Open your deployment's side navigation menu (placed under the Elastic logo in the upper left corner)
+and go to **Dev Tools > Console**.
++
+[role="screenshot"]
+image::images/kibana-console.png[{kib} Console,align="center"]
+
+. To view the affected indices using the <<cat-indices,cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&health=red&h=index,status,health
+----
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 open   red
+kibana_sample_data_flights           open   red
+----
+// TEST[skip:illustration purposes only]
++
+The `red` health of the indices above indicates that these indices are missing primary shards,
+meaning they are missing data.
++
+. In order to restore the data we need to find a snapshot that contains these two indices. To find
+such a snapshot use the
+<<get-snapshot-api,get snapshot API>>.
++
+[source,console]
+----
+GET _snapshot/my_repository/*?verbose=false
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+{
+  "snapshots" : [
+    {
+      "snapshot" : "snapshot-20200617",                                     <1>
+      "uuid" : "dZyPs1HyTwS-cnKdH08EPg",
+      "repository" : "my_repository",                                       <2>
+      "indices" : [                                                         <3>
+        ".apm-agent-configuration",
+        ".apm-custom-link",
+        ".ds-ilm-history-5-2022.06.17-000001",
+        ".ds-my-data-stream-2022.06.17-000001",
+        ".geoip_databases",
+        ".kibana-event-log-8.2.2-000001",
+        ".kibana_8.2.2_001",
+        ".kibana_task_manager_8.2.2_001",
+        "kibana_sample_data_ecommerce",
+        "kibana_sample_data_flights",
+        "kibana_sample_data_logs"
+      ],
+      "data_streams" : [ ],
+      "state" : "SUCCESS"                                                     <4>
+    }
+  ],
+  "total" : 1,
+  "remaining" : 0
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The name of the snapshot.
++
+<2> The repository of the snapshot.
++
+<3> The indices backed up in the snapshot.
++
+<4> If the snapshot was successful.
+
+. The snapshot `snapshot-20200617` contains the two indices we want to restore.
+You might have multiple snapshots from which you could restore the target indices. Choose the latest snapshot.
++
+. Now that we found a snapshot, we will proceed with the data stream preparation for restoring the lost data. We will
+check the index metadata to see if any index is part of a data stream:
++
+[source,console]
+----
+GET kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001?features=settings&flat_settings
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+{
+  ".ds-my-data-stream-2022.06.17-000001" : {                                <1>
+    "aliases" : { },
+    "mappings" : { },
+    "settings" : {                                                          <2>
+      "index.creation_date" : "1658406121699",
+      "index.hidden" : "true",
+      "index.lifecycle.name" : "my-lifecycle-policy",
+      "index.number_of_replicas" : "1",
+      "index.number_of_shards" : "1",
+      "index.provided_name" : ".ds-my-data-stream-2022.06.17-000001",
+      "index.routing.allocation.include._tier_preference" : "data_hot",
+      "index.uuid" : "HmlFXp6VSu2XbQ-O3hVrwQ",
+      "index.version.created" : "8020299"
+    },
+    "data_stream" : "my-data-stream"                                        <3>
+  },
+  "kibana_sample_data_flights" : {                                          <4>
+    "aliases" : { },
+    "mappings" : { },
+    "settings" : {
+      "index.creation_date" : "1655121541454",
+      "index.number_of_replicas" : "0",
+      "index.number_of_shards" : "1",
+      "index.provided_name" : "kibana_sample_data_flights",
+      "index.routing.allocation.include._tier_preference" : "data_content",
+      "index.uuid" : "jMOlwKPPSzSraeeBWyuoDA",
+      "index.version.created" : "8020299"
+    }
+  }
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The name of an index.
++
+<2> The settings of this index that contains the metadata we are looking for.
++
+<3> This indicates that this index is part of a data stream and displays the data stream name.
++
+<4> The name of the other index we requested.
++
+The response above shows that `kibana_sample_data_flights` is not part of a data stream because it doesn't have a
+field called `data_stream` in the settings.
++
+On the contrary, `.ds-my-data-stream-2022.06.17-000001` is part of the data stream called `my-data-stream`. When you
+find an index like this, which belongs to a data stream, you need to check if data are still being indexed.
+You can see that by checking the `settings`, if you can find this property: `"index.lifecycle.indexing_complete" : "true"`,
+it means that indexing is completed in this index and you can continue to the next step.
++
+If `index.lifecycle.indexing_complete` is not there or is configured to `false` you need to rollover the data stream so you can restore the missing data without blocking
+the ingestion of new data. The following command will achieve that.
++
+[source,console]
+----
+POST my-data-stream/_rollover
+----
+// TEST[skip:illustration purposes only]
+
+. Now that the data stream preparation is done, we will close the target indices by using the
+<<indices-close, close indices API>>.
++
+[source,console]
+----
+POST kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001/_close
+----
+// TEST[skip:illustration purposes only]
++
+You can confirm that they are closed with
+the <<cat-indices, cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&health=red&h=index,status,health
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 close   red
+kibana_sample_data_flights           close   red
+----
++
+. The indices are closed, now we can restore them from snapshots without causing
+any complications using the <<restore-snapshot-api, restore snapshot API>>:
++
+[source,console]
+----
+POST _snapshot/my_repository/snapshot-20200617/_restore
+{
+  "indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001", <1>
+  "include_aliases": true                                                       <2>
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The indices to restore.
++
+<2> We also want to restore the aliases.
+
+. Finally we can verify that the indices health is now `green` via the <<cat-indices,cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&index=.ds-my-data-stream-2022.06.17-000001,kibana_sample_data_flightsh=index,status,health
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 open   green
+kibana_sample_data_flights           open   green
+----
+// TEST[skip:illustration purposes only]
++
+As we can see above the indices are `green` and open. The issue is resolved.
+
+For more guidance on creating and restoring snapshots see
+<<snapshot-restore, this guide>>.
+
+
+//end::kibana-api-ex[]
+// end::cloud[]
+
+// tag::self-managed[]
+In order to restore the indices that are missing shards:
+
+. View the affected indices using the <<cat-indices,cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&health=red&h=index,status,health
+----
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 open   red
+kibana_sample_data_flights           open   red
+----
+// TEST[skip:illustration purposes only]
++
+The `red` health of the indices above indicates that these indices are missing primary shards,
+meaning they are missing data.
++
+. In order to restore the data we need to find a snapshot that contains these two indices. To find
+such a snapshot use the
+<<get-snapshot-api,get snapshot API>>.
++
+[source,console]
+----
+GET _snapshot/my_repository/*?verbose=false
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+{
+  "snapshots" : [
+    {
+      "snapshot" : "snapshot-20200617",                                     <1>
+      "uuid" : "dZyPs1HyTwS-cnKdH08EPg",
+      "repository" : "my_repository",                                       <2>
+      "indices" : [                                                         <3>
+        ".apm-agent-configuration",
+        ".apm-custom-link",
+        ".ds-ilm-history-5-2022.06.17-000001",
+        ".ds-my-data-stream-2022.06.17-000001",
+        ".geoip_databases",
+        ".kibana-event-log-8.2.2-000001",
+        ".kibana_8.2.2_001",
+        ".kibana_task_manager_8.2.2_001",
+        "kibana_sample_data_ecommerce",
+        "kibana_sample_data_flights",
+        "kibana_sample_data_logs"
+      ],
+      "data_streams" : [ ],
+      "state" : "SUCCESS"                                                     <4>
+    }
+  ],
+  "total" : 1,
+  "remaining" : 0
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The name of the snapshot.
++
+<2> The repository of the snapshot.
++
+<3> The indices backed up in the snapshot.
++
+<4> If the snapshot was successful.
+
+. The snapshot `snapshot-20200617` contains the two indices we want to restore.
+You might have multiple snapshots from which you could restore the target indices. Choose the latest snapshot.
++
+. Now that we found a snapshot, we will proceed with the data stream preparation for restoring the lost data. We will
+check the index metadata to see if any index is part of a data stream:
++
+[source,console]
+----
+GET kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001?features=settings&flat_settings
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+{
+  ".ds-my-data-stream-2022.06.17-000001" : {                                <1>
+    "aliases" : { },
+    "mappings" : { },
+    "settings" : {                                                          <2>
+      "index.creation_date" : "1658406121699",
+      "index.hidden" : "true",
+      "index.lifecycle.name" : "my-lifecycle-policy",
+      "index.number_of_replicas" : "1",
+      "index.number_of_shards" : "1",
+      "index.provided_name" : ".ds-my-data-stream-2022.06.17-000001",
+      "index.routing.allocation.include._tier_preference" : "data_hot",
+      "index.uuid" : "HmlFXp6VSu2XbQ-O3hVrwQ",
+      "index.version.created" : "8020299"
+    },
+    "data_stream" : "my-data-stream"                                        <3>
+  },
+  "kibana_sample_data_flights" : {                                          <4>
+    "aliases" : { },
+    "mappings" : { },
+    "settings" : {
+      "index.creation_date" : "1655121541454",
+      "index.number_of_replicas" : "0",
+      "index.number_of_shards" : "1",
+      "index.provided_name" : "kibana_sample_data_flights",
+      "index.routing.allocation.include._tier_preference" : "data_content",
+      "index.uuid" : "jMOlwKPPSzSraeeBWyuoDA",
+      "index.version.created" : "8020299"
+    }
+  }
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The name of an index.
++
+<2> The settings of this index that contains the metadata we are looking for.
++
+<3> This indicates that this index is part of a data stream and displays the data stream name.
++
+<4> The name of the other index we requested.
++
+The response above shows that `kibana_sample_data_flights` is not part of a data stream because it doesn't have a
+field called `data_stream` in the settings.
++
+On the contrary, `.ds-my-data-stream-2022.06.17-000001` is part of the data stream called `my-data-stream`. When you
+find an index like this, which belongs to a data stream, you need to check if data are still being indexed.
+You can see that by checking the `settings`, if you can find this property: `"index.lifecycle.indexing_complete" : "true"`,
+it means that indexing is completed in this index and you can continue to the next step.
++
+If `index.lifecycle.indexing_complete` is not there or is configured to `false` you need to rollover the data stream so you can restore the missing data without blocking
+the ingestion of new data. The following command will achieve that.
++
+[source,console]
+----
+POST my-data-stream/_rollover
+----
+// TEST[skip:illustration purposes only]
+
+. Now that the data stream preparation is done, we will close the target indices by using the
+<<indices-close, close indices API>>.
++
+[source,console]
+----
+POST kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001/_close
+----
+// TEST[skip:illustration purposes only]
++
+You can confirm that they are closed with
+the <<cat-indices, cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&health=red&h=index,status,health
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 close   red
+kibana_sample_data_flights           close   red
+----
++
+. The indices are closed, now we can restore them from snapshots without causing
+any complications using the <<restore-snapshot-api, restore snapshot API>>:
++
+[source,console]
+----
+POST _snapshot/my_repository/snapshot-20200617/_restore
+{
+  "indices": "kibana_sample_data_flights,.ds-my-data-stream-2022.06.17-000001", <1>
+  "include_aliases": true                                                       <2>
+}
+----
+// TEST[skip:illustration purposes only]
++
+<1> The indices to restore.
++
+<2> We also want to restore the aliases.
+
+. Finally we can verify that the indices health is now `green` via the <<cat-indices,cat indices API>>.
++
+[source,console]
+----
+GET _cat/indices?v&index=.ds-my-data-stream-2022.06.17-000001,kibana_sample_data_flightsh=index,status,health
+----
+// TEST[skip:illustration purposes only]
++
+The response will look like this:
++
+[source,console-result]
+----
+index                                status health
+.ds-my-data-stream-2022.06.17-000001 open   green
+kibana_sample_data_flights           open   green
+----
+// TEST[skip:illustration purposes only]
++
+As we can see above the indices are `green` and open. The issue is resolved.
+
+For more guidance on creating and restoring snapshots see
+<<snapshot-restore, this guide>>.
+// end::self-managed[]

docs/reference/troubleshooting.asciidoc

@@ -58,6 +58,8 @@ include::troubleshooting/data/start-ilm.asciidoc[]
 
 include::troubleshooting/data/start-slm.asciidoc[]
 
+include::troubleshooting/data/restore-from-snapshot.asciidoc[]
+
 include::troubleshooting/snapshot/add-repository.asciidoc[]
 
 include::monitoring/troubleshooting.asciidoc[]

docs/reference/troubleshooting/data/restore-from-snapshot.asciidoc

@@ -0,0 +1,12 @@
+[[restore-from-snapshot]]
+== Restore from snapshot
+
+Elasticsearch is using snapshots to store a copy of your data outside a cluster. You can restore a snapshot to recover
+indices and data streams for which there are no copies of the shards in the cluster. This can happen if the data
+(indices or data streams) was deleted or if the cluster membership changed and the current nodes in the system do not
+contain a copy of the data anymore.
+
+IMPORTANT: Restoring the missing data requires you to have a backup of the affected indices and data streams that is
+up-to-date enough for your use case. Please do not proceed without confirming this.
+
+include::{es-repo-dir}/tab-widgets/troubleshooting/data/restore-from-snapshot-widget.asciidoc[]