Add migrate to tiers troubleshooting doc (#86738)

This adds a troubleshooting doc for indices that mix index filtering allocation
with data tiers routing.

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

docs/reference/tab-widgets/troubleshooting/data/migrate-to-data-tiers-routing-guide-widget.asciidoc

@@ -0,0 +1,40 @@
+++++
+<div class="tabs" data-tab-group="host">
+  <div role="tablist" aria-label="Data tiers routing conflict">
+    <button role="tab"
+            aria-selected="true"
+            aria-controls="cloud-tab-data-tiers-migration"
+            id="cloud-data-tiers-migration">
+      Elasticsearch Service
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="self-managed-tab-data-tiers-migration"
+            id="self-managed-data-tiers-migration"
+            tabindex="-1">
+      Self-managed
+    </button>
+  </div>
+  <div tabindex="0"
+       role="tabpanel"
+       id="cloud-tab-data-tiers-migration"
+       aria-labelledby="cloud-data-tiers-migration">
+++++
+
+include::migrate-to-data-tiers-routing-guide.asciidoc[tag=cloud]
+
+++++
+  </div>
+  <div tabindex="0"
+       role="tabpanel"
+       id="self-managed-tab-data-tiers-migration"
+       aria-labelledby="self-managed-data-tiers-migration"
+       hidden="">
+++++
+
+include::migrate-to-data-tiers-routing-guide.asciidoc[tag=self-managed]
+
+++++
+  </div>
+</div>
+++++

docs/reference/tab-widgets/troubleshooting/data/migrate-to-data-tiers-routing-guide.asciidoc

@@ -0,0 +1,224 @@
+// tag::cloud[]
+In order to get the shards assigned we need to call the 
+<<ilm-migrate-to-data-tiers, migrate to data tiers routing>> API which will
+resolve the conflicting routing configurations towards using the standardized
+<<data-tiers, data tiers>>. This will also future-proof the system by migrating 
+the index templates and ILM policies if needed.
+
+**Use {kib}**
+
+//tag::kibana-api-ex[]
+. Log in to the {ess-console}[{ecloud} console].
++
+
+. On the **Elasticsearch Service** panel, click the name of your deployment. 
++
+
+NOTE:
+If the name of your deployment is disabled your {kib} instances might be
+unhealthy, in which case please contact https://support.elastic.co[Elastic Support],
+or your deployment doesn't include {kib}, in which case all you need to do is 
+{cloud}/ec-access-kibana.html[enable Kibana first].
+
+. Open your deployment's side navigation menu (placed under the Elastic logo in the upper left corner)
+and go to **Dev Tools > Console**.
++
+[role="screenshot"]
+image::images/kibana-console.png[{kib} Console,align="center"]
+
+. First, let's <<ilm-stop,stop>> {ilm}
++
+[source,console]
+----
+POST /_ilm/stop
+----
+//TEST[skip:stopping ILM requires waiting]
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "acknowledged": true
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only]
+
+. Wait for {ilm} to stop. Check the status until it returns `STOPPED` as follows:
++
+[source,console]
+----
+GET /_ilm/status
+----
++
+When {ilm} has succesfully stopped the response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "operation_mode": "STOPPED"
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:not waiting for ILM to stop]
+
+. <<ilm-migrate-to-data-tiers, Migrate to data tiers>>
++
+[source,console]
+----
+POST /_ilm/migrate_to_data_tiers
+----
+//TEST[skip:this can flake as we're not waiting for ILM to stop]
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "dry_run": false,
+  "migrated_ilm_policies":["policy_with_allocate_action"], <1>
+  "migrated_indices":["warm-index-to-migrate-000001"], <2>
+  "migrated_legacy_templates":["a-legacy-template"], <3>
+  "migrated_composable_templates":["a-composable-template"], <4>
+  "migrated_component_templates":["a-component-template"] <5>
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only as we're not waiting for ILM to stop]
++
+<1> The ILM policies that were updated.
+<2> The indices that were migrated to <<tier-preference-allocation-filter,tier preference>> routing.
+<3> The legacy index templates that were updated to not contain custom routing settings for the
+provided data attribute.
+<4> The composable index templates that were updated to not contain custom routing settings for the
+provided data attribute.
+<5> The component templates that were updated to not contain custom routing settings for the
+provided data attribute.
+
+. <<ilm-start,Restart>> {ilm}
++
+[source,console]
+----
+POST /_ilm/start
+----
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "acknowledged": true
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:didn't wait to stop it]
+
+//end::kibana-api-ex[]
+// end::cloud[]
+
+// tag::self-managed[]
+In order to get the shards assigned we need to make sure the deployment is 
+using the <<data-tiers,data tiers>> node roles and then call the 
+<<ilm-migrate-to-data-tiers, migrate to data tiers routing>> API which will
+resolve the conflicting routing configurations towards using the standardized
+<<data-tiers, data tiers>>. This will also future-proof the system by migrating 
+the index templates and ILM policies if needed.
+
+
+. In case your deployment is not yet using <<data-tiers, data tiers>> <<assign-data-tier, assign data nodes>> 
+to the appropriate data tier.
+Configure the appropriate roles for each data node to assign it to one or more
+data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`.
+For example, the following setting configures a node to be a data-only
+node in the hot and content tiers.
++
+[source,yaml]
+----
+node.roles [ data_hot, data_content ]
+----
+
+. <<ilm-stop,Stop>> {ilm}
++
+[source,console]
+----
+POST /_ilm/stop
+----
+//TEST[skip:stopping ILM requires waiting]
+
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "acknowledged": true
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only]
+
+. Wait for {ilm} to stop. Check the status until it returns `STOPPED` as follows:
++
+[source,console]
+----
+GET /_ilm/status
+----
++
+When {ilm} has succesfully stopped the response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "operation_mode": "STOPPED"
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:not waiting for ILM to stop]
+
+. <<ilm-migrate-to-data-tiers, Migrate to data tiers>>
++
+[source,console]
+----
+POST /_ilm/migrate_to_data_tiers
+----
+//TEST[skip:this can flake as we're not waiting for ILM to stop]
+
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "dry_run": false,
+  "migrated_ilm_policies":["policy_with_allocate_action"], <1>
+  "migrated_indices":["warm-index-to-migrate-000001"], <2>
+  "migrated_legacy_templates":["a-legacy-template"], <3>
+  "migrated_composable_templates":["a-composable-template"], <4>
+  "migrated_component_templates":["a-component-template"] <5>
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:the result is for illustrating purposes only as we're not waiting for ILM to stop]
++
+<1> The ILM policies that were updated.
+<2> The indices that were migrated to <<tier-preference-allocation-filter,tier preference>> routing.
+<3> The legacy index templates that were updated to not contain custom routing settings for the
+provided data attribute.
+<4> The composable index templates that were updated to not contain custom routing settings for the
+provided data attribute.
+<5> The component templates that were updated to not contain custom routing settings for the
+provided data attribute.
+
+. <<ilm-start,Restart>> {ilm}
++
+[source,console]
+----
+POST /_ilm/start
+----
++
+The response will look like this:
++
+[source,console-result]
+------------------------------------------------------------------------------
+{
+  "acknowledged": true
+}
+------------------------------------------------------------------------------
+// TESTRESPONSE[skip:didn't wait to stop it]
+
+// end::self-managed[]
+

docs/reference/troubleshooting.asciidoc

@@ -14,5 +14,6 @@ include::troubleshooting/data/increase-shard-limit.asciidoc[]
 
 include::troubleshooting/data/increase-cluster-shard-limit.asciidoc[]
 
-include::troubleshooting/data/diagnose-unassigned-shards.asciidoc[]
+include::troubleshooting/data/data-tiers-mixed-with-node-attr.asciidoc[]
 
+include::troubleshooting/data/diagnose-unassigned-shards.asciidoc[]

docs/reference/troubleshooting/data/data-tiers-mixed-with-node-attr.asciidoc

@@ -0,0 +1,18 @@
+[[troubleshoot-migrate-to-tiers]]
+== Indices mix index allocation filters with data tiers node roles to move through data tiers
+
+Elasticsearch standardized the implementation of https://www.elastic.co/blog/elasticsearch-data-lifecycle-management-with-data-tiers[hot-warm-cold architectures]
+to <<data-tiers, data tiers>> in version 7.10.
+Some indices and deployments might have not fully transitioned to <<data-tiers, data tiers>>
+and mix the new way of implementing the hot-warm-cold architecture with 
+<<shard-allocation-filtering, legacy>> based node attributes.
+
+This could lead to unassigned shards or shards not transitioning to the desired
+<<data-tiers, tier>>.
+
+In order to fix this follow the next steps:
+
+include::{es-repo-dir}/tab-widgets/troubleshooting/data/migrate-to-data-tiers-routing-guide-widget.asciidoc[]
+
+
+