[DOCS] Add troubleshooting searches guide (#89583)

* [DOCS] Adds troubleshooting searches guide

* Additional troubleshooting steps

* Apply review suggestions

* Replace separate _cat aliases/indices requests with one get indices call

* Reorder steps to move field caps forward

* Add note about ignore_unavailable

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

docs/reference/troubleshooting.asciidoc

@@ -44,6 +44,7 @@ fix problems that an {es} deployment might encounter.
 * <<monitoring-troubleshooting,Troubleshooting monitoring>>
 * <<transform-troubleshooting,Troubleshooting transforms>>
 * <<watcher-troubleshooting,Troubleshooting Watcher>>
+* <<troubleshooting-searches,Troubleshooting searches>>
 
 If none of these solutions relate to your issue, you can still get help:
 
@@ -102,4 +103,6 @@ include::monitoring/troubleshooting.asciidoc[]
 
 include::transform/troubleshooting.asciidoc[leveloffset=+1]
 
-include::../../x-pack/docs/en/watcher/troubleshooting.asciidoc[]
+include::../../x-pack/docs/en/watcher/troubleshooting.asciidoc[]
+
+include::troubleshooting/troubleshooting-searches.asciidoc[]

docs/reference/troubleshooting/troubleshooting-searches.asciidoc

@@ -0,0 +1,300 @@
+
+[[troubleshooting-searches]]
+== Troubleshooting searches
+
+When you query your data, Elasticsearch may return an error, no search results,
+or results in an unexpected order. This guide describes how to troubleshoot
+searches.
+
+[discrete]
+[[troubleshooting-searches-exists]]
+=== Ensure the data stream, index, or alias exists
+
+Elasticsearch returns an `index_not_found_exception` when the data stream, index
+or alias you try to query does not exist. This can happen when you misspell the
+name or when the data has been indexed to a different data stream or index.
+
+Use the <<indices-exists,exists API>> to check whether a data stream, index, or
+alias exists:
+
+[source,console]
+----
+HEAD my-data-stream
+----
+
+Use the <<data-stream-stats-api,data stream stats API>> to list all data
+streams:
+
+[source,console]
+----
+GET /_data_stream/_stats?human=true
+----
+
+Use the <<indices-get-index,get index API>> to list all indices and their
+aliases:
+
+[source,console]
+----
+GET _all?filter_path=*.aliases
+----
+
+Instead of an error, it is possible to retrieve partial search results if some
+of the indices you're querying are unavailable. Set `ignore_unavailable` to
+`true`:
+
+[source,console]
+----
+GET /my-alias/_search?ignore_unavailable=true
+----
+
+[discrete]
+[[troubleshooting-searches-data]]
+=== Ensure the data stream or index contains data
+
+When a search request returns no hits, the data stream or index may contain no
+data. This can happen when there is a data ingestion issue. For example, the
+data may have been indexed to a data stream or index with another name.
+
+Use the <<search-count,count API>> to retrieve the number of documents in a data
+stream or index. Check that `count` in the response is not 0.
+
+////
+[source,console]
+----
+PUT my-index-000001
+{
+  "mappings": {
+    "properties": {
+      "my-field": {
+        "type": "keyword"
+      },
+      "my-num-field": {
+        "type": "integer"
+      }
+    }
+  }
+}
+----
+////
+
+[source,console]
+----
+GET /my-index-000001/_count
+----
+//TEST[continued]
+
+NOTE: When getting no search results in {kib}, check that you have selected the
+correct data view and a valid time range. Also, ensure the data view has been
+configured with the correct time field.
+
+[discrete]
+[[troubleshooting-searches-field-exists-caps]]
+=== Check that the field exists and its capabilities
+
+Querying a field that does not exist will not return any results. Use the
+<<search-field-caps,field capabilities API>> to check whether a field exists:
+
+[source,console]
+----
+GET /my-index-000001/_field_caps?fields=my-field
+----
+//TEST[continued]
+
+If the field does not exist, check the data ingestion process. The field may
+have a different name.
+
+If the field exists, the request will return the field's type and whether it is
+searchable and aggregatable.
+
+[source,console-response]
+----
+{
+  "indices": [
+    "my-index-000001"
+  ],
+  "fields": {
+    "my-field": {
+      "keyword": {
+        "type": "keyword",         <1>
+        "metadata_field": false,
+        "searchable": true,        <2>
+        "aggregatable": true       <3>
+      }
+    }
+  }
+}
+----
+
+<1> The field is of type `keyword` in this index.
+<2> The field is searchable in this index.
+<3> The field is aggregatable in this index.
+
+[discrete]
+[[troubleshooting-searches-mappings]]
+=== Check the field's mappings
+
+A field's capabilities are determined by its <<mapping,mapping>>. To retrieve
+the mapping, use the <<indices-get-mapping,get mapping API>>:
+
+[source,console]
+----
+GET /my-index-000001/_mappings
+----
+//TEST[continued]
+
+If you query a `text` field, pay attention to the analyzer that may have been
+configured. You can use the <<indices-analyze,analyze API>> to check how a
+field's analyzer processes values and query terms:
+
+[source,console]
+----
+GET /my-index-000001/_analyze
+{
+  "field" : "my-field",
+  "text" : "this is a test"
+}
+----
+//TEST[continued]
+
+To change the mapping of an existing field, refer to
+<<updating-field-mappings,Changing the mapping of a field>>.
+
+[discrete]
+[[troubleshooting-check-field-values]]
+=== Check the field's values
+
+Use the <<query-dsl-exists-query,`exists` query>> to check whether there are
+documents that return a value for a field. Check that `count` in the response is
+not 0.
+
+[source,console]
+----
+GET /my-index-000001/_count
+{
+  "query": {
+    "exists": {
+      "field": "my-field"
+    }
+  }
+}
+----
+//TEST[continued]
+
+If the field is aggregatable, you can use <<search-aggregations,aggregations>>
+to check the field's values. For `keyword` fields, you can use a
+<<search-aggregations-bucket-terms-aggregation,terms aggregation>> to retrieve
+the field's most common values:
+
+[source,console]
+----
+GET /my-index-000001/_search?filter_path=aggregations
+{
+  "size": 0,
+  "aggs": {
+    "top_values": {
+      "terms": {
+        "field": "my-field",
+        "size": 10
+      }
+    }
+  }
+}
+----
+//TEST[continued]
+
+For numeric fields, you can use the
+<<search-aggregations-metrics-stats-aggregation,stats aggregation>> to get an
+idea of the field's value distribution:
+
+[source,console]
+----
+GET my-index-000001/_search?filter_path=aggregations
+{
+  "aggs": {
+    "my-num-field-stats": {
+      "stats": {
+        "field": "my-num-field"
+      }
+    }
+  }
+}
+----
+//TEST[continued]
+
+If the field does not return any values, check the data ingestion process. The
+field may have a different name.
+
+[discrete]
+[[troubleshooting-searches-validate-explain-profile]]
+=== Validate, explain, and profile queries
+
+When a query returns unexpected results, Elasticsearch offers several tools to
+investigate why.
+
+The <<search-validate,validate API>> enables you to validate a query. Use the
+`rewrite` parameter to return the Lucene query an Elasticsearch query is
+rewritten into:
+
+[source,console]
+--------------------------------------------------
+GET /my-index-000001/_validate/query?rewrite=true
+{
+  "query": {
+    "match": {
+      "user.id": {
+        "query": "kimchy",
+        "fuzziness": "auto"
+      }
+    }
+  }
+}
+--------------------------------------------------
+//TEST[continued]
+
+Use the <<search-explain,explain API>> to find out why a specific document
+matches or doesn’t match a query:
+
+[source,console]
+--------------------------------------------------
+GET /my-index-000001/_explain/0
+{
+  "query" : {
+    "match" : { "message" : "elasticsearch" }
+  }
+}
+--------------------------------------------------
+// TEST[setup:messages]
+
+The <<search-profile,profile API>> provides detailed timing information about a
+search request. For a visual representation of the results, use the
+{kibana-ref}/xpack-profiler.html[Search Profiler] in {kib}.
+
+NOTE: To troubleshoot queries in {kib}, select **Inspect** in the toolbar. Next,
+select **Request**. You can now copy the query {kib} sent to {es} for
+further analysis in Console.
+
+[discrete]
+[[troubleshooting-searches-settings]]
+=== Check index settings
+
+<<index-modules-settings,Index settings>> can influence search results. For
+example, the `index.query.default_field` setting, which determines the field
+that is queried when a query specifies no explicit field. Use the
+<<indices-get-settings,get index settings API>> to retrieve the settings for an
+index:
+
+[source,console]
+----
+GET /my-index-000001/_settings
+----
+//TEST[continued]
+
+You can update dynamic index settings with the <<indices-update-settings,update
+index settings API>>. <<change-dynamic-index-setting-for-a-data-stream,Changing
+dynamic index settings for a data stream>> requires changing the index template
+used by the data stream.
+
+For static settings, you need to create a new index with the correct settings.
+Next, you can reindex the data into that index. For data streams, refer to
+<<change-static-index-setting-for-a-data-stream,Change a static index setting
+for a data stream>>.