[Query Rules] Add documentation for rule_query (#97667)

* Add docs for rule query

* Add test

* Fix formatting in rule query dsl

* Remove query string as required from rule query docs

* PR feedback

* Update with API changes

* Expand and clarify 'search using query rules' doc

* Clean up wording

* Update put syntax

* Fix examples after refactor

* Update docs/reference/query-dsl/rule-query.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* PR feedback + update privilege

* PR feedback

* More PR feedback

* Small correction

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

docs/reference/query-dsl/rule-query.asciidoc

@@ -0,0 +1,80 @@
+[role="xpack"]
+[[query-dsl-rule-query]]
+=== Rule query
+++++
+<titleabbrev>Rule</titleabbrev>
+++++
+
+preview::[]
+
+Applies rules configured in the <<query-rules-apis, Query Rules API>> to the query.
+This feature is used to promote documents in the manner of a <<query-dsl-pinned-query>> based on matching defined rules.
+If no matching query rules are defined, the "organic" matches for the query are returned.
+
+==== Example request
+
+////
+
+[source,console]
+--------------------------------------------------
+PUT _query_rules/my-ruleset
+{
+  "rules": [
+    {
+      "rule_id": "my-rule1",
+      "type": "pinned",
+      "criteria": [
+        {
+          "type": "exact",
+          "metadata": "query_string",
+          "values": ["pugs"]
+        }
+      ],
+      "actions": {
+        "ids": [ "id1" ]
+      }
+    }
+  ]
+}
+--------------------------------------------------
+// TESTSETUP
+
+[source,console]
+--------------------------------------------------
+DELETE _query_rules/my-ruleset
+--------------------------------------------------
+// TEARDOWN
+
+////
+
+[source,console]
+--------------------------------------------------
+GET /_search
+{
+  "query": {
+    "rule_query": {
+      "match_criteria": {
+        "query_string": "pugs"
+      },
+      "ruleset_id": "my-ruleset",
+      "organic": {
+        "match": {
+          "description": "pugs"
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+
+[[rule-query-top-level-parameters]]
+==== Top-level parameters for `rule_query`
+
+`ruleset_id`::
+(Required, string) A unique <<query-rules-apis, query ruleset>> ID with query-based rules to match and apply as applicable.
+`match_criteria`::
+(Required, object) Defines the match criteria to apply to rules in the given query ruleset.
+Match criteria should match the keys defined in the `criteria.metadata` field of the rule.
+`organic`::
+(Required, object) Any choice of <<query-dsl, query>> used to return results, that may be modified by matching query rules.
+If no query rules are matched and applied, this query will be executed with no modification.

docs/reference/query-dsl/special-queries.asciidoc

@@ -34,6 +34,9 @@ A query that accepts other queries as json or yaml string.
 <<query-dsl-pinned-query,`pinned` query>>::
 A query that promotes selected documents over others matching a given query.
 
+<<query-dsl-rule-query,`rule` query>>::
+A query that supports applying query-based contextual rules, defined using the <<query-rules-apis,Query Rules API>>, to a given query.
+
 include::distance-feature-query.asciidoc[]
 
 include::mlt-query.asciidoc[]
@@ -48,4 +51,6 @@ include::script-score-query.asciidoc[]
 
 include::wrapper-query.asciidoc[]
 
-include::pinned-query.asciidoc[]
+include::pinned-query.asciidoc[]
+
+include::rule-query.asciidoc[]

docs/reference/query-rules/apis/delete-query-ruleset.asciidoc

@@ -19,7 +19,7 @@ This is a destructive action that is not recoverable.
 [[delete-query-ruleset-prereq]]
 ==== {api-prereq-title}
 
-Requires the `manage_cluster` privilege.
+Requires the `manage_search_query_rules` privilege.
 
 [[delete-query_ruleset-path-params]]
 ==== {api-path-parms-title}
@@ -54,7 +54,7 @@ PUT _query_rules/my-ruleset
                 {
                     "type": "exact",
                     "metadata": "query_string",
-                    "values": ["marvel"]
+                    "values": [ "marvel" ]
                 }
             ],
             "actions": {

docs/reference/query-rules/apis/get-query-ruleset.asciidoc

@@ -18,7 +18,7 @@ Retrieves information about a query ruleset.
 [[get-query-ruleset-prereq]]
 ==== {api-prereq-title}
 
-Requires the `manage_cluster` privilege.
+Requires the `manage_search_query_rules` privilege.
 
 [[get-query-ruleset-path-params]]
 ==== {api-path-parms-title}
@@ -54,7 +54,7 @@ PUT _query_rules/my-ruleset
                 {
                     "type": "exact",
                     "metadata": "query_string",
-                    "values": ["marvel"]
+                    "values": [ "marvel" ]
                 }
             ],
             "actions": {
@@ -71,7 +71,7 @@ PUT _query_rules/my-ruleset
                 {
                     "type": "exact",
                     "metadata": "query_string",
-                    "values": ["dc"]
+                    "values": [ "dc" ]
                 }
             ],
             "actions": {
@@ -119,7 +119,7 @@ A sample response:
                 {
                     "type": "exact",
                     "metadata": "query_string",
-                    "values": ["marvel"]
+                    "values": [ "marvel" ]
                 }
             ],
             "actions": {
@@ -136,7 +136,7 @@ A sample response:
                 {
                     "type": "exact",
                     "metadata": "query_string",
-                    "values": ["dc"]
+                    "values": [ "dc" ]
                 }
             ],
             "actions": {

docs/reference/query-rules/apis/index.asciidoc

@@ -9,7 +9,14 @@ preview::[]
 
 ---
 
-Use query rules APIs to manage tasks and resources related to query rules.
+<<search-using-query-rules,Query rules>> allow you to configure per-query rules that are applied at query time to queries that match the specific rule.
+Query rules are organized into _rulesets_, collections of query rules that are matched against incoming queries.
+Query rules are applied using the <<query-dsl-rule-query, rule query>>.
+
+If a query matches one or more rules in the ruleset, the query is re-written to apply the rules before searching.
+This allows pinning documents for only queries that match a specific term.
+
+Use the following APIs to manage query rulesets:
 
 * <<put-query-ruleset>>
 * <<get-query-ruleset>>

docs/reference/query-rules/apis/list-query-rulesets.asciidoc

@@ -19,7 +19,7 @@ Summary information on the number of rules per ruleset will be returned, and ful
 [[list-query-rules-prereq]]
 ==== {api-prereq-title}
 
-Requires the `manage_cluster` privilege.
+Requires the `manage_search_query_rules` privilege.
 
 [[list-query-rules-path-params]]
 ==== {api-path-parms-title}
@@ -30,9 +30,6 @@ Requires the `manage_cluster` privilege.
 `from`::
 (Optional, integer) The offset from the first result to fetch.
 
-[[list-query-rules-response-codes]]
-==== {api-response-codes-title}
-
 [[list-query-rules-example]]
 ==== {api-examples-title}
 

docs/reference/query-rules/apis/put-query-ruleset.asciidoc

@@ -18,16 +18,14 @@ Creates or updates a query ruleset.
 [[put-query-ruleset-prereqs]]
 ==== {api-prereq-title}
 
-Requires the `manage_cluster` privilege.
+Requires the `manage_search_query_rules` privilege.
 
-[[put-query-ruleset-path-params]]
-==== {api-path-parms-title}
-
-`<body>`::
+[role="child_attributes"]
+[[put-query-ruleset-request-body]]
 (Required, object)
 Contains parameters for a query ruleset:
 
-====
+==== {api-request-body-title}
 `rules`::
 (Required, array of objects)
 The specific rules included in this query ruleset.
@@ -39,24 +37,58 @@ Each rule must have the following information:
 - `type` (Required, string)
   The type of rule. At this time only `pinned` query rule types are allowed.
 - `criteria` (Required, array of objects)
+  The criteria that must be met for the rule to be applied. If multiple criteria are specified for a rule, all criteria must be met for the rule to be applied.
 - `actions` (Required, object)
   The actions to take when the rule is matched. The format of this action depends on the rule type.
 
 Criteria must have the following information:
 
 - `type` (Required, string)
-  The type of criteria. At this time only `exact` criteria types are allowed.
-- `metadata` (Required, string)
-  The metadata field to match against. At this time only `query_string` metadata types are allowed.
-- `value` (Required, string)
-  The value to match against the metadata field.
+  The type of criteria. The following criteria types are supported:
++
+--
+- `exact`
+Only exact matches meet the criteria defined by the rule.
+Applicable for string or numerical values.
+- `fuzzy`
+Exact matches or matches within the allowed {wikipedia}/Levenshtein_distance[Levenshtein Edit Distance] meet the criteria defined by the rule.
+Only applicable for string values.
+- `prefix`
+Matches that start with this value meet the criteria defined by the rule.
+Only applicable for string values.
+- `suffix`
+Matches that end with this value meet the criteria defined by the rule.
+Only applicable for string values.
+- `contains`
+Matches that contain this value anywhere in the field meet the criteria defined by the rule.
+Only applicable for string values.
+- `lt`
+Matches with a value less than this value meet the criteria defined by the rule.
+Only applicable for numerical values.
+- `lte`
+Matches with a value less than or equal to this value meet the criteria defined by the rule.
+Only applicable for numerical values.
+- `gt`
+Matches with a value greater than this value meet the criteria defined by the rule.
+Only applicable for numerical values.
+- `gte`
+Matches with a value greater than or equal to this value meet the criteria defined by the rule.
+Only applicable for numerical values.
+- `always`
+Matches all queries, regardless of input.
+--
+- `metadata` (Optional, string)
+  The metadata field to match against. Required for all criteria types except `global`.
+- `values` (Optional, array of strings)
+  The values to match against the metadata field. Only one value must match for the criteria to be met. Required for all criteria types except `global`.
 
 Actions depend on the rule type.
 For `pinned` rules, actions follow the format specified by the <<query-dsl-pinned-query,Pinned Query>>.
 The following actions are allowed:
 
 - `ids` (Optional, array of strings)
-  The The unique <<mapping-id-field, document IDs>> of the documents to pin. Only one of `ids` or `docs` may be specified, and at least one must be specified.
+  The The unique <<mapping-id-field, document IDs>> of the documents to pin.
+  Only one of `ids` or `docs` may be specified, and at least one must be specified.
 - `docs` (Optional, array of objects)
   The documents to pin. Only one of `ids` or `docs` may be specified, and at least one must be specified.
   You can specify the following attributes for each document:
@@ -68,7 +100,7 @@ The following actions are allowed:
   The unique <<mapping-id-field, document ID>>.
 --
 
-====
+IMPORTANT: Due to limitations within <<query-dsl-pinned-query,Pinned queries>>, you can only pin documents using `ids` or `docs`, but cannot use both in single rule. It is advised to use one or the other in query rulesets, to avoid errors. Additionally, pinned queries have a maximum limit of 100 pinned hits. If multiple matching rules pin more than 100 documents, only the first 100 documents are pinned in the order they are specified in the ruleset.
 
 [[put-query-ruleset-example]]
 ==== {api-examples-title}
@@ -77,8 +109,8 @@ The following example creates a new query ruleset called `my-ruleset`.
 
 Two rules are associated with `my-ruleset`:
 
-- `my-rule1` will pin documents with IDs `id1` and `id2` when the `query_string` exactly matches `marvel`.
-- `my-rule2` will pin documents from different, specified indices with IDs `id3` and `id4` when the `query_string` exactly matches `dc`.
+- `my-rule1` will pin documents with IDs `id1` and `id2` when `user.query` exactly matches `marvel` _or_ `dc` **and** `user.country` exactly matches `us`.
+- `my-rule2` will pin documents from different, specified indices with IDs `id3` and `id4` when the `query_string` fuzzily matches `comic`.
 
 [source,console]
 ----
@@ -89,10 +121,15 @@ PUT _query_rules/my-ruleset
             "rule_id": "my-rule1",
             "type": "pinned",
             "criteria": [
+                {
+                    "type": "contains",
+                    "metadata": "user.query",
+                    "values": [ "marvel", "dc" ]
+                },
                 {
                     "type": "exact",
-                    "metadata": "query_string",
-                    "values": ["marvel"]
+                    "metadata": "user.country",
+                    "values": [ "us" ]
                 }
             ],
             "actions": {
@@ -107,9 +144,9 @@ PUT _query_rules/my-ruleset
             "type": "pinned",
             "criteria": [
                 {
-                    "type": "exact",
+                    "type": "fuzzy",
                     "metadata": "query_string",
-                    "values": ["dc"]
+                    "values": [ "comic" ]
                 }
             ],
             "actions": {

docs/reference/search/search-your-data/search-using-query-rules.asciidoc

@@ -0,0 +1,210 @@
+[[search-using-query-rules]]
+== Searching with query rules
+++++
+<titleabbrev>Searching with query rules</titleabbrev>
+++++
+
+[[query-rules]]
+
+preview::[]
+
+_Query rules_ allow customization of search results for queries that match specified criteria metadata.
+This allows for more control over results, for example ensuring that promoted documents that match defined criteria are returned at the top of the result list.
+Metadata is defined in the query rule, and is matched against the query criteria.
+Query rules use metadata to match a query.
+Metadata is provided as part of the `rule_query` as an object and can be anything that helps differentiate the query, for example:
+
+* A user-entered query string
+* Personalized metadata about users (e.g. country, language, etc)
+* A particular topic
+* A referring site
+* etc.
+
+Query rules define a metadata key that will be used to match the metadata provided in the `rule_query` with the criteria specified in the rule.
+
+When a query rule matches the `rule_query` metadata according to its defined criteria, the query rule action is applied to the underlying `organic_query`.
+
+For example, a query rule could be defined to match a user-entered query string of `pugs` and a country `us` and promote adoptable shelter dogs if the rule query met both criteria.
+
+Rules are defined using the <<query-rules-apis, query rules API>> and searched using the <<query-dsl-rule-query,rule query>>.
+
+[discrete]
+[[query-rule-definition]]
+=== Rule definition
+
+When defining a rule, consider the following:
+
+[discrete]
+[[query-rule-type]]
+==== Rule type
+
+The type of rule we want to apply. For the moment there is a single rule type:
+
+* `pinned` will re-write the query into a <<query-dsl-pinned-query, pinned query>>, pinning specified results matching the query rule at the top of the returned result set.
+
+
+[discrete]
+[[query-rule-criteria]]
+==== Rule criteria
+
+The criteria for which this rule will match. Criteria is defined as `type`, `metadata`, and `values`.
+Allowed criteria types are:
+
+[cols="2*", options="header"]
+|===
+|Type
+|Match Requirements
+
+|`exact`
+|Rule metadata matches the specified value exactly.
+
+|`fuzzy`
+|Rule metadata matches the specified value within an allowed {wikipedia}/Levenshtein_distance[Levenshtein edit distance].
+
+|`prefix`
+|Rule metadata starts with the specified value.
+
+|`suffix`
+|Rule metadata ends with the specified value.
+
+|`contains`
+|Rule metadata contains the specified value.
+
+|`lt`
+|Rule metadata is less than the specified value.
+
+|`lte`
+|Rule metadata is less than or equal to the specified value.
+
+|`gt`
+|Rule metadata is greater than the specified value.
+
+|`gte`
+|Rule metadata is greater than or equal to the specified value.
+
+|`always`
+|Always matches for all rule queries.
+|===
+
+[discrete]
+[[query-rule-actions]]
+==== Rule actions
+
+The actions to take when the rule matches a query:
+
+* `ids` will pin the specified <<mapping-id-field,`_id`>>s.
+* `docs` will pin the specified documents in the specified indices.
+
+Use `ids` when searching over a single index, and `docs` when searching over multiple indices. See <<query-dsl-pinned-query,pinned query>> for details.
+
+[discrete]
+[[add-query-rules]]
+=== Add query rules
+
+You can add query rules using the <<put-query-ruleset>> call. This adds a ruleset containing one or more query rules that will be applied to queries that match their specified criteria.
+
+The following command will create a query ruleset called `my-ruleset` with two pinned document rules:
+
+* The first rule will generate a <<query-dsl-pinned-query>> pinning the <<mapping-id-field,`_id`>>s `id1` and `id2` when the `user.query` metadata value is a fuzzy match to either `puggles` or `pugs` _and_ the user's location is in the US.
+* The second rule will generate a <<query-dsl-pinned-query>> pinning the <<mapping-id-field, `_id`>> of `id3` specifically from the `my-index-000001` index and `id4` from the `my-index-000002` index when the `user.query` metadata value matches `beagles`.
+
+[source,console]
+----
+PUT /_query_rules/my-ruleset
+{
+  "rules": [
+    {
+      "rule_id": "rule1",
+      "type": "pinned",
+      "criteria": [
+        {
+          "type": "fuzzy",
+          "metadata": "user.query",
+          "values": [ "puggles", "pugs" ]
+        },
+        {
+          "type": "exact",
+          "metadata": "user.country",
+          "values": [ "us" ]
+        }
+      ],
+      "actions": {
+        "ids": [
+          "id1",
+          "id2"
+        ]
+      }
+    },
+    {
+      "rule_id": "rule2",
+      "type": "pinned",
+      "criteria": [
+        {
+          "type": "fuzzy",
+          "metadata": "user.query",
+          "values": [ "beagles" ]
+        }
+      ],
+      "actions": {
+        "docs": [
+          {
+            "_index": "my-index-000001",
+            "_id": "id3"
+          },
+          {
+            "_index": "my-index-000002",
+            "_id": "id4"
+          }
+        ]
+      }
+    }
+  ]
+}
+----
+// TEST[skip:TBD]
+
+The API response returns a results of `created` or `updated` depending on whether this was a new or edited ruleset.
+
+[source,console-result]
+----
+{
+  "result": "created"
+}
+----
+// TEST[skip:TBD]
+
+You can use the <<get-query-ruleset>> call to retrieve the ruleset you just created,
+the <<list-query-rulesets>> call to retrieve a summary of all query rulesets,
+and the <<delete-query-ruleset>> call to delete a query ruleset.
+
+[discrete]
+[[rule-query-search]]
+=== Perform a rule query
+
+Once you have defined a query ruleset, you can search this ruleset using the <<query-dsl-rule-query>> query.
+An example query for the `my-ruleset` defined above is:
+
+[source,console]
+----
+GET /my-index-000001/_search
+{
+  "query": {
+    "rule_query": {
+      "organic": {
+        "query_string": {
+          "query": "puggles"
+        }
+      },
+      "match_criteria": {
+        "query_string": "puggles",
+        "user.country": "us"
+      },
+      "ruleset_id": "my-ruleset"
+    }
+  }
+}
+----
+// TEST[skip:TBD]
+
+This rule query will match against `rule1` in the defined query ruleset, and will convert the organic query into a pinned query with `id1` and `id2` pinned as the top hits.
+Any other matches from the `user.query` query specified in the organic query will be returned below the pinned results.

docs/reference/search/search-your-data/search-your-data.asciidoc

@@ -532,3 +532,4 @@ include::search-template.asciidoc[]
 include::sort-search-results.asciidoc[]
 include::knn-search.asciidoc[]
 include::semantic-search.asciidoc[]
+include::search-using-query-rules.asciidoc[]