[DOCS] Overhaul search template docs (#72583)

Changes:

* Adds a tutorial for search templates.
* Adds reference docs for the render search template API.
* Improves parameter documentation for the multi search template API.
* Removes duplicate examples from the search template API, multi search API, and create stored script API docs.
* Splits the source files for the search template API and the multi search template API docs.

docs/reference/scripting/apis/create-stored-script-api.asciidoc

@@ -82,34 +82,14 @@ Contains the script or search template, its parameters, and its language.
 
 `source`::
 (Required, string or object)
-Script or search template.
+For scripts, a string containing the script.
++
+For search templates, an object containing the search template. The object
+supports the same parameters as the <<search-search,search API>>'s request body.
+Also supports https://mustache.github.io/[Mustache] variables. See
+<<search-template>>.
 
 `params`::
 (Optional, object)
 Parameters for the script or search template.
 ====
-
-[[create-stored-script-api-example]]
-==== {api-examples-title}
-
-The following request stores a search template. Search templates must use a
-`lang` of `mustache`.
-
-[source,console]
-----
-PUT _scripts/my-search-template
-{
-  "script": {
-    "lang": "mustache",
-    "source": {
-      "from": "{{from}}{{^from}}0{{/from}}",
-      "size": "{{size}}{{^size}}10{{/size}}",
-      "query": {
-        "match": {
-          "content": "{{query_string}}"
-        }
-      }
-    }
-  }
-}
-----

docs/reference/search.asciidoc

@@ -35,8 +35,9 @@ exception of the <<search-explain,explain API>>.
 [[search-template-apis]]
 === Search templates
 
-* <<search-template>>
+* <<search-template-api>>
 * <<multi-search-template>>
+* <<render-search-template-api>>
 
 [discrete]
 [[eql-search-apis]]
@@ -60,7 +61,11 @@ include::search/scroll-api.asciidoc[]
 
 include::search/clear-scroll-api.asciidoc[]
 
-include::search/search-template.asciidoc[]
+include::search/search-template-api.asciidoc[]
+
+include::search/multi-search-template-api.asciidoc[]
+
+include::search/render-search-template-api.asciidoc[]
 
 include::search/search-shards.asciidoc[]
 

docs/reference/search/multi-search-template-api.asciidoc

@@ -0,0 +1,163 @@
+[[multi-search-template]]
+=== Multi search template API
+++++
+<titleabbrev>Multi search template</titleabbrev>
+++++
+
+Runs multiple <<run-multiple-templated-searches,templated searches>> with a single
+request.
+
+////
+[source,console]
+----
+PUT _scripts/my-search-template
+{
+  "script": {
+    "lang": "mustache",
+    "source": {
+      "query": {
+        "match": {
+          "message": "{{query_string}}"
+        }
+      },
+      "from": "{{from}}",
+      "size": "{{size}}"
+    },
+    "params": {
+      "query_string": "My query string"
+    }
+  }
+}
+
+PUT my-index/_doc/1?refresh
+{
+  "message": "hello world"
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+GET my-index/_msearch/template
+{ }
+{ "id": "my-search-template", "params": { "query_string": "hello world", "from": 0, "size": 10 }}
+{ }
+{ "id": "my-other-search-template", "params": { "query_type": "match_all" }}
+----
+// TEST[s/my-other-search-template/my-search-template/]
+
+[[multi-search-template-api-request]]
+==== {api-request-title}
+
+`GET <target>/_msearch/template`
+
+`GET _msearch/template`
+
+`POST <target>/_msearch/template`
+
+`POST _msearch/template`
+
+[[multi-search-template-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `read`
+<<privileges-list-indices,index privilege>> for the target data stream, index,
+or alias. For cross-cluster search, see <<cross-cluster-configuring>>.
+
+[[multi-search-template-api-path-params]]
+==== {api-path-parms-title}
+
+`<target>`::
+(Optional, string) Comma-separated list of data streams, indices, and aliases to
+search. Supports wildcards (`*`). To search all data streams and indices, omit
+this parameter or use `*`.
+
+[[multi-search-template-api-query-params]]
+==== {api-query-parms-title}
+
+`ccs_minimize_roundtrips`::
+(Optional, Boolean) If `true`, network round-trips are minimized for
+cross-cluster search requests. Defaults to `true`.
+
+`max_concurrent_searches`::
+(Optional, integer) Maximum number of concurrent searches the API can run.
+Defaults to +max(1, (# of <<data-node,data nodes>> *
+min(<<search-threadpool,search thread pool size>>, 10)))+.
+
+`rest_total_hits_as_int`::
+(Optional, Boolean) If `true`, the response returns `hits.total` as an integer.
+If false, it returns `hits.total` as an object. Defaults to `false`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
+
+`typed_keys`::
+(Optional, Boolean) If `true`, the response prefixes aggregation and suggester
+names with their respective types. Defaults to `false`.
+
+[role="child_attributes"]
+[[multi-search-template-api-request-body]]
+==== {api-request-body-title}
+
+The request body must be newline-delimited JSON (NDJSON) in the following
+format:
+
+[source,js]
+----
+<header>\n
+<body>\n
+<header>\n
+<body>\n
+----
+// NOTCONSOLE
+
+Each `<header>` and `<body>` pair represents a search request.
+
+The `<header>` supports the same parameters as the <<search-multi-search,multi
+search API>>'s `<header>`. The `<body>` supports the same parameters as the
+<<search-search-api-request-body,search template API>>'s request body.
+
+include::multi-search.asciidoc[tag=header-params]
+
+`<body>`::
+(Request, object) Parameters for the search.
++
+=====
+include::search-template-api.asciidoc[tag=body-params]
+=====
+
+[[multi-search-template-api-response-codes]]
+==== {api-response-codes-title}
+
+The API returns a `400` status code only if the request itself fails. If one or
+more searches in the request fail, the API returns a `200` status code with an
+`error` object for each failed search in the response.
+
+[[multi-search-template-api-response-body]]
+==== {api-response-body-title}
+
+`responses`::
+(array of objects) Results for each search, returned in the order submitted.
+Each object uses the same properties as the <<search-search,search API>>'s
+response.
++
+If a search fails, the response includes an `error` object containing an error
+message.
+
+[[multi-search-template-api-curl-requests]]
+==== curl requests
+
+If a providing text file or text input to `curl`, use the `--data-binary` flag
+instead of `-d` to preserve newlines.
+
+[source,sh]
+----
+$ cat requests
+{ "index": "my-index" }
+{ "id": "my-search-template", "params": { "query_string": "hello world", "from": 0, "size": 10 }}
+{ "index": "my-other-index" }
+{ "id": "my-other-search-template", "params": { "query_type": "match_all" }}
+
+$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch/template --data-binary "@requests"; echo
+----
+// NOTCONSOLE

docs/reference/search/multi-search.asciidoc

@@ -168,9 +168,10 @@ respective types in the response.
 The request body contains a newline-delimited list of search `<header>` and
 search `<body>` objects.
 
+// tag::header-params[]
 `<header>`::
 (Required, object)
-Contains parameters used to limit or change the subsequent search body request.
+Parameters used to limit or change the search.
 +
 This object is required for each search body but can be empty (`{}`) or a blank
 line.
@@ -235,6 +236,7 @@ Documents are scored using global term and document frequencies across all
 shards. This is usually slower but more accurate.
 --
 ====
+// end::header-params[]
 
 `<body>`::
 (Optional, object)
@@ -335,76 +337,6 @@ all search requests.
 See <<url-access-control>>
 
 
-[[template-msearch]]
-==== Template support
-
-Much like described in <<search-template>> for the _search resource, _msearch
-also provides support for templates. Submit them like follows for inline
-templates:
-
-[source,console]
------------------------------------------------
-GET _msearch/template
-{"index" : "my-index-000001"}
-{ "source" : "{ \"query\": { \"match\": { \"message\" : \"{{keywords}}\" } } } }", "params": { "query_type": "match", "keywords": "some message" } }
-{"index" : "my-index-000001"}
-{ "source" : "{ \"query\": { \"match_{{template}}\": {} } }", "params": { "template": "all" } }
------------------------------------------------
-// TEST[setup:my_index]
-
-
-You can also create search templates:
-
-[source,console]
-------------------------------------------
-POST /_scripts/my_template_1
-{
-  "script": {
-    "lang": "mustache",
-    "source": {
-      "query": {
-        "match": {
-          "message": "{{query_string}}"
-        }
-      }
-    }
-  }
-}
-------------------------------------------
-// TEST[setup:my_index]
-
-
-[source,console]
-------------------------------------------
-POST /_scripts/my_template_2
-{
-  "script": {
-    "lang": "mustache",
-    "source": {
-      "query": {
-        "term": {
-          "{{field}}": "{{value}}"
-        }
-      }
-    }
-  }
-}
-------------------------------------------
-// TEST[continued]
-
-You can use search templates in a _msearch:
-
-[source,console]
------------------------------------------------
-GET _msearch/template
-{"index" : "main"}
-{ "id": "my_template_1", "params": { "query_string": "some message" } }
-{"index" : "main"}
-{ "id": "my_template_2", "params": { "field": "user", "value": "test" } }
------------------------------------------------
-// TEST[continued]
-
-
 [[multi-search-partial-responses]]
 ==== Partial responses
 

docs/reference/search/rank-eval.asciidoc

@@ -205,7 +205,7 @@ GET /my-index-000001/_rank_eval
 <3> a reference to a previously defined template
 <4> the parameters to use to fill the template
 
-It is also possible to use <<pre-registered-templates,templates already stored>>  in the cluster state by referencing their id in the templates section.
+You can also use a <<create-search-template,stored search template>>.
 
 [source,js]
 --------------------------------

docs/reference/search/render-search-template-api.asciidoc

@@ -0,0 +1,89 @@
+[[render-search-template-api]]
+=== Render search template API
+++++
+<titleabbrev>Render search template</titleabbrev>
+++++
+
+Renders a <<search-template,search
+template>> as a <<search-search,search request body>>.
+
+////
+[source,console]
+----
+PUT _scripts/my-search-template
+{
+  "script": {
+    "lang": "mustache",
+    "source": {
+      "query": {
+        "match": {
+          "message": "{{query_string}}"
+        }
+      },
+      "from": "{{from}}",
+      "size": "{{size}}"
+    },
+    "params": {
+      "query_string": "My query string"
+    }
+  }
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+POST _render/template
+{
+  "id": "my-search-template",
+  "params": {
+    "query_string": "hello world",
+    "from": 20,
+    "size": 10
+  }
+}
+----
+
+[[render-search-template-api-request]]
+==== {api-request-title}
+
+`GET _render/template`
+
+`GET _render/template/<template-id>`
+
+`POST _render/template`
+
+`POST _render/template/<template-id>`
+
+[[render-search-template-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `read`
+<<privileges-list-indices,index privilege>> for at least one index pattern.
+
+[[render-search-template-api-path-params]]
+==== {api-path-parms-title}
+
+`<template-id>`::
+(Required*, string) ID of the search template to render. If no `source` is
+specified, this or the `id` request body parameter is required.
+
+[[render-search-template-api-request-body]]
+==== {api-request-body-title}
+
+`id`::
+(Required*, string) ID of the search template to render. If no `source` is
+specified, this or the `<template-id>` request path parameter is required. If
+you specify both this parameter and the `<template-id>` parameter, the API uses
+only `<template-id>`.
+
+`params`::
+(Optional, object) Key-value pairs used to replace Mustache variables in the
+template. The key is the variable name. The value is the variable value.
+
+`source`::
+(Required*, object) An inline search template. Supports the same parameters as
+the <<search-search,search API>>'s request body. These parameters also support
+https://mustache.github.io/[Mustache] variables. If no `id` or `<templated-id>`
+is specified, this parameter is required.

docs/reference/search/search-template-api.asciidoc

@@ -0,0 +1,148 @@
+[[search-template-api]]
+=== Search template API
+++++
+<titleabbrev>Search template</titleabbrev>
+++++
+
+Runs a search with a <<search-template,search template>>.
+
+////
+[source,console]
+----
+PUT _scripts/my-search-template
+{
+  "script": {
+    "lang": "mustache",
+    "source": {
+      "query": {
+        "match": {
+          "message": "{{query_string}}"
+        }
+      },
+      "from": "{{from}}",
+      "size": "{{size}}"
+    },
+    "params": {
+      "query_string": "My query string"
+    }
+  }
+}
+
+PUT my-index/_doc/1?refresh
+{
+  "message": "hello world"
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+GET my-index/_search/template
+{
+  "id": "my-search-template",
+  "params": {
+    "query_string": "hello world",
+    "from": 0,
+    "size": 10
+  }
+}
+----
+
+[[search-template-api-request]]
+==== {api-request-title}
+
+`GET <target>/_search/template`
+
+`GET _search/template`
+
+`POST <target>/_search/template`
+
+`POST _search/template`
+
+[[search-template-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `read`
+<<privileges-list-indices,index privilege>> for the target data stream, index,
+or alias. For cross-cluster search, see <<cross-cluster-configuring>>.
+
+[[search-template-api-path-params]]
+==== {api-path-parms-title}
+
+`<target>`::
+(Optional, string) Comma-separated list of data streams, indices, and aliases to
+search. Supports wildcards (`*`). To search all data streams and indices, omit
+this parameter or use `*`.
+
+[[search-template-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
++
+Defaults to `true`.
+
+`ccs_minimize_roundtrips`::
+(Optional, Boolean) If `true`, network round-trips are minimized for
+cross-cluster search requests. Defaults to `true`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
+
+`explain`::
+(Optional, Boolean) If `true`, the response includes additional details about
+score computation as part of a hit. Defaults to `false`.
+
+`ignore_throttled`::
+(Optional, Boolean) If `true`, specified concrete, expanded, or aliased indices
+are not included in the response when throttled. Defaults to `true`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=preference]
+
+`rest_total_hits_as_int`::
+(Optional, Boolean) If `true`, the response returns `hits.total` as an integer.
+If false, it returns `hits.total` as an object. Defaults to `false`.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=routing]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
+
+`typed_keys`::
+(Optional, Boolean) If `true`, the response prefixes aggregation and suggester
+names with their respective types. Defaults to `false`.
+
+[[search-template-api-request-body]]
+==== {api-request-body-title}
+
+// tag::body-params[]
+`explain`::
+(Optional, Boolean) If `true`, returns detailed information about score
+calculation as part of each hit. Defaults to `false`.
+// end::body-params[]
++
+If you specify both this and the `explain` query parameter, the API uses only
+the query parameter.
+
+// tag::body-params[]
+`id`::
+(Required*, string) ID of the search template to use. If no `source` is
+specified, this parameter is required.
+
+`params`::
+(Optional, object) Key-value pairs used to replace Mustache variables in the
+template. The key is the variable name. The value is the variable value.
+
+`profile`::
+(Optional, Boolean) If `true`, the query execution is profiled. Defaults to
+`false`.
+
+`source`::
+(Required*, object) An inline search template. Supports the same parameters as
+the <<search-search,search API>>'s request body. Also supports
+https://mustache.github.io/[Mustache] variables.
++
+If no `id` is specified, this parameter is required.
+// end::body-params[]

docs/reference/search/search-template.asciidoc

@@ -1,725 +0,0 @@
-[[search-template]]
-=== Search template API
-++++
-<titleabbrev>Search template</titleabbrev>
-++++
-
-Allows you to use the mustache language to pre render search requests.
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source" : {
-    "query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
-    "size" : "{{my_size}}"
-  },
-  "params" : {
-    "my_field" : "message",
-    "my_value" : "foo",
-    "my_size" : 5
-  }
-}
-------------------------------------------
-// TEST[setup:my_index]
-
-[[search-template-api-request]]
-==== {api-request-title}
-
-`GET _search/template`
-
-[[search-template-api-prereqs]]
-==== {api-prereq-title}
-
-* If the {es} {security-features} are enabled, you must have the `read`
-<<privileges-list-indices,index privilege>> for the target data stream, index,
-or alias. For cross-cluster search, see <<cross-cluster-configuring>>.
-
-[[search-template-api-desc]]
-==== {api-description-title}
-
-The `/_search/template` endpoint allows you to use the mustache language to pre-
-render search requests, before they are executed and fill existing templates
-with template parameters.
-
-For more information on how Mustache templating and what kind of templating you
-can do with it check out the https://mustache.github.io/mustache.5.html[online
-documentation of the mustache project].
-
-NOTE: The mustache language is implemented in {es} as a sandboxed scripting
-language, hence it obeys settings that may be used to enable or disable scripts
-per type and context as described in the
-<<allowed-script-types-setting, scripting docs>>.
-
-
-[[search-template-api-path-params]]
-==== {api-path-parms-title}
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index]
-
-
-[[search-template-api-query-params]]
-==== {api-query-parms-title}
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
-+
-Defaults to `true`.
-
-`ccs_minimize_roundtrips`::
-  (Optional, Boolean) If `true`, network round-trips are minimized for
- cross-cluster search requests. Defaults to `true`.
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
-
-`explain`::
-  (Optional, Boolean) If `true`, the response includes additional details about
-  score computation as part of a hit. Defaults to `false`.
-
-`ignore_throttled`::
-  (Optional, Boolean) If `true`, specified concrete, expanded or aliased indices
-  are not included in the response when throttled. Defaults to `true`.
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=preference]
-
-`profile`::
-  (Optional, Boolean) If `true`, the query execution is profiled. Defaults
-  to `false`.
-
-`rest_total_hits_as_int`::
-  (Optional, Boolean) If `true`, `hits.total` are rendered as an integer in
-  the response. Defaults to `false`.
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=routing]
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll]
-
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
-
-`typed_keys`::
-  (Optional, Boolean) If `true`, aggregation and suggester names are
-  prefixed by their respective types in the response. Defaults to `false`.
-
-
-[[search-template-api-request-body]]
-==== {api-request-body-title}
-
-The API request body must contain the search definition template and its parameters.
-
-
-[[search-template-api-example]]
-==== {api-examples-title}
-
-
-[[pre-registered-templates]]
-===== Store a search template
-
-To store a search template, use the <<create-stored-script-api,create stored
-script API>>. Specify `mustache` as the `lang`.
-
-[source,console]
-------------------------------------------
-POST _scripts/<templateid>
-{
-  "script": {
-    "lang": "mustache",
-    "source": {
-      "query": {
-        "match": {
-          "title": "{{query_string}}"
-        }
-      }
-    }
-  }
-}
-------------------------------------------
-// TEST[continued]
-
-//////////////////////////
-
-The API returns the following result if the template has been successfully
-created:
-
-[source,console-result]
---------------------------------------------------
-{
-  "acknowledged" : true
-}
---------------------------------------------------
-
-//////////////////////////
-
-
-To retrieve the template, use the <<get-stored-script-api,get stored script
-API>>.
-
-[source,console]
-------------------------------------------
-GET _scripts/<templateid>
-------------------------------------------
-// TEST[continued]
-
-The API returns:
-
-[source,console-result]
-------------------------------------------
-{
-  "script" : {
-    "lang" : "mustache",
-    "source" : """{"query":{"match":{"title":"{{query_string}}"}}}""",
-    "options": {
-      "content_type" : "application/json;charset=utf-8"
-    }
-  },
-  "_id": "<templateid>",
-  "found": true
-}
-------------------------------------------
-
-To delete the template, use the <<delete-stored-script-api,delete stored script
-API>>.
-
-[source,console]
-------------------------------------------
-DELETE _scripts/<templateid>
-------------------------------------------
-// TEST[continued]
-
-[[use-registered-templates]]
-===== Using a stored search template
-
-To use a stored template at search time send the following request:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "id": "<templateid>", <1>
-  "params": {
-    "query_string": "search for these words"
-  }
-}
-------------------------------------------
-// TEST[catch:missing]
-<1> Name of the stored template script.
-
-
-[[_validating_templates]]
-==== Validating a search template
-
-A template can be rendered in a response with given parameters by using the
-following request:
-
-[source,console]
-------------------------------------------
-GET _render/template
-{
-  "source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
-  "params": {
-    "statuses" : {
-        "status": [ "pending", "published" ]
-    }
-  }
-}
-------------------------------------------
-
-
-The API returns the rendered template:
-
-[source,console-result]
-------------------------------------------
-{
-  "template_output": {
-    "query": {
-      "terms": {
-        "status": [ <1>
-          "pending",
-          "published"
-        ]
-      }
-    }
-  }
-}
-------------------------------------------
-
-<1> `status` array has been populated with values from the `params` object.
-
-
-Stored templates can also be rendered by calling the following request:
-
-[source,js]
-------------------------------------------
-GET _render/template/<template_name>
-{
-  "params": {
-    "..."
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-[[search-template-explain-parameter]]
-===== Using the explain parameter
-
-You can use the `explain` parameter when running a template:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "id": "my_template",
-  "params": {
-    "status": [ "pending", "published" ]
-  },
-  "explain": true
-}
-------------------------------------------
-// TEST[catch:missing]
-
-
-[[search-template-profile-parameter]]
-===== Profiling
-
-You can use the `profile` parameter when running a template:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "id": "my_template",
-  "params": {
-    "status": [ "pending", "published" ]
-  },
-  "profile": true
-}
-------------------------------------------
-// TEST[catch:missing]
-
-
-[[search-template-query-string-single]]
-===== Filling in a query string with a single value
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source": {
-    "query": {
-      "term": {
-        "message": "{{query_string}}"
-      }
-    }
-  },
-  "params": {
-    "query_string": "search for these words"
-  }
-}
-------------------------------------------
-// TEST[setup:my_index]
-
-[[search-template-converting-to-json]]
-===== Converting parameters to JSON
-
-The `{{#toJson}}parameter{{/toJson}}` function can be used to convert parameters
-like maps and array to their JSON representation:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
-  "params": {
-    "statuses" : {
-        "status": [ "pending", "published" ]
-    }
-  }
-}
-------------------------------------------
-
-which is rendered as:
-
-[source,js]
-------------------------------------------
-{
-  "query": {
-    "terms": {
-      "status": [
-        "pending",
-        "published"
-      ]
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-A more complex example substitutes an array of JSON objects:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source": "{\"query\":{\"bool\":{\"must\": {{#toJson}}clauses{{/toJson}} }}}",
-  "params": {
-    "clauses": [
-      { "term": { "user" : "foo" } },
-      { "term": { "user" : "bar" } }
-    ]
-  }
-}
-------------------------------------------
-
-which is rendered as:
-
-[source,js]
-------------------------------------------
-{
-  "query": {
-    "bool": {
-      "must": [
-        {
-          "term": {
-            "user": "foo"
-          }
-        },
-        {
-          "term": {
-            "user": "bar"
-          }
-        }
-      ]
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-[[search-template-concatenate-array]]
-===== Concatenating array of values
-
-The `{{#join}}array{{/join}}` function can be used to concatenate the
-values of an array as a comma delimited string:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source": {
-    "query": {
-      "match": {
-        "emails": "{{#join}}emails{{/join}}"
-      }
-    }
-  },
-  "params": {
-    "emails": [ "username@email.com", "lastname@email.com" ]
-  }
-}
-------------------------------------------
-
-which is rendered as:
-
-[source,js]
-------------------------------------------
-{
-  "query" : {
-    "match" : {
-      "emails" : "username@email.com,lastname@email.com"
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-The function also accepts a custom delimiter:
-
-[source,console]
-------------------------------------------
-GET _search/template
-{
-  "source": {
-    "query": {
-      "range": {
-        "born": {
-            "gte"   : "{{date.min}}",
-            "lte"   : "{{date.max}}",
-            "format": "{{#join delimiter='||'}}date.formats{{/join delimiter='||'}}"
-	    }
-      }
-    }
-  },
-  "params": {
-    "date": {
-        "min": "2016",
-        "max": "31/12/2017",
-        "formats": ["dd/MM/yyyy", "yyyy"]
-    }
-  }
-}
-------------------------------------------
-
-which is rendered as:
-
-[source,js]
-------------------------------------------
-{
-  "query": {
-    "range": {
-      "born": {
-        "gte": "2016",
-        "lte": "31/12/2017",
-        "format": "dd/MM/yyyy||yyyy"
-      }
-    }
-  }
-}
-
-------------------------------------------
-// NOTCONSOLE
-
-[[search-template-default-values]]
-===== Default values
-
-A default value is written as `{{var}}{{^var}}default{{/var}}` for instance:
-
-[source,js]
-------------------------------------------
-{
-  "source": {
-    "query": {
-      "range": {
-        "line_no": {
-          "gte": "{{start}}",
-          "lte": "{{end}}{{^end}}20{{/end}}"
-        }
-      }
-    }
-  },
-  "params": { ... }
-}
-------------------------------------------
-// NOTCONSOLE
-
-When `params` is `{ "start": 10, "end": 15 }` this query would be rendered as:
-
-[source,js]
-------------------------------------------
-{
-  "range": {
-    "line_no": {
-      "gte": "10",
-      "lte": "15"
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-But when `params` is `{ "start": 10 }` this query would use the default value
-for `end`:
-
-[source,js]
-------------------------------------------
-{
-  "range": {
-    "line_no": {
-      "gte": "10",
-      "lte": "20"
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-
-[[search-template-conditional-clauses]]
-===== Conditional clauses
-
-Conditional clauses cannot be expressed using the JSON form of the template.
-Instead, the template *must* be passed as a string. For instance, let's say
-we wanted to run a `match` query on the `line` field, and optionally wanted
-to filter by line numbers, where `start` and `end` are optional.
-
-The `params` would look like:
-
-[source,js]
-------------------------------------------
-{
-  "params": {
-    "text": "words to search for",
-    "line_no": {                      <1>
-      "start": 10,
-      "end": 20
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-<1> The `line_no`, `start`, and `end` parameters are optional.
-
-When written as a query, the template would include invalid JSON, such as
-section markers like `{{#line_no}}`:
-
-[source,js]
-------------------------------------------
-{
-  "query": {
-    "bool": {
-      "must": {
-        "match": {
-          "line": "{{text}}" <1>
-        }
-      },
-      "filter": {
-        {{#line_no}} <2>
-          "range": {
-            "line_no": {
-              {{#start}} <3>
-                "gte": "{{start}}" <4>
-                {{#end}},{{/end}} <5>
-              {{/start}}
-              {{#end}} <6>
-                "lte": "{{end}}" <7>
-              {{/end}}
-            }
-          }
-        {{/line_no}}
-      }
-    }
-  }
-}
-------------------------------------------
-// NOTCONSOLE
-<1> Fill in the value of param `text`
-<2> Include the `range` filter only if `line_no` is specified
-<3> Include the `gte` clause only if `line_no.start` is specified
-<4> Fill in the value of param `line_no.start`
-<5> Add a comma after the `gte` clause only if `line_no.start`
-    AND `line_no.end` are specified
-<6> Include the `lte` clause only if `line_no.end` is specified
-<7> Fill in the value of param `line_no.end`
-
-Because search templates cannot include invalid JSON, you can pass the same
-query as a string instead:
-
-[source,js]
---------------------
-"source": "{\"query\":{\"bool\":{\"must\":{\"match\":{\"line\":\"{{text}}\"}},\"filter\":{{{#line_no}}\"range\":{\"line_no\":{{{#start}}\"gte\":\"{{start}}\"{{#end}},{{/end}}{{/start}}{{#end}}\"lte\":\"{{end}}\"{{/end}}}}{{/line_no}}}}}}"
---------------------
-// NOTCONSOLE
-
-
-[[search-template-encode-urls]]
-===== Encoding URLs
-
-The `{{#url}}value{{/url}}` function can be used to encode a string value
-in a HTML encoding form as defined in by the
-https://www.w3.org/TR/html4/[HTML specification].
-
-As an example, it is useful to encode a URL:
-
-[source,console]
-------------------------------------------
-GET _render/template
-{
-  "source": {
-    "query": {
-      "term": {
-        "http_access_log": "{{#url}}{{host}}/{{page}}{{/url}}"
-      }
-    }
-  },
-  "params": {
-    "host": "https://www.elastic.co/",
-    "page": "learn"
-  }
-}
-------------------------------------------
-
-
-The previous query will be rendered as:
-
-[source,console-result]
-------------------------------------------
-{
-  "template_output": {
-    "query": {
-      "term": {
-        "http_access_log": "https%3A%2F%2Fwww.elastic.co%2F%2Flearn"
-      }
-    }
-  }
-}
-------------------------------------------
-
-
-[[multi-search-template]]
-=== Multi search template API
-++++
-<titleabbrev>Multi search template</titleabbrev>
-++++
-
-Allows to execute several search template requests.
-
-[[multi-search-template-api-request]]
-==== {api-request-title}
-
-`GET _msearch/template`
-
-[[multi-search-template-api-prereqs]]
-==== {api-prereq-title}
-
-* If the {es} {security-features} are enabled, you must have the `read`
-<<privileges-list-indices,index privilege>> for the target data stream, index,
-or alias. For cross-cluster search, see <<cross-cluster-configuring>>.
-
-[[multi-search-template-api-desc]]
-==== {api-description-title}
-
-Allows to execute several search template requests within the same API using the
-`_msearch/template` endpoint.
-
-The format of the request is similar to the <<search-multi-search, Multi
-Search API>> format:
-
-[source,js]
---------------------------------------------------
-header\n
-body\n
-header\n
-body\n
---------------------------------------------------
-// NOTCONSOLE
-
-The header part supports the same `index`, `search_type`, `preference`, and
-`routing` options as the Multi Search API.
-
-The body includes a search template body request and supports inline, stored and
-file templates.
-
-
-[[multi-search-template-api-example]]
-==== {api-examples-title}
-
-[source,js]
---------------------------------------------------
-$ cat requests
-{"index": "test"}
-{"source": {"query": {"match":  {"user" : "{{username}}" }}}, "params": {"username": "john"}} <1>
-{"source": {"query": {"{{query_type}}": {"name": "{{name}}" }}}, "params": {"query_type": "match_phrase_prefix", "name": "Smith"}}
-{"index": "_all"}
-{"id": "template_1", "params": {"query_string": "search for these words" }} <2>
-
-$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch/template --data-binary "@requests"; echo
---------------------------------------------------
-// NOTCONSOLE
-// Not converting to console because this shows how curl works
-<1> Inline search template request
-
-<2> Search template request based on a stored template
-
-The response returns a `responses` array, which includes the search template
-response for each search template request matching its order in the original
-multi search template request. If there was a complete failure for that specific
-search template request, an object with `error` message will be returned in
-place of the actual search response.

docs/reference/search/search-your-data/search-template.asciidoc

@@ -0,0 +1,632 @@
+[[search-template]]
+== Search templates
+
+A search template is a stored search you can run with different variables.
+
+If you use {es} as a search backend, you can pass user input from a search bar
+as parameters for a search template. This lets you run searches without exposing
+{es}'s query syntax to your users.
+
+If you use {es} for a custom application, search templates let you change
+your searches without modifying your app's code.
+
+[discrete]
+[[create-search-template]]
+=== Create a search template
+
+To create or update a search template, use the <<create-stored-script-api,create
+stored script API>>.
+
+The request's `source` supports the same parameters as the
+<<search-search-api-request-body,search API>>'s request body. `source` also
+supports https://mustache.github.io/[Mustache] variables, typically enclosed in
+double curly brackets: `{{my-var}}`. When you run a templated search, {es}
+replaces these variables with values from `params`.
+
+Search templates must use a `lang` of `mustache`.
+
+The following request creates a search template with an `id` of
+`my-search-template`.
+
+[source,console]
+----
+PUT _scripts/my-search-template
+{
+  "script": {
+    "lang": "mustache",
+    "source": {
+      "query": {
+        "match": {
+          "message": "{{query_string}}"
+        }
+      },
+      "from": "{{from}}",
+      "size": "{{size}}"
+    },
+    "params": {
+      "query_string": "My query string"
+    }
+  }
+}
+----
+
+{es} stores search templates as Mustache <<modules-scripting,scripts>> in the
+cluster state. {es} compiles search templates in the `template` script context.
+Settings that limit or disable scripts also affect search templates.
+
+[discrete]
+[[validate-search-template]]
+=== Validate a search template
+
+[[_validating_templates]]
+To test a template with different `params`, use the
+<<render-search-template-api,render search template API>>.
+
+[source,console]
+----
+POST _render/template
+{
+  "id": "my-search-template",
+  "params": {
+    "query_string": "hello world",
+    "from": 20,
+    "size": 10
+  }
+}
+----
+// TEST[continued]
+
+When rendered, the template outputs a <<search-search-api-request-body,search
+request body>>.
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "match": {
+        "message": "hello world"
+      }
+    },
+    "from": "20",
+    "size": "10"
+  }
+}
+----
+
+You can also use the API to test inline templates.
+
+[source,console]
+----
+POST _render/template
+{
+    "source": {
+      "query": {
+        "match": {
+          "message": "{{query_string}}"
+        }
+      },
+      "from": "{{from}}",
+      "size": "{{size}}"
+    },
+  "params": {
+    "query_string": "hello world",
+    "from": 20,
+    "size": 10
+  }
+}
+----
+// TEST[continued]
+
+[discrete]
+[[run-templated-search]]
+=== Run a templated search
+
+To run a search with a search template, use the <<search-template-api,search
+template API>>. You can specify different `params` with each request.
+
+////
+[source,console]
+----
+PUT my-index/_doc/1?refresh
+{
+  "message": "hello world"
+}
+----
+// TEST[continued]
+////
+
+[source,console]
+----
+GET my-index/_search/template
+{
+  "id": "my-search-template",
+  "params": {
+    "query_string": "hello world",
+    "from": 0,
+    "size": 10
+  }
+}
+----
+// TEST[continued]
+
+The response uses the same properties as the <<search-search,search API>>'s
+response.
+
+[source,console-result]
+----
+{
+  "took": 36,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "my-index",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "message": "hello world"
+        }
+      }
+    ]
+  }
+}
+----
+// TESTRESPONSE[s/"took": 36/"took": "$body.took"/]
+
+[discrete]
+[[run-multiple-templated-searches]]
+=== Run multiple templated searches
+
+To run multiple templated searches with a single request, use the
+<<multi-search-template,multi search template API>>. These requests often have
+less overhead and faster speeds than multiple individual searches.
+
+[source,console]
+----
+GET my-index/_msearch/template
+{ }
+{ "id": "my-search-template", "params": { "query_string": "hello world", "from": 0, "size": 10 }}
+{ }
+{ "id": "my-other-search-template", "params": { "query_type": "match_all" }}
+----
+// TEST[continued]
+// TEST[s/my-other-search-template/my-search-template/]
+
+[discrete]
+[[get-search-templates]]
+=== Get search templates
+
+To retrieve a search template, use the <<get-stored-script-api,get stored
+script API>>.
+
+[source,console]
+----
+GET _scripts/my-search-template
+----
+// TEST[continued]
+
+To get a list of all search templates and other stored scripts, use the
+<<cluster-state,cluster state API>>.
+
+[source,console]
+----
+GET _cluster/state/metadata?pretty&filter_path=metadata.stored_scripts
+----
+// TEST[continued]
+
+[discrete]
+[[delete-search-template]]
+=== Delete a search template
+
+To delete a search template, use the <<delete-stored-script-api,delete stored
+script API>>.
+
+[source,console]
+----
+DELETE _scripts/my-search-template
+----
+// TEST[continued]
+
+[discrete]
+[[search-template-set-default-values]]
+=== Set default values
+
+To set a default value for a variable, use the following syntax:
+
+[source,mustache]
+----
+{{my-var}}{{^my-var}}default value{{/my-var}}
+----
+
+If a templated search doesn't specify a value in its `params`, the search uses
+the default value instead. For example, the following template sets defaults
+for `from` and `size`.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": {
+    "query": {
+      "match": {
+        "message": "{{query_string}}"
+      }
+    },
+    "from": "{{from}}{{^from}}0{{/from}}",
+    "size": "{{size}}{{^size}}10{{/size}}"
+  },
+  "params": {
+    "query_string": "hello world"
+  }
+}
+----
+
+[discrete]
+[[search-template-url-encode-strings]]
+=== URL encode strings
+
+Use the `{{#url}}` function to URL encode a string.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": {
+    "query": {
+      "term": {
+        "url.full": "{{#url}}{{host}}/{{page}}{{/url}}"
+      }
+    }
+  },
+  "params": {
+    "host": "http://example.com",
+    "page": "hello-world"
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "term": {
+        "url.full": "http%3A%2F%2Fexample.com%2Fhello-world"
+      }
+    }
+  }
+}
+----
+
+[discrete]
+[[search-template-concatenate-values]]
+=== Concatenate values
+
+Use the `{{#join}}` function to concatenate array values as a comma-delimited
+string. For example, the following template concatenates two email addresses.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": {
+    "query": {
+      "match": {
+        "user.group.emails": "{{#join}}emails{{/join}}"
+      }
+    }
+  },
+  "params": {
+    "emails": [ "user1@example.com", "user_one@example.com" ]
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "match": {
+        "user.group.emails": "user1@example.com,user_one@example.com"
+      }
+    }
+  }
+}
+----
+
+You can also specify a custom delimiter.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": {
+    "query": {
+      "range": {
+        "user.effective.date": {
+          "gte": "{{date.min}}",
+          "lte": "{{date.max}}",
+          "format": "{{#join delimiter='||'}}date.formats{{/join delimiter='||'}}"
+	      }
+      }
+    }
+  },
+  "params": {
+    "date": {
+      "min": "2098",
+      "max": "06/05/2099",
+      "formats": ["dd/MM/yyyy", "yyyy"]
+    }
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "range": {
+        "user.effective.date": {
+          "gte": "2098",
+          "lte": "06/05/2099",
+          "format": "dd/MM/yyyy||yyyy"
+        }
+      }
+    }
+  }
+}
+----
+
+[discrete]
+[[search-template-convert-json]]
+=== Convert to JSON
+
+Use the `{{#toJson}}` function to convert a variable value to its JSON
+representation.
+
+For example, the following template uses `{{#toJson}}` to pass an array. To
+ensure the request body is valid JSON, the `source` is written in the string
+format.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": { \"terms\": { \"tags\": {{#toJson}}tags{{/toJson}} }}}",
+  "params": {
+    "tags": [
+      "prod",
+      "es01"
+    ]
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "terms": {
+        "tags": [
+          "prod",
+          "es01"
+        ]
+      }
+    }
+  }
+}
+----
+
+You can also use `{{#toJson}}` to pass objects.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": {{#toJson}}my_query{{/toJson}} }",
+  "params": {
+    "my_query": {
+      "match_all": { }
+    }
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output" : {
+    "query" : {
+      "match_all" : { }
+    }
+  }
+}
+
+----
+
+You can also pass an array of objects.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": { \"bool\": { \"must\": {{#toJson}}clauses{{/toJson}} }}}",
+  "params": {
+    "clauses": [
+      {
+        "term": {
+          "user.id": "kimchy"
+        }
+      },
+      {
+        "term": {
+          "url.domain": "example.com"
+        }
+      }
+    ]
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output": {
+    "query": {
+      "bool": {
+        "must": [
+          {
+            "term": {
+              "user.id": "kimchy"
+            }
+          },
+          {
+            "term": {
+              "url.domain": "example.com"
+            }
+          }
+        ]
+      }
+    }
+  }
+}
+----
+
+[discrete]
+[[search-template-use-conditions]]
+=== Use conditions
+
+To create if conditions, use the following syntax:
+
+[source,mustache]
+----
+{{#condition}}content{{/condition}}
+----
+
+If the condition variable is `true`, {es} displays its content. For example, the
+following template searches data from the past year if `year_scope` is `true`.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": { \"bool\": { \"filter\": [ {{#year_scope}} { \"range\": { \"@timestamp\": { \"gte\": \"now-1y/d\", \"lt\": \"now/d\" } } }, {{/year_scope}} { \"term\": { \"user.id\": \"{{user_id}}\" }}]}}}",
+  "params": {
+    "year_scope": true,
+    "user_id": "kimchy"
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output" : {
+    "query" : {
+      "bool" : {
+        "filter" : [
+          {
+            "range" : {
+              "@timestamp" : {
+                "gte" : "now-1y/d",
+                "lt" : "now/d"
+              }
+            }
+          },
+          {
+            "term" : {
+              "user.id" : "kimchy"
+            }
+          }
+        ]
+      }
+    }
+  }
+}
+----
+
+If `year_scope` is `false`, the template searches data from any time period.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": { \"bool\": { \"filter\": [ {{#year_scope}} { \"range\": { \"@timestamp\": { \"gte\": \"now-1y/d\", \"lt\": \"now/d\" } } }, {{/year_scope}} { \"term\": { \"user.id\": \"{{user_id}}\" }}]}}}",
+  "params": {
+    "year_scope": false,
+    "user_id": "kimchy"
+  }
+}
+----
+
+The template renders as:
+
+[source,console-result]
+----
+{
+  "template_output" : {
+    "query" : {
+      "bool" : {
+        "filter" : [
+          {
+            "term" : {
+              "user.id" : "kimchy"
+            }
+          }
+        ]
+      }
+    }
+  }
+}
+----
+
+To create if-else conditions, use the following syntax:
+
+[source,mustache]
+----
+{{#condition}}if content{{/condition}} {{^condition}}else content{{/condition}}
+----
+
+For example, the following template searches data from the past year if
+`year_scope` is `true`. Otherwise, it searches data from the past day.
+
+[source,console]
+----
+POST _render/template
+{
+  "source": "{ \"query\": { \"bool\": { \"filter\": [ { \"range\": { \"@timestamp\": { \"gte\": {{#year_scope}} \"now-1y/d\" {{/year_scope}} {{^year_scope}} \"now-1d/d\" {{/year_scope}} , \"lt\": \"now/d\" }}}, { \"term\": { \"user.id\": \"{{user_id}}\" }}]}}}",
+  "params": {
+    "year_scope": true,
+    "user_id": "kimchy"
+  }
+}
+----

docs/reference/search/search-your-data/search-your-data.asciidoc

@@ -524,4 +524,5 @@ include::retrieve-selected-fields.asciidoc[]
 include::search-across-clusters.asciidoc[]
 include::search-multiple-indices.asciidoc[]
 include::search-shard-routing.asciidoc[]
+include::search-template.asciidoc[]
 include::sort-search-results.asciidoc[]

docs/reference/search/suggesters/phrase-suggest.asciidoc

@@ -206,8 +206,8 @@ The response contains suggestions scored by the most likely spelling correction
     Checks each suggestion against the specified `query` to prune suggestions
     for which no matching docs exist in the index. The collate query for a
     suggestion is run only on the local shard from which the suggestion has
-    been generated from. The `query` must be specified and it can be templated,
-    see <<search-template,search templates>> for more information.
+    been generated from. The `query` must be specified and it can be templated.
+    See <<search-template>>.
     The current suggestion is automatically made available as the `{{suggestion}}`
     variable, which should be used in your query. You can still specify
     your own template `params` -- the `suggestion` value will be added to the

rest-api-spec/src/main/resources/rest-api-spec/api/render_search_template.json

@@ -1,7 +1,7 @@
 {
   "render_search_template":{
     "documentation":{
-      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/current/search-template.html#_validating_templates",
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/current/render-search-template-api.html",
       "description":"Allows to use the Mustache language to pre-render a search definition."
     },
     "stability":"stable",

x-pack/docs/en/watcher/transform/search.asciidoc

@@ -96,7 +96,7 @@ The following table lists all available settings for the search
 The search {watcher-transform} support mustache <<templates, templates>>. This
 can either be as part of the body definition or alternatively point to an
 existing template (either defined in a file or
-<<pre-registered-templates,registered>> as a script in Elasticsearch).
+<<create-search-template,stored>> as a script in Elasticsearch).
 
 For example, the following snippet shows a search that refers to the scheduled
 time of the watch: