[DOCS] Updates to the runtime fields docs for 7.11+ (#67484)

* Moving examples to the page for retrieving runtime fields.

* Adding runtime_mappings to request body of search API.

* Updating runtime_mappings properties and adding runtime fields to search your data.

* Updating examples and hopefully fixing build failure.

* Fixing snippet formatting that was causing test failure.

* Adding page in Painless guide for runtime fields.

* Fixing typo.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

docs/painless/painless-guide/index.asciidoc

@@ -1,5 +1,7 @@
 include::painless-walkthrough.asciidoc[]
 
+include::painless-runtime-fields.asciidoc[]
+
 include::painless-datetime.asciidoc[]
 
 include::painless-method-dispatch.asciidoc[]

docs/painless/painless-guide/painless-runtime-fields.asciidoc

@@ -0,0 +1,89 @@
+[[painless-runtime-fields]]
+=== Use Painless scripts in runtime fields
+A runtime field is a field that is evaluated at query time. When you define a
+runtime field, you can immediately use it in search requests, aggregations,
+filtering, and sorting.
+
+When defining a runtime field, you can include a Painless script that is
+evaluated at query time. This script has access to the entire context of a
+document, including the original `_source` and any mapped fields plus their
+values. At query time, the script runs and generates values for each scripted
+field that is included in the query.
+
+You can map a runtime field in the `runtime` section under the mapping
+definition, or define runtime fields that exist only as part of a search
+request. The script syntax is the same, regardless of where you define the
+runtime field.
+
+IMPORTANT: When defining a Painless script to use with runtime fields, you must
+include `emit` to return calculated values.
+
+[discrete]
+[[painless-runtime-fields-mapping]]
+==== Define a runtime field in the mapping
+Add a `runtime` section under the {ref}/runtime-mapping-fields.html[mapping definition] to explore your data without indexing fields.
+
+The script in the following request extracts the day of the week from the
+`@timestamp` field, which is defined as a `date` type. The script calculates
+the day of the week based on the value of `timestamp`, and uses `emit` to
+return the calculated value.
+
+[source,console]
+----
+PUT my-index/
+{
+  "mappings": {
+    "runtime": {
+      "day_of_week": {
+        "type": "keyword",
+        "script": {
+          "source":
+          """emit(doc['@timestamp'].value.dayOfWeekEnum
+          .getDisplayName(TextStyle.FULL, Locale.ROOT))"""
+        }
+      }
+    },
+    "properties": {
+      "timestamp": {"type": "date"}
+    }
+  }
+}
+----
+
+[discrete]
+[[painless-runtime-fields-query]]
+==== Define a runtime field only in a search request
+Use runtime fields in a search request to create a field that exists
+{ref}/runtime-search-request.html[only as part of the query]. You can also {ref}/runtime-override-values.html[override field values] at query time for existing fields without
+modifying the field itself.
+
+This flexibility allows you to experiment with your data schema and fix
+mistakes in your index mapping without reindexing your data.
+
+In the following request, the values for the `day_of_week` field are calculated
+dynamically, and only within the context of this search request:
+
+[source,console]
+----
+GET my-index/_search
+{
+  "runtime_mappings": {
+    "day_of_week": {
+      "type": "keyword",
+      "script": {
+        "source":
+        """emit(doc['@timestamp'].value.dayOfWeekEnum
+        .getDisplayName(TextStyle.FULL, Locale.ROOT))"""
+      }
+    }
+  },
+  "aggs": {
+    "day_of_week": {
+      "terms": {
+        "field": "day_of_week"
+      }
+    }
+  }
+}
+----
+//TEST[continued]

docs/reference/mapping/dynamic/field-mapping.asciidoc

@@ -17,7 +17,7 @@ explicitly map all other data types.
 // tag::dynamic-field-mapping-types-tag[]
 [cols="3"]
 |===
-h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
+h| JSON data type h| `"dynamic":"true"` h| `"dynamic":"runtime"`
  |`null` 2*| No field added
  |`true` or `false` 2*| `boolean`
  |`double` | `float` | `double`

docs/reference/mapping/dynamic/templates.asciidoc

@@ -68,12 +68,12 @@ reordered or deleted after they were initially added.
 [[dynamic-mapping-runtime-fields]]
 ==== Mapping runtime fields in a dynamic template
 If you want {es} to dynamically map new fields of a certain type as runtime
-fields, set `dynamic:runtime` in the index mappings. These fields are not
+fields, set `"dynamic":"runtime"` in the index mappings. These fields are not
 indexed, and are loaded from `_source` at query time.
 
 Alternatively, you can use the default dynamic mapping rules and then create
 dynamic templates to map specific fields as runtime fields. You set
-`dynamic:true` in your index mapping, and then create a dynamic template to map
+`"dynamic":"true"` in your index mapping, and then create a dynamic template to map
 new fields of a certain type as runtime fields.
 
 Let's say you have data where each of the fields start with `ip_`. Based on the
@@ -84,8 +84,8 @@ template that maps new strings as runtime fields of type `ip`.
 The following request defines a dynamic template named `strings_as_ip`. When
 {es} detects new `string` fields matching the `ip*` pattern, it maps those
 fields as runtime fields of type `ip`. Because `ip` fields aren't mapped
-dynamically, you can use this template with either `dynamic:true` or
-`dynamic:runtime`.
+dynamically, you can use this template with either `"dynamic":"true"` or
+`"dynamic":"runtime"`.
 
 [source,console]
 ----
@@ -337,7 +337,7 @@ Here are some examples of potentially useful dynamic templates:
 
 ===== Structured search
 
-When you set `dynamic:true`, {es} will map string fields as a `text` field with
+When you set `"dynamic":"true"`, {es} will map string fields as a `text` field with
 a `keyword` subfield. If you are only indexing structured content and not
 interested in full text search, you can make {es} map your fields
 only as `keyword` fields. However, you must search on the exact same value that

docs/reference/mapping/runtime.asciidoc

@@ -105,6 +105,7 @@ PUT my-index/
 
 The `runtime` section can be any of these data types:
 
+// tag::runtime-data-types[]
 * `boolean`
 * `date`
 * `double`
@@ -112,6 +113,7 @@ The `runtime` section can be any of these data types:
 * `ip`
 * `keyword`
 * `long`
+// end::runtime-data-types[]
 
 Runtime fields with a `type` of `date` can accept the
 <<mapping-date-format,`format`>> parameter exactly as the `date` field type.
@@ -410,10 +412,77 @@ the values of runtime fields. Runtime fields won't display in `_source`, but
 the `fields` API works for all fields, even those that were not sent as part of
 the original `_source`.
 
+[discrete]
+[[runtime-define-field-dayofweek]]
+==== Define a runtime field to calculate the day of week
+For example, the following request adds a runtime field called `day_of_week`.
+The runtime field includes a script that calculates the day of the week based
+on the value of the `@timestamp` field. We'll include `"dynamic":"runtime"` in
+the request so that new fields are added to the mapping as runtime fields.
+
+[source,console]
+----
+PUT my-index/
+{
+  "mappings": {
+    "dynamic": "runtime",
+    "runtime": {
+      "day_of_week": {
+        "type": "keyword",
+        "script": {
+          "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
+        }
+      }
+    },
+    "properties": {
+      "timestamp": {"type": "date"}
+    }
+  }
+}
+----
+
+[discrete]
+[[runtime-ingest-data]]
+==== Ingest some data
+Let's ingest some sample data, which will result in two indexed fields:
+`@timestamp` and `message`.
+
+[source,console]
+----
+POST /my-index/_bulk?refresh
+{ "index": {}}
+{ "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
+{ "index": {}}
+{ "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:30:17-05:00", "message" : "40.135.0.0 - - [2020-04-30T14:30:17-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:30:53-05:00", "message" : "232.0.0.0 - - [2020-04-30T14:30:53-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:12-05:00", "message" : "26.1.0.0 - - [2020-04-30T14:31:12-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:19-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:19-05:00] \"GET /french/splash_inet.html HTTP/1.0\" 200 3781"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:27-05:00", "message" : "252.0.0.0 - - [2020-04-30T14:31:27-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_brdl.gif HTTP/1.0\" 304 0"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_arw.gif HTTP/1.0\" 304 0"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:32-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:32-05:00] \"GET /images/nav_bg_top.gif HTTP/1.0\" 200 929"}
+{ "index": {}}
+{ "@timestamp": "2020-04-30T14:31:43-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:43-05:00] \"GET /french/images/nav_venue_off.gif HTTP/1.0\" 304 0"}
+----
+//TEST[continued]
+
+[discrete]
+[[runtime-search-dayofweek]]
+==== Search for the calculated day of week
 The following request uses the search API to retrieve the `day_of_week` field
-that <<runtime-mapping-fields,this previous request>> defined as a runtime field
-in the mapping. The value for the `day_of_week` field is calculated at query
-time based on the evaluation of the defined script.
+that the original request defined as a runtime field in the mapping. The value
+for this field is calculated dynamically at query time without reindexing
+documents or indexing the `day_of_week` field. This flexibility allows you to
+modify the mapping without changing any field values.
 
 [source,console]
 ----
@@ -428,6 +497,92 @@ GET my-index/_search
 ----
 // TEST[continued]
 
+The previous request returns the `day_of_week` field for all matching documents.
+We can define another runtime field called `client_ip` that also operates on
+the `message` field and will further refine the query:
+
+[source,console]
+----
+PUT /my-index/_mapping
+{
+  "runtime": {
+    "client_ip": {
+      "type": "ip",
+      "script" : {
+      "source" : "String m = doc[\"message\"].value; int end = m.indexOf(\" \"); emit(m.substring(0, end));"
+      }
+    }
+  }
+}
+----
+//TEST[continued]
+
+Run another query, but search for a specific IP address using the `client_ip`
+runtime field:
+
+[source,console]
+----
+GET my-index/_search
+{
+  "size": 1,
+  "query": {
+    "match": {
+      "client_ip": "211.11.9.0"
+    }
+  },
+  "fields" : ["*"]
+}
+----
+//TEST[continued]
+
+This time, the response includes only two hits. The value for `day_of_week`
+(`Sunday`) was calculated at query time using the runtime script defined in the
+mapping, and the result includes only documents matching the `211.11.9.0` IP
+address.
+
+[source,console-result]
+----
+{
+  ...
+  "hits" : {
+    "total" : {
+      "value" : 2,
+      "relation" : "eq"
+    },
+    "max_score" : 1.0,
+    "hits" : [
+      {
+        "_index" : "my-index",
+        "_id" : "oWs5KXYB-XyJbifr9mrz",
+        "_score" : 1.0,
+        "_source" : {
+          "@timestamp" : "2020-06-21T15:00:01-05:00",
+          "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
+        },
+        "fields" : {
+          "@timestamp" : [
+            "2020-06-21T20:00:01.000Z"
+          ],
+          "client_ip" : [
+            "211.11.9.0"
+          ],
+          "message" : [
+            "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
+          ],
+          "day_of_week" : [
+            "Sunday"
+          ]
+        }
+      }
+    ]
+  }
+}
+----
+// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
+// TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
+// TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]
+
+
 [[runtime-examples]]
 === Explore your data with runtime fields
 Consider a large set of log data that you want to extract fields from.
@@ -612,96 +767,3 @@ and determine which fields to index.
 ----
 // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
 // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
-
-[[runtime-examples-calculate-value]]
-==== Define a runtime field to calculate the day of week
-You can add the `day_of_week` field to the mapping using the request from
-<<runtime-mapping-fields,mapping a runtime field>>:
-
-[source,console]
-----
-PUT /my-index/_mapping
-{
-  "runtime": {
-    "day_of_week": {
-      "type": "keyword",
-      "script": {
-        "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
-      }
-    }
-  },
-  "properties": {
-    "timestamp": {
-      "type": "date"
-    }
-  }
-}
-----
-// TEST[continued]
-
-Then, you can re-run the previous search request and also retrieve the day of
-the week based on the `@timestamp` field:
-
-[source,console]
-----
-GET my-index/_search
-{
-  "size": 1,
-  "query": {
-    "match": {
-      "clientip": "211.11.9.0"
-    }
-  },
-  "fields" : ["*"]
-}
-----
-// TEST[continued]
-
-The value for this field is calculated dynamically at runtime without
-reindexing the document or adding the `day_of_week` field. This flexibility
-allows you to modify the mapping without changing any field values.
-
-In the following response, the value for `day_of_week` (`Sunday`) was
-calculated at query time using the runtime script defined in the mapping.
-
-[source,console-result]
-----
-{
-  ...
-  "hits" : {
-    "total" : {
-      "value" : 2,
-      "relation" : "eq"
-    },
-    "max_score" : 1.0,
-    "hits" : [
-      {
-        "_index" : "my-index",
-        "_id" : "oWs5KXYB-XyJbifr9mrz",
-        "_score" : 1.0,
-        "_source" : {
-          "@timestamp" : "2020-06-21T15:00:01-05:00",
-          "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
-        },
-        "fields" : {
-          "@timestamp" : [
-            "2020-06-21T20:00:01.000Z"
-          ],
-          "clientip" : [
-            "211.11.9.0"
-          ],
-          "message" : [
-            "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
-          ],
-          "day_of_week" : [
-            "Sunday"
-          ]
-        }
-      }
-    ]
-  }
-}
-----
-// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
-// TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
-// TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]

docs/reference/search/search-your-data/search-your-data.asciidoc

@@ -99,6 +99,72 @@ The API response returns the top 10 documents matching the query in the
 // TESTRESPONSE[s/"took": 5/"took": "$body.took"/]
 // TESTRESPONSE[s/"_id": "kxWFcnMByiguvud1Z8vC"/"_id": "$body.hits.hits.0._id"/]
 
+[discrete]
+[[run-search-runtime-fields]]
+== Define fields that exist only in a query
+Instead of indexing your data and then searching it, you can define
+<<runtime-search-request,runtime fields>> that only exist as part of your
+search query. You specify a `runtime_mappings` section in your search request
+to define the runtime field, which can optionally include a Painless script.
+
+For example, the following query defines a runtime field called `day_of_week`.
+The included script calculates the day of the week based on the value of the
+`@timestamp` field, and uses `emit` to return the calculated value.
+
+The query also includes a <<search-aggregations-bucket-terms-aggregation,terms aggregation>> that operates on `day_of_week`.
+
+[source,console]
+----
+GET /my-index-000001/_search
+{
+  "runtime_mappings": {
+    "day_of_week": {
+      "type": "keyword",
+      "script": {
+        "source":
+        """emit(doc['@timestamp'].value.dayOfWeekEnum
+        .getDisplayName(TextStyle.FULL, Locale.ROOT))"""
+      }
+    }
+  },
+  "aggs": {
+    "day_of_week": {
+      "terms": {
+        "field": "day_of_week"
+      }
+    }
+  }
+}
+----
+// TEST[setup:my_index]
+
+The response includes an aggregation based on the `day_of_week` runtime field.
+Under `buckets` is a `key` with a value of `Sunday`. The query dynamically
+calculated this value based on the script defined in the `day_of_week` runtime
+field without ever indexing the field.
+
+[source,console-result]
+----
+{
+  ...
+  ***
+  "aggregations" : {
+    "day_of_week" : {
+      "doc_count_error_upper_bound" : 0,
+      "sum_other_doc_count" : 0,
+      "buckets" : [
+        {
+          "key" : "Sunday",
+          "doc_count" : 5
+        }
+      ]
+    }
+  }
+}
+----
+// TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
+// TESTRESPONSE[s/\*\*\*/"hits" : $body.hits,/]
+
 [discrete]
 [[common-search-options]]
 === Common search options

docs/reference/search/search.asciidoc

@@ -399,6 +399,40 @@ lower `_score` are not included in the search results.
 (Optional, <<query-dsl,query object>>) Defines the search definition using the
 <<query-dsl,Query DSL>>.
 
+[[search-api-body-runtime]]
+`runtime_mappings`::
+(Optional, object)
+Defines a <<runtime-search-request,runtime field>> in the search request that
+exist only as part of the query. Fields defined in the search request take
+precedence over fields defined with the same name in the index mappings.
++
+.Properties of `runtime_mappings` objects
+[%collapsible%open]
+====
+`type`::
+(Required, string)
+<<mapping-types,Data type>> of the object, which can be any of the following:
++
+include::{es-ref-dir}/mapping/runtime.asciidoc[tag=runtime-data-types]
+
+`script`::
+(Optional, string)
+<<modules-scripting-using,Painless script>> that is executed at query time. The
+script has access to the entire context of a document, including the original
+`_source` and any mapped fields plus their values.
++
+Your script must include `emit` to return calculated values. For
+example:
++
+[source,js]
+----
+"script": {
+  "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
+        }
+----
+// NOTCONSOLE
+====
+
 [[request-body-search-seq-no-primary-term]]
 `seq_no_primary_term`::
 (Optional, Boolean) If `true`, returns sequence number and primary term of the