Browse Source

[DOCS] Adding page for indexing runtime fields (#71366)

* [DOCS] Adding page for indexing runtime fields

* Fixing tests.

* Incorporating review feedback to enhance and improve examples.

* Changing note to indicate immutable script when indexing, plus adding on_script_error.
Adam Locke 4 years ago
parent
commit
343c52c19f
1 changed files with 320 additions and 0 deletions
  1. 320 0
      docs/reference/mapping/runtime.asciidoc

+ 320 - 0
docs/reference/mapping/runtime.asciidoc

@@ -747,6 +747,326 @@ address.
 // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
 // TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]
 
+[[runtime-indexed]]
+=== Index a runtime field
+Runtime fields are defined by the context where they run. For example, you
+can define runtime fields in the
+<<runtime-search-request,context of a search query>> or within the
+<<runtime-mapping-fields,`runtime` section>> of an index mapping. If you
+decide to index a runtime field for greater performance, just move the full
+runtime field definition (including the script) to the context of an index
+mapping. This capability means you can write a script only once, and apply
+it to any context that supports runtime fields.
+
+IMPORTANT: After indexing a runtime field, you cannot update the included
+script. If you need to change the script, create a new field with the updated
+script.
+
+For example, let's say your company wants to replace some old pressure
+valves. The connected sensors are only capable of reporting a fraction of
+the true readings. Rather than outfit the pressure valves with new sensors,
+you decide to calculate the values based on reported readings. Based on the
+reported data, you define the following fields in your mapping for
+`my-index`:
+
+[source,console]
+----
+PUT my-index/
+{
+  "mappings": {
+    "properties": {
+      "timestamp": {
+        "type": "date"
+      },
+      "temperature": {
+        "type": "long"
+      },
+      "voltage": {
+        "type": "double"
+      },
+      "node": {
+        "type": "keyword"
+      }
+    }
+  }
+}
+----
+
+You then bulk index some sample data from your sensors. This data includes
+`voltage` readings for each sensor:
+
+[source,console]
+----
+POST my-index/_bulk?refresh=true
+{"index":{}}
+{"timestamp": 1516729294000, "temperature": 200, "voltage": 5.2, "node": "a"}
+{"index":{}}
+{"timestamp": 1516642894000, "temperature": 201, "voltage": 5.8, "node": "b"}
+{"index":{}}
+{"timestamp": 1516556494000, "temperature": 202, "voltage": 5.1, "node": "a"}
+{"index":{}}
+{"timestamp": 1516470094000, "temperature": 198, "voltage": 5.6, "node": "b"}
+{"index":{}}
+{"timestamp": 1516383694000, "temperature": 200, "voltage": 4.2, "node": "c"}
+{"index":{}}
+{"timestamp": 1516297294000, "temperature": 202, "voltage": 4.0, "node": "c"}
+----
+// TEST[continued]
+
+After talking to a few site engineers, you realize that the sensors should
+be reporting at least _double_ the current values, but potentially higher.
+You create a runtime field named `voltage_corrected` that retrieves the current
+voltage and multiplies it by `2`:
+
+[source,console]
+----
+PUT my-index/_mapping
+{
+  "runtime": {
+    "voltage_corrected": {
+      "type": "double",
+      "script": {
+        "source": """
+        emit(doc['voltage'].value * params['multiplier'])
+        """,
+        "params": {
+          "multiplier": 2
+        }
+      }
+    }
+  }
+}
+----
+// TEST[continued]
+
+You retrieve the calculated values using the <<search-fields,`fields`>>
+parameter on the `_search` API:
+
+[source,console]
+----
+GET my-index/_search
+{
+  "fields": [
+    "voltage_corrected",
+    "node"
+  ],
+  "size": 2
+}
+----
+// TEST[continued]
+// TEST[s/_search/_search\?filter_path=hits/]
+
+//
+////
+[source,console-result]
+----
+{
+  "hits" : {
+    "total" : {
+      "value" : 6,
+      "relation" : "eq"
+    },
+    "max_score" : 1.0,
+    "hits" : [
+      {
+        "_index" : "my-index",
+        "_id" : "z4TCrHgBdg9xpPrU6z9k",
+        "_score" : 1.0,
+        "_source" : {
+          "timestamp" : 1516729294000,
+          "temperature" : 200,
+          "voltage" : 5.2,
+          "node" : "a"
+        },
+        "fields" : {
+          "voltage_corrected" : [
+            10.4
+          ],
+          "node" : [
+            "a"
+          ]
+        }
+      },
+      {
+        "_index" : "my-index",
+        "_id" : "0ITCrHgBdg9xpPrU6z9k",
+        "_score" : 1.0,
+        "_source" : {
+          "timestamp" : 1516642894000,
+          "temperature" : 201,
+          "voltage" : 5.8,
+          "node" : "b"
+        },
+        "fields" : {
+          "voltage_corrected" : [
+            11.6
+          ],
+          "node" : [
+            "b"
+          ]
+        }
+      }
+    ]
+  }
+}
+----
+// TESTRESPONSE[s/"_id" : "z4TCrHgBdg9xpPrU6z9k"/"_id": $body.hits.hits.0._id/]
+// TESTRESPONSE[s/"_id" : "0ITCrHgBdg9xpPrU6z9k"/"_id": $body.hits.hits.1._id/]
+////
+//
+
+After reviewing the sensor data and running some tests, you determine that the
+multiplier for reported sensor data should be `4`. To gain greater performance,
+you decide to index the `voltage_corrected` runtime field with the new
+`multiplier` parameter.
+
+In a new index named `my-index-00001`, copy the `voltage_corrected` runtime
+field definition into the mappings of the new index. It's that simple! You can
+add an optional parameter named `on_script_error` that determines whether to
+reject the entire document if the script throws an error at index time
+(default).
+
+[source,console]
+----
+PUT my-index-00001/
+{
+  "mappings": {
+    "properties": {
+      "timestamp": {
+        "type": "date"
+      },
+      "temperature": {
+        "type": "long"
+      },
+      "voltage": {
+        "type": "double"
+      },
+      "node": {
+        "type": "keyword"
+      },
+      "voltage_corrected": {
+        "type": "double",
+        "on_script_error": "reject", <1>
+        "script": {
+          "source": """
+        emit(doc['voltage'].value * params['multiplier'])
+        """,
+          "params": {
+            "multiplier": 4
+          }
+        }
+      }
+    }
+  }
+}
+----
+<1> Causes the entire document to be rejected if the script throws an error at
+index time. Setting the value to `ignore` will register the field in the
+document’s `_ignored` metadata field and continue indexing.
+
+Bulk index some sample data from your sensors into the `my-index-00001` index:
+
+[source,console]
+----
+POST my-index-00001/_bulk?refresh=true
+{ "index": {}}
+{ "timestamp": 1516729294000, "temperature": 200, "voltage": 5.2, "node": "a"}
+{ "index": {}}
+{ "timestamp": 1516642894000, "temperature": 201, "voltage": 5.8, "node": "b"}
+{ "index": {}}
+{ "timestamp": 1516556494000, "temperature": 202, "voltage": 5.1, "node": "a"}
+{ "index": {}}
+{ "timestamp": 1516470094000, "temperature": 198, "voltage": 5.6, "node": "b"}
+{ "index": {}}
+{ "timestamp": 1516383694000, "temperature": 200, "voltage": 4.2, "node": "c"}
+{ "index": {}}
+{ "timestamp": 1516297294000, "temperature": 202, "voltage": 4.0, "node": "c"}
+----
+// TEST[continued]
+
+You can now retrieve calculated values in a search query, and find documents
+based on precise values. The following range query returns all documents where
+the calculated `voltage_corrected` is greater than or equal to `10`, but less
+than or equal to `16`. Again, use the <<search-fields,`fields`>> parameter on
+the `_search` API to retrieve the fields you want:
+
+[source,console]
+----
+POST my-index-00001/_search
+{
+  "query": {
+    "range": {
+      "voltage_corrected": {
+        "gte": 16,
+        "lte": 20,
+        "boost": 1.0
+      }
+    }
+  },
+  "fields": [
+    "voltage_corrected", "node"]
+}
+----
+// TEST[continued]
+// TEST[s/_search/_search\?filter_path=hits/]
+
+The response includes the `voltage_corrected` field for the documents that
+match the range query, based on the calculated value of the included script:
+
+[source,console-result]
+----
+{
+  "hits" : {
+    "total" : {
+      "value" : 2,
+      "relation" : "eq"
+    },
+    "max_score" : 1.0,
+    "hits" : [
+      {
+        "_index" : "my-index-00001",
+        "_id" : "yoSLrHgBdg9xpPrUZz_P",
+        "_score" : 1.0,
+        "_source" : {
+          "timestamp" : 1516383694000,
+          "temperature" : 200,
+          "voltage" : 4.2,
+          "node" : "c"
+        },
+        "fields" : {
+          "voltage_corrected" : [
+            16.8
+          ],
+          "node" : [
+            "c"
+          ]
+        }
+      },
+      {
+        "_index" : "my-index-00001",
+        "_id" : "y4SLrHgBdg9xpPrUZz_P",
+        "_score" : 1.0,
+        "_source" : {
+          "timestamp" : 1516297294000,
+          "temperature" : 202,
+          "voltage" : 4.0,
+          "node" : "c"
+        },
+        "fields" : {
+          "voltage_corrected" : [
+            16.0
+          ],
+          "node" : [
+            "c"
+          ]
+        }
+      }
+    ]
+  }
+}
+----
+// TESTRESPONSE[s/"_id" : "yoSLrHgBdg9xpPrUZz_P"/"_id": $body.hits.hits.0._id/]
+// TESTRESPONSE[s/"_id" : "y4SLrHgBdg9xpPrUZz_P"/"_id": $body.hits.hits.1._id/]
 
 [[runtime-examples]]
 === Explore your data with runtime fields