runtime.asciidoc 25 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791
  1. [[runtime]]
  2. == Runtime fields
  3. beta::[]
  4. A _runtime field_ is a field that is evaluated at query time. Runtime fields
  5. enable you to:
  6. * Add fields to existing documents without reindexing your data
  7. * Start working with your data without understanding how it’s structured
  8. * Override the value returned from an indexed field at query time
  9. * Define fields for a specific use without modifying the underlying schema
  10. You access runtime fields from the search API like any other field, and {es}
  11. sees runtime fields no differently. You can define runtime fields in the
  12. <<runtime-mapping-fields,index mapping>> or in the
  13. <<runtime-search-request,search request>>. Your choice, which is part of the
  14. inherent flexibility of runtime fields.
  15. Runtime fields are useful when working with log data
  16. (see <<runtime-examples,examples>>), especially when you're unsure about the
  17. data structure. Your search speed decreases, but your index size is much
  18. smaller and you can more quickly process logs without having to index them.
  19. [discrete]
  20. [[runtime-benefits]]
  21. === Benefits
  22. Because runtime fields aren't indexed, adding a runtime field doesn't increase
  23. the index size. You define runtime fields directly in the index mapping, saving
  24. storage costs and increasing ingestion speed. You can more quickly ingest
  25. data into the Elastic Stack and access it right away. When you define a runtime
  26. field, you can immediately use it in search requests, aggregations, filtering,
  27. and sorting.
  28. If you make a runtime field an indexed field, you don't need to modify any
  29. queries that refer to the runtime field. Better yet, you can refer to some
  30. indices where the field is a runtime field, and other indices where the field
  31. is an indexed field. You have the flexibility to choose which fields to index
  32. and which ones to keep as runtime fields.
  33. At its core, the most important benefit of runtime fields is the ability to
  34. add fields to documents after you've ingested them. This capability simplifies
  35. mapping decisions because you don't have to decide how to parse your data up
  36. front, and can use runtime fields to amend the mapping at any time. Using
  37. runtime fields allows for a smaller index and faster ingest time, which
  38. combined use less resources and reduce your operating costs.
  39. [discrete]
  40. [[runtime-compromises]]
  41. === Compromises
  42. Runtime fields use less disk space and provide flexibility in how you access
  43. your data, but can impact search performance based on the computation defined in
  44. the runtime script.
  45. To balance search performance and flexibility, index fields that you'll
  46. commonly search for and filter on, such as a timestamp. {es} automatically uses
  47. these indexed fields first when running a query, resulting in a fast response
  48. time. You can then use runtime fields to limit the number of fields that {es}
  49. needs to calculate values for. Using indexed fields in tandem with runtime
  50. fields provides flexibility in the data that you index and how you define
  51. queries for other fields.
  52. Use the <<async-search,asynchronous search API>> to run searches that include
  53. runtime fields. This method of search helps to offset the performance impacts
  54. of computing values for runtime fields in each document containing that field.
  55. If the query can't return the result set synchronously, you'll get results
  56. asynchronously as they become available.
  57. IMPORTANT: Queries against runtime fields are considered expensive. If
  58. <<query-dsl-allow-expensive-queries,`search.allow_expensive_queries`>> is set
  59. to `false`, expensive queries are not allowed and {es} will reject any queries
  60. against runtime fields.
  61. [[runtime-mapping-fields]]
  62. === Map a runtime field
  63. beta::[]
  64. You map runtime fields by adding a `runtime` section under the mapping
  65. definition and defining
  66. <<modules-scripting-using,a Painless script>>. This script has access to the
  67. entire context of a document, including the original `_source` and any mapped
  68. fields plus their values. At query time, the script runs and generates values
  69. for each scripted field that is required for the query.
  70. When defining a Painless script to use with runtime fields, you must include
  71. `emit` to emit calculated values. For example, the script in the following
  72. request extracts the day of the week from the `@timestamp` field, which is
  73. defined as a `date` type. The script calculates the day of the week based on
  74. the value of `timestamp`, and uses `emit` to return the calculated value.
  75. [source,console]
  76. ----
  77. PUT my-index/
  78. {
  79. "mappings": {
  80. "runtime": {
  81. "day_of_week": {
  82. "type": "keyword",
  83. "script": {
  84. "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
  85. }
  86. }
  87. },
  88. "properties": {
  89. "timestamp": {"type": "date"}
  90. }
  91. }
  92. }
  93. ----
  94. The `runtime` section can be any of these data types:
  95. // tag::runtime-data-types[]
  96. * `boolean`
  97. * `date`
  98. * `double`
  99. * `geo_point`
  100. * `ip`
  101. * `keyword`
  102. * `long`
  103. // end::runtime-data-types[]
  104. Runtime fields with a `type` of `date` can accept the
  105. <<mapping-date-format,`format`>> parameter exactly as the `date` field type.
  106. If <<dynamic-field-mapping,dynamic field mapping>> is enabled where the
  107. `dynamic` parameter is set to `runtime`, new fields are automatically added to
  108. the index mapping as runtime fields:
  109. [source,console]
  110. ----
  111. PUT my-index
  112. {
  113. "mappings": {
  114. "dynamic": "runtime",
  115. "properties": {
  116. "timestamp": {
  117. "type": "date"
  118. }
  119. }
  120. }
  121. }
  122. ----
  123. [[runtime-fields-scriptless]]
  124. ==== Define runtime fields without a script
  125. You can define a runtime field in the mapping definition without a
  126. script. At query time, {es} looks in `_source` for a field with the same name
  127. and returns a value if one exists. If a field with the same name doesn’t
  128. exist, the response doesn't include any values for that runtime field.
  129. [source,console]
  130. ----
  131. PUT my-index/
  132. {
  133. "mappings": {
  134. "runtime": {
  135. "day_of_week": {
  136. "type": "keyword"
  137. }
  138. }
  139. }
  140. }
  141. ----
  142. [[runtime-updating-scripts]]
  143. ==== Updating and removing runtime fields
  144. You can update or remove runtime fields at any time. To replace an existing
  145. runtime field, add a new runtime field to the mappings with the same name. To
  146. remove a runtime field from the mappings, set the value of the runtime field to
  147. `null`:
  148. [source,console]
  149. ----
  150. PUT my-index/_mapping
  151. {
  152. "runtime": {
  153. "day_of_week": null
  154. }
  155. }
  156. ----
  157. //TEST[continued]
  158. .Downstream impacts
  159. ****
  160. Updating or removing a runtime field while a dependent query is running can return
  161. inconsistent results. Each shard might have access to different versions of the
  162. script, depending on when the mapping change takes effect.
  163. Existing queries or visualizations in {kib} that rely on runtime fields can
  164. fail if you remove or update the field. For example, a bar chart visualization
  165. that uses a runtime field of type `ip` will fail if the type is changed
  166. to `boolean`, or if the runtime field is removed.
  167. ****
  168. [[runtime-search-request]]
  169. === Define runtime fields in a search request
  170. beta::[]
  171. You can specify a `runtime_mappings` section in a search request to create
  172. runtime fields that exist only as part of the query. You specify a script
  173. as part of the `runtime_mappings` section, just as you would if adding a
  174. runtime field to the mappings.
  175. Fields defined in the search request take precedence over fields defined with
  176. the same name in the index mappings. This flexibility allows you to shadow
  177. existing fields and calculate a different value in the search request, without
  178. modifying the field itself. If you made a mistake in your index mapping, you
  179. can use runtime fields to calculate values that override values in the mapping
  180. during the search request.
  181. In the following request, the values for the `day_of_week` field are calculated
  182. dynamically, and only within the context of this search request:
  183. [source,console]
  184. ----
  185. GET my-index/_search
  186. {
  187. "runtime_mappings": {
  188. "day_of_week": {
  189. "type": "keyword",
  190. "script": {
  191. "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
  192. }
  193. }
  194. },
  195. "aggs": {
  196. "day_of_week": {
  197. "terms": {
  198. "field": "day_of_week"
  199. }
  200. }
  201. }
  202. }
  203. ----
  204. //TEST[continued]
  205. Defining a runtime field in a search request uses the same format as defining
  206. a runtime field in the index mapping. That consistency means you can promote a
  207. runtime field from a search request to the index mapping by moving the field
  208. definition from `runtime_mappings` in the search request to the `runtime`
  209. section of the index mapping.
  210. [[runtime-override-values]]
  211. === Override field values at query time
  212. beta::[]
  213. If you create a runtime field with the same name as a field that
  214. already exists in the mapping, the runtime field shadows the mapped field. At
  215. query time, {es} evaluates the runtime field, calculates a value based on the
  216. script, and returns the value as part of the query. Because the runtime field
  217. shadows the mapped field, you can override the value returned in search without
  218. modifying the mapped field.
  219. For example, let's say you indexed the following documents into `my-index`:
  220. [source,console]
  221. ----
  222. POST my-index/_bulk?refresh=true
  223. {"index":{}}
  224. {"timestamp":1516729294000,"model_number":"QVKC92Q","measures":{"voltage":5.2}}
  225. {"index":{}}
  226. {"timestamp":1516642894000,"model_number":"QVKC92Q","measures":{"voltage":5.8}}
  227. {"index":{}}
  228. {"timestamp":1516556494000,"model_number":"QVKC92Q","measures":{"voltage":5.1}}
  229. {"index":{}}
  230. {"timestamp":1516470094000,"model_number":"QVKC92Q","measures":{"voltage":5.6}}
  231. {"index":{}}
  232. {"timestamp":1516383694000,"model_number":"HG537PU","measures":{"voltage":4.2}}
  233. {"index":{}}
  234. {"timestamp":1516297294000,"model_number":"HG537PU","measures":{"voltage":4.0}}
  235. ----
  236. You later realize that the `HG537PU` sensors aren't reporting their true
  237. voltage. The indexed values are supposed to be 1.7 times higher than
  238. the reported values! Instead of reindexing your data, you can define a script in
  239. the `runtime_mappings` section of the `_search` request to shadow the `voltage`
  240. field and calculate a new value at query time.
  241. If you search for documents where the model number matches `HG537PU`:
  242. [source,console]
  243. ----
  244. GET my-index/_search
  245. {
  246. "query": {
  247. "match": {
  248. "model_number": "HG537PU"
  249. }
  250. }
  251. }
  252. ----
  253. //TEST[continued]
  254. The response includes indexed values for documents matching model number
  255. `HG537PU`:
  256. [source,console-result]
  257. ----
  258. {
  259. ...
  260. "hits" : {
  261. "total" : {
  262. "value" : 2,
  263. "relation" : "eq"
  264. },
  265. "max_score" : 1.0296195,
  266. "hits" : [
  267. {
  268. "_index" : "my-index",
  269. "_id" : "F1BeSXYBg_szTodcYCmk",
  270. "_score" : 1.0296195,
  271. "_source" : {
  272. "timestamp" : 1516383694000,
  273. "model_number" : "HG537PU",
  274. "measures" : {
  275. "voltage" : 4.2
  276. }
  277. }
  278. },
  279. {
  280. "_index" : "my-index",
  281. "_id" : "l02aSXYBkpNf6QRDO62Q",
  282. "_score" : 1.0296195,
  283. "_source" : {
  284. "timestamp" : 1516297294000,
  285. "model_number" : "HG537PU",
  286. "measures" : {
  287. "voltage" : 4.0
  288. }
  289. }
  290. }
  291. ]
  292. }
  293. }
  294. ----
  295. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
  296. // TESTRESPONSE[s/"_id" : "F1BeSXYBg_szTodcYCmk"/"_id": $body.hits.hits.0._id/]
  297. // TESTRESPONSE[s/"_id" : "l02aSXYBkpNf6QRDO62Q"/"_id": $body.hits.hits.1._id/]
  298. The following request defines a runtime field where the script evaluates the
  299. `model_number` field where the value is `HG537PU`. For each match, the script
  300. multiplies the value for the `voltage` field by `1.7`.
  301. Using the <<search-fields,`fields`>> parameter on the `_search` API, you can
  302. retrieve the value that the script calculates for the `measures.voltage` field
  303. for documents matching the search request:
  304. [source,console]
  305. ----
  306. POST my-index/_search
  307. {
  308. "runtime_mappings": {
  309. "measures.voltage": {
  310. "type": "double",
  311. "script": {
  312. "source":
  313. """if (doc['model_number.keyword'].value.equals('HG537PU'))
  314. {emit(1.7 * params._source['measures']['voltage']);}
  315. else{emit(params._source['measures']['voltage']);}"""
  316. }
  317. }
  318. },
  319. "query": {
  320. "match": {
  321. "model_number": "HG537PU"
  322. }
  323. },
  324. "fields": ["measures.voltage"]
  325. }
  326. ----
  327. //TEST[continued]
  328. Looking at the response, the calculated values for `measures.voltage` on each
  329. result are `7.14` and `6.8`. That's more like it! The runtime field calculated
  330. this value as part of the search request without modifying the mapped value,
  331. which still returns in the response:
  332. [source,console-result]
  333. ----
  334. {
  335. ...
  336. "hits" : {
  337. "total" : {
  338. "value" : 2,
  339. "relation" : "eq"
  340. },
  341. "max_score" : 1.0296195,
  342. "hits" : [
  343. {
  344. "_index" : "my-index",
  345. "_id" : "F1BeSXYBg_szTodcYCmk",
  346. "_score" : 1.0296195,
  347. "_source" : {
  348. "timestamp" : 1516383694000,
  349. "model_number" : "HG537PU",
  350. "measures" : {
  351. "voltage" : 4.2
  352. }
  353. },
  354. "fields" : {
  355. "measures.voltage" : [
  356. 7.14
  357. ]
  358. }
  359. },
  360. {
  361. "_index" : "my-index",
  362. "_id" : "l02aSXYBkpNf6QRDO62Q",
  363. "_score" : 1.0296195,
  364. "_source" : {
  365. "timestamp" : 1516297294000,
  366. "model_number" : "HG537PU",
  367. "measures" : {
  368. "voltage" : 4.0
  369. }
  370. },
  371. "fields" : {
  372. "measures.voltage" : [
  373. 6.8
  374. ]
  375. }
  376. }
  377. ]
  378. }
  379. }
  380. ----
  381. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
  382. // TESTRESPONSE[s/"_id" : "F1BeSXYBg_szTodcYCmk"/"_id": $body.hits.hits.0._id/]
  383. // TESTRESPONSE[s/"_id" : "l02aSXYBkpNf6QRDO62Q"/"_id": $body.hits.hits.1._id/]
  384. [[runtime-retrieving-fields]]
  385. === Retrieve a runtime field
  386. beta::[]
  387. Use the <<search-fields,`fields`>> parameter on the `_search` API to retrieve
  388. the values of runtime fields. Runtime fields won't display in `_source`, but
  389. the `fields` API works for all fields, even those that were not sent as part of
  390. the original `_source`.
  391. [discrete]
  392. [[runtime-define-field-dayofweek]]
  393. ==== Define a runtime field to calculate the day of week
  394. For example, the following request adds a runtime field called `day_of_week`.
  395. The runtime field includes a script that calculates the day of the week based
  396. on the value of the `@timestamp` field. We'll include `"dynamic":"runtime"` in
  397. the request so that new fields are added to the mapping as runtime fields.
  398. [source,console]
  399. ----
  400. PUT my-index/
  401. {
  402. "mappings": {
  403. "dynamic": "runtime",
  404. "runtime": {
  405. "day_of_week": {
  406. "type": "keyword",
  407. "script": {
  408. "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
  409. }
  410. }
  411. },
  412. "properties": {
  413. "timestamp": {"type": "date"}
  414. }
  415. }
  416. }
  417. ----
  418. [discrete]
  419. [[runtime-ingest-data]]
  420. ==== Ingest some data
  421. Let's ingest some sample data, which will result in two indexed fields:
  422. `@timestamp` and `message`.
  423. [source,console]
  424. ----
  425. POST /my-index/_bulk?refresh
  426. { "index": {}}
  427. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  428. { "index": {}}
  429. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  430. { "index": {}}
  431. { "@timestamp": "2020-04-30T14:30:17-05:00", "message" : "40.135.0.0 - - [2020-04-30T14:30:17-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  432. { "index": {}}
  433. { "@timestamp": "2020-04-30T14:30:53-05:00", "message" : "232.0.0.0 - - [2020-04-30T14:30:53-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  434. { "index": {}}
  435. { "@timestamp": "2020-04-30T14:31:12-05:00", "message" : "26.1.0.0 - - [2020-04-30T14:31:12-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  436. { "index": {}}
  437. { "@timestamp": "2020-04-30T14:31:19-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:19-05:00] \"GET /french/splash_inet.html HTTP/1.0\" 200 3781"}
  438. { "index": {}}
  439. { "@timestamp": "2020-04-30T14:31:27-05:00", "message" : "252.0.0.0 - - [2020-04-30T14:31:27-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  440. { "index": {}}
  441. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_brdl.gif HTTP/1.0\" 304 0"}
  442. { "index": {}}
  443. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_arw.gif HTTP/1.0\" 304 0"}
  444. { "index": {}}
  445. { "@timestamp": "2020-04-30T14:31:32-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:32-05:00] \"GET /images/nav_bg_top.gif HTTP/1.0\" 200 929"}
  446. { "index": {}}
  447. { "@timestamp": "2020-04-30T14:31:43-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:43-05:00] \"GET /french/images/nav_venue_off.gif HTTP/1.0\" 304 0"}
  448. ----
  449. //TEST[continued]
  450. [discrete]
  451. [[runtime-search-dayofweek]]
  452. ==== Search for the calculated day of week
  453. The following request uses the search API to retrieve the `day_of_week` field
  454. that the original request defined as a runtime field in the mapping. The value
  455. for this field is calculated dynamically at query time without reindexing
  456. documents or indexing the `day_of_week` field. This flexibility allows you to
  457. modify the mapping without changing any field values.
  458. [source,console]
  459. ----
  460. GET my-index/_search
  461. {
  462. "fields": [
  463. "@timestamp",
  464. "day_of_week"
  465. ],
  466. "_source": false
  467. }
  468. ----
  469. // TEST[continued]
  470. The previous request returns the `day_of_week` field for all matching documents.
  471. We can define another runtime field called `client_ip` that also operates on
  472. the `message` field and will further refine the query:
  473. [source,console]
  474. ----
  475. PUT /my-index/_mapping
  476. {
  477. "runtime": {
  478. "client_ip": {
  479. "type": "ip",
  480. "script" : {
  481. "source" : "String m = doc[\"message\"].value; int end = m.indexOf(\" \"); emit(m.substring(0, end));"
  482. }
  483. }
  484. }
  485. }
  486. ----
  487. //TEST[continued]
  488. Run another query, but search for a specific IP address using the `client_ip`
  489. runtime field:
  490. [source,console]
  491. ----
  492. GET my-index/_search
  493. {
  494. "size": 1,
  495. "query": {
  496. "match": {
  497. "client_ip": "211.11.9.0"
  498. }
  499. },
  500. "fields" : ["*"]
  501. }
  502. ----
  503. //TEST[continued]
  504. This time, the response includes only two hits. The value for `day_of_week`
  505. (`Sunday`) was calculated at query time using the runtime script defined in the
  506. mapping, and the result includes only documents matching the `211.11.9.0` IP
  507. address.
  508. [source,console-result]
  509. ----
  510. {
  511. ...
  512. "hits" : {
  513. "total" : {
  514. "value" : 2,
  515. "relation" : "eq"
  516. },
  517. "max_score" : 1.0,
  518. "hits" : [
  519. {
  520. "_index" : "my-index",
  521. "_id" : "oWs5KXYB-XyJbifr9mrz",
  522. "_score" : 1.0,
  523. "_source" : {
  524. "@timestamp" : "2020-06-21T15:00:01-05:00",
  525. "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  526. },
  527. "fields" : {
  528. "@timestamp" : [
  529. "2020-06-21T20:00:01.000Z"
  530. ],
  531. "client_ip" : [
  532. "211.11.9.0"
  533. ],
  534. "message" : [
  535. "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  536. ],
  537. "day_of_week" : [
  538. "Sunday"
  539. ]
  540. }
  541. }
  542. ]
  543. }
  544. }
  545. ----
  546. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
  547. // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]
  548. // TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/]
  549. [[runtime-examples]]
  550. === Explore your data with runtime fields
  551. beta::[]
  552. Consider a large set of log data that you want to extract fields from.
  553. Indexing the data is time consuming and uses a lot of disk space, and you just
  554. want to explore the data structure without committing to a schema up front.
  555. You know that your log data contains specific fields that you want to extract.
  556. In this case, we want to focus on the `@timestamp` and `message` fields. By
  557. using runtime fields, you can define scripts to calculate values at search
  558. time for these fields.
  559. [[runtime-examples-define-fields]]
  560. ==== Define indexed fields as a starting point
  561. You can start with a simple example by adding the `@timestamp` and `message`
  562. fields to the `my-index` mapping as indexed fields. To remain flexible, use
  563. `wildcard` as the field type for `message`:
  564. [source,console]
  565. ----
  566. PUT /my-index/
  567. {
  568. "mappings": {
  569. "properties": {
  570. "@timestamp": {
  571. "format": "strict_date_optional_time||epoch_second",
  572. "type": "date"
  573. },
  574. "message": {
  575. "type": "wildcard"
  576. }
  577. }
  578. }
  579. }
  580. ----
  581. [[runtime-examples-ingest-data]]
  582. ==== Ingest some data
  583. After mapping the fields you want to retrieve, index a few records from
  584. your log data into {es}. The following request uses the <<docs-bulk,bulk API>>
  585. to index raw log data into `my-index`. Instead of indexing all of your log
  586. data, you can use a small sample to experiment with runtime fields.
  587. [source,console]
  588. ----
  589. POST /my-index/_bulk?refresh
  590. { "index": {}}
  591. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  592. { "index": {}}
  593. { "@timestamp": "2020-06-21T15:00:01-05:00", "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"}
  594. { "index": {}}
  595. { "@timestamp": "2020-04-30T14:30:17-05:00", "message" : "40.135.0.0 - - [2020-04-30T14:30:17-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  596. { "index": {}}
  597. { "@timestamp": "2020-04-30T14:30:53-05:00", "message" : "232.0.0.0 - - [2020-04-30T14:30:53-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  598. { "index": {}}
  599. { "@timestamp": "2020-04-30T14:31:12-05:00", "message" : "26.1.0.0 - - [2020-04-30T14:31:12-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  600. { "index": {}}
  601. { "@timestamp": "2020-04-30T14:31:19-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:19-05:00] \"GET /french/splash_inet.html HTTP/1.0\" 200 3781"}
  602. { "index": {}}
  603. { "@timestamp": "2020-04-30T14:31:27-05:00", "message" : "252.0.0.0 - - [2020-04-30T14:31:27-05:00] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736"}
  604. { "index": {}}
  605. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_brdl.gif HTTP/1.0\" 304 0"}
  606. { "index": {}}
  607. { "@timestamp": "2020-04-30T14:31:29-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:29-05:00] \"GET /images/hm_arw.gif HTTP/1.0\" 304 0"}
  608. { "index": {}}
  609. { "@timestamp": "2020-04-30T14:31:32-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:32-05:00] \"GET /images/nav_bg_top.gif HTTP/1.0\" 200 929"}
  610. { "index": {}}
  611. { "@timestamp": "2020-04-30T14:31:43-05:00", "message" : "247.37.0.0 - - [2020-04-30T14:31:43-05:00] \"GET /french/images/nav_venue_off.gif HTTP/1.0\" 304 0"}
  612. ----
  613. // TEST[continued]
  614. At this point, you can view how {es} stores your raw data.
  615. [source,console]
  616. ----
  617. GET /my-index
  618. ----
  619. // TEST[continued]
  620. The mapping contains two fields: `@timestamp` and `message`.
  621. [source,console-result]
  622. ----
  623. {
  624. "my-index" : {
  625. "aliases" : { },
  626. "mappings" : {
  627. "properties" : {
  628. "@timestamp" : {
  629. "type" : "date",
  630. "format" : "strict_date_optional_time||epoch_second"
  631. },
  632. "message" : {
  633. "type" : "wildcard"
  634. }
  635. }
  636. },
  637. ...
  638. }
  639. }
  640. ----
  641. // TESTRESPONSE[s/\.\.\./"settings": $body.my-index.settings/]
  642. [[runtime-examples-runtime-field]]
  643. ==== Define a runtime field to search by IP address
  644. If you want to retrieve results that include `clientip`, you can add that field
  645. as a runtime field in the mapping. The runtime script operates on the `clientip`
  646. field at runtime to calculate values for that field.
  647. [source,console]
  648. ----
  649. PUT /my-index/_mapping
  650. {
  651. "runtime": {
  652. "clientip": {
  653. "type": "ip",
  654. "script" : {
  655. "source" : "String m = doc[\"message\"].value; int end = m.indexOf(\" \"); emit(m.substring(0, end));"
  656. }
  657. }
  658. }
  659. }
  660. ----
  661. // TEST[continued]
  662. Using the `clientip` runtime field, you can define a simple query to run a
  663. search for a specific IP address and return all related fields.
  664. [source,console]
  665. ----
  666. GET my-index/_search
  667. {
  668. "size": 1,
  669. "query": {
  670. "match": {
  671. "clientip": "211.11.9.0"
  672. }
  673. },
  674. "fields" : ["*"]
  675. }
  676. ----
  677. // TEST[continued]
  678. The API returns the following result. Without building your data structure in
  679. advance, you can search and explore your data in meaningful ways to experiment
  680. and determine which fields to index.
  681. [source,console-result]
  682. ----
  683. {
  684. ...
  685. "hits" : {
  686. "total" : {
  687. "value" : 2,
  688. "relation" : "eq"
  689. },
  690. "max_score" : 1.0,
  691. "hits" : [
  692. {
  693. "_index" : "my-index",
  694. "_id" : "oWs5KXYB-XyJbifr9mrz",
  695. "_score" : 1.0,
  696. "_source" : {
  697. "@timestamp" : "2020-06-21T15:00:01-05:00",
  698. "message" : "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  699. },
  700. "fields" : {
  701. "@timestamp" : [
  702. "2020-06-21T20:00:01.000Z"
  703. ],
  704. "clientip" : [
  705. "211.11.9.0"
  706. ],
  707. "message" : [
  708. "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0"
  709. ]
  710. }
  711. }
  712. ]
  713. }
  714. }
  715. ----
  716. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/]
  717. // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/]