search.asciidoc 25 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[eql-search]]
  4. == Run an EQL search
  5. experimental::[]
  6. To start using EQL in {es}, first ensure your event data meets
  7. <<eql-requirements,EQL requirements>>. You can then use the <<eql-search-api,EQL
  8. search API>> to search event data stored in one or more {es} data streams or
  9. indices.
  10. .*Example*
  11. [%collapsible]
  12. ====
  13. To get started, ingest or add the data to an {es} data stream or index.
  14. The following <<docs-bulk,bulk API>> request adds some example log data to the
  15. `sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema
  16. (ECS)].
  17. [source,console]
  18. ----
  19. PUT /sec_logs/_bulk?refresh
  20. {"index":{"_index" : "sec_logs", "_id" : "1"}}
  21. { "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "edwCRnyD","sequence": 1 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
  22. {"index":{"_index" : "sec_logs", "_id" : "2"}}
  23. { "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "dGCHwoeS", "sequence": 2 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
  24. {"index":{"_index" : "sec_logs", "_id" : "3"}}
  25. { "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "cMyt5SZ2", "sequence": 3 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
  26. {"index":{"_index" : "sec_logs", "_id" : "4"}}
  27. { "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file", "id": "bYA7gPay", "sequence": 4 }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
  28. {"index":{"_index" : "sec_logs", "_id" : "5"}}
  29. { "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "aR3NWVOs", "sequence": 5 }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
  30. {"index":{"_index" : "sec_logs", "_id" : "6"}}
  31. { "@timestamp": "2020-12-07T11:07:10.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process", "id": "GTSmSqgz0U", "sequence": 6, "type": "termination" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
  32. ----
  33. // TESTSETUP
  34. [TIP]
  35. =====
  36. You also can set up {beats-ref}/getting-started.html[{beats}], such as
  37. {auditbeat-ref}/auditbeat-installation-configuration.html[{auditbeat}] or
  38. {winlogbeat-ref}/winlogbeat-installation-configuration.html[{winlogbeat}], to automatically
  39. send and index your event data in {es}. See
  40. {beats-ref}/getting-started.html[Getting started with {beats}].
  41. =====
  42. You can now use the EQL search API to search this index using an EQL query.
  43. The following request searches the `sec_logs` index using the EQL query
  44. specified in the `query` parameter. The EQL query matches events with an
  45. `event.category` of `process` that have a `process.name` of `cmd.exe`.
  46. [source,console]
  47. ----
  48. GET /sec_logs/_eql/search
  49. {
  50. "query": """
  51. process where process.name == "cmd.exe"
  52. """
  53. }
  54. ----
  55. // TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/]
  56. Because the `sec_log` index follows the ECS, you don't need to specify the
  57. required <<eql-required-fields,event category or timestamp>> fields. The request
  58. uses the `event.category` and `@timestamp` fields by default.
  59. The API returns the following response containing the matching events. Events
  60. in the response are sorted by timestamp, converted to milliseconds since the
  61. https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
  62. [source,console-result]
  63. ----
  64. {
  65. "is_partial": false,
  66. "is_running": false,
  67. "took": 60,
  68. "timed_out": false,
  69. "hits": {
  70. "total": {
  71. "value": 2,
  72. "relation": "eq"
  73. },
  74. "events": [
  75. {
  76. "_index": "sec_logs",
  77. "_id": "1",
  78. "_score": null,
  79. "_source": {
  80. "@timestamp": "2020-12-06T11:04:05.000Z",
  81. "agent": {
  82. "id": "8a4f500d"
  83. },
  84. "event": {
  85. "category": "process",
  86. "id": "edwCRnyD",
  87. "sequence": 1
  88. },
  89. "process": {
  90. "name": "cmd.exe",
  91. "path": "C:\\Windows\\System32\\cmd.exe"
  92. }
  93. }
  94. },
  95. {
  96. "_index": "sec_logs",
  97. "_id": "3",
  98. "_score": null,
  99. "_source": {
  100. "@timestamp": "2020-12-07T11:06:07.000Z",
  101. "agent": {
  102. "id": "8a4f500d"
  103. },
  104. "event": {
  105. "category": "process",
  106. "id": "cMyt5SZ2",
  107. "sequence": 3
  108. },
  109. "process": {
  110. "name": "cmd.exe",
  111. "path": "C:\\Windows\\System32\\cmd.exe"
  112. }
  113. }
  114. }
  115. ]
  116. }
  117. }
  118. ----
  119. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  120. ====
  121. [discrete]
  122. [[eql-search-sequence]]
  123. === Search for a sequence of events
  124. Many query languages allow you to match single events. However, EQL's
  125. <<eql-sequences,sequence syntax>> lets you match an ordered series of events.
  126. .*Example*
  127. [%collapsible]
  128. ====
  129. The following EQL search request matches a sequence that:
  130. . Starts with an event with:
  131. +
  132. --
  133. * An `event.category` of `file`
  134. * A `file.name` of `cmd.exe`
  135. --
  136. . Followed by an event with:
  137. +
  138. --
  139. * An `event.category` of `process`
  140. * A `process.name` that contains the substring `regsvr32`
  141. --
  142. [source,console]
  143. ----
  144. GET /sec_logs/_eql/search
  145. {
  146. "query": """
  147. sequence
  148. [ file where file.name == "cmd.exe" ]
  149. [ process where stringContains(process.name, "regsvr32") ]
  150. """
  151. }
  152. ----
  153. // TEST[s/search/search\?filter_path\=\-\*\.sequences\.events\.\*fields/]
  154. The API returns the following response. Matching events in
  155. the `hits.sequences.events` property are sorted by
  156. <<eql-search-api-timestamp-field,timestamp>>, converted to milliseconds since
  157. the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.
  158. [source,console-result]
  159. ----
  160. {
  161. "is_partial": false,
  162. "is_running": false,
  163. "took": 60,
  164. "timed_out": false,
  165. "hits": {
  166. "total": {
  167. "value": 1,
  168. "relation": "eq"
  169. },
  170. "sequences": [
  171. {
  172. "events": [
  173. {
  174. "_index": "sec_logs",
  175. "_id": "4",
  176. "_score": null,
  177. "_source": {
  178. "@timestamp": "2020-12-07T11:07:08.000Z",
  179. "agent": {
  180. "id": "8a4f500d"
  181. },
  182. "event": {
  183. "category": "file",
  184. "id": "bYA7gPay",
  185. "sequence": 4
  186. },
  187. "file": {
  188. "accessed": "2020-12-07T11:07:08.000Z",
  189. "name": "cmd.exe",
  190. "path": "C:\\Windows\\System32\\cmd.exe",
  191. "type": "file",
  192. "size": 16384
  193. },
  194. "process": {
  195. "name": "cmd.exe",
  196. "path": "C:\\Windows\\System32\\cmd.exe"
  197. }
  198. }
  199. },
  200. {
  201. "_index": "sec_logs",
  202. "_id": "5",
  203. "_score": null,
  204. "_source": {
  205. "@timestamp": "2020-12-07T11:07:09.000Z",
  206. "agent": {
  207. "id": "8a4f500d"
  208. },
  209. "event": {
  210. "category": "process",
  211. "id": "aR3NWVOs",
  212. "sequence": 5
  213. },
  214. "process": {
  215. "name": "regsvr32.exe",
  216. "path": "C:\\Windows\\System32\\regsvr32.exe"
  217. }
  218. }
  219. }
  220. ]
  221. }
  222. ]
  223. }
  224. }
  225. ----
  226. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  227. // TESTRESPONSE[skip: response format updated]
  228. You can use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to
  229. constrain a sequence to a specified timespan.
  230. The following EQL search request adds `with maxspan=1h` to the previous query.
  231. This ensures all events in a matching sequence occur within one hour (`1h`) of
  232. the first event's timestamp.
  233. [source,console]
  234. ----
  235. GET /sec_logs/_eql/search
  236. {
  237. "query": """
  238. sequence with maxspan=1h
  239. [ file where file.name == "cmd.exe" ]
  240. [ process where stringContains(process.name, "regsvr32") ]
  241. """
  242. }
  243. ----
  244. // TEST[s/search/search\?filter_path\=\-\*\.sequences\.events\.\*fields/]
  245. You can further constrain matching event sequences using the
  246. <<eql-by-keyword,`by` keyword>>.
  247. The following EQL search request adds `by agent.id` to each event item. This
  248. ensures events matching the sequence share the same `agent.id` field value.
  249. [source,console]
  250. ----
  251. GET /sec_logs/_eql/search
  252. {
  253. "query": """
  254. sequence with maxspan=1h
  255. [ file where file.name == "cmd.exe" ] by agent.id
  256. [ process where stringContains(process.name, "regsvr32") ] by agent.id
  257. """
  258. }
  259. ----
  260. Because the `agent.id` field is shared across all events in the sequence, it
  261. can be included using `sequence by`. The following query is equivalent to the
  262. prior one.
  263. [source,console]
  264. ----
  265. GET /sec_logs/_eql/search
  266. {
  267. "query": """
  268. sequence by agent.id with maxspan=1h
  269. [ file where file.name == "cmd.exe" ]
  270. [ process where stringContains(process.name, "regsvr32") ]
  271. """
  272. }
  273. ----
  274. // TEST[s/search/search\?filter_path\=\-\*\.sequences\.\*events\.\*fields/]
  275. The API returns the following response. The `hits.sequences.join_keys` property
  276. contains the shared `agent.id` value for each matching event.
  277. [source,console-result]
  278. ----
  279. {
  280. "is_partial": false,
  281. "is_running": false,
  282. "took": 60,
  283. "timed_out": false,
  284. "hits": {
  285. "total": {
  286. "value": 1,
  287. "relation": "eq"
  288. },
  289. "sequences": [
  290. {
  291. "join_keys": [
  292. "8a4f500d"
  293. ],
  294. "events": [
  295. {
  296. "_index": "sec_logs",
  297. "_id": "4",
  298. "_score": null,
  299. "_source": {
  300. "@timestamp": "2020-12-07T11:07:08.000Z",
  301. "agent": {
  302. "id": "8a4f500d"
  303. },
  304. "event": {
  305. "category": "file",
  306. "id": "bYA7gPay",
  307. "sequence": 4
  308. },
  309. "file": {
  310. "accessed": "2020-12-07T11:07:08.000Z",
  311. "name": "cmd.exe",
  312. "path": "C:\\Windows\\System32\\cmd.exe",
  313. "type": "file",
  314. "size": 16384
  315. },
  316. "process": {
  317. "name": "cmd.exe",
  318. "path": "C:\\Windows\\System32\\cmd.exe"
  319. }
  320. }
  321. },
  322. {
  323. "_index": "sec_logs",
  324. "_id": "5",
  325. "_score": null,
  326. "_source": {
  327. "@timestamp": "2020-12-07T11:07:09.000Z",
  328. "agent": {
  329. "id": "8a4f500d"
  330. },
  331. "event": {
  332. "category": "process",
  333. "id": "aR3NWVOs",
  334. "sequence": 5
  335. },
  336. "process": {
  337. "name": "regsvr32.exe",
  338. "path": "C:\\Windows\\System32\\regsvr32.exe"
  339. }
  340. }
  341. }
  342. ]
  343. }
  344. ]
  345. }
  346. }
  347. ----
  348. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  349. // TESTRESPONSE[skip: response format updated]
  350. You can use the <<eql-until-keyword,`until` keyword>> to specify an expiration
  351. event for sequences. Matching sequences must end before this event.
  352. The following request adds
  353. `until [ process where event.type == "termination" ]` to the previous EQL query.
  354. This ensures matching sequences end before a process termination event.
  355. [source,console]
  356. ----
  357. GET /sec_logs/_eql/search
  358. {
  359. "query": """
  360. sequence by agent.id with maxspan=1h
  361. [ file where file.name == "cmd.exe" ]
  362. [ process where stringContains(process.name, "regsvr32") ]
  363. until [ process where event.type == "termination" ]
  364. """
  365. }
  366. ----
  367. // TEST[s/search/search\?filter_path\=\-\*\.sequences\.\*events\.\*fields/]
  368. ====
  369. [discrete]
  370. [[eql-search-specify-event-category-field]]
  371. === Specify an event category field
  372. The EQL search API uses `event.category` as the required
  373. <<eql-required-fields,event category field>> by default. You can use the
  374. `event_category_field` parameter to specify another event category field.
  375. .*Example*
  376. [%collapsible]
  377. ====
  378. The following request specifies `file.type` as the event category
  379. field.
  380. [source,console]
  381. ----
  382. GET /sec_logs/_eql/search
  383. {
  384. "event_category_field": "file.type",
  385. "query": """
  386. file where agent.id == "8a4f500d"
  387. """
  388. }
  389. ----
  390. ====
  391. [discrete]
  392. [[eql-search-specify-timestamp-field]]
  393. === Specify a timestamp field
  394. The EQL search API uses `@timestamp` as the required <<eql-required-fields,event
  395. timestamp field>> by default. You can use the `timestamp_field` parameter to
  396. specify another timestamp field.
  397. .*Example*
  398. [%collapsible]
  399. ====
  400. The following request specifies `file.accessed` as the event
  401. timestamp field.
  402. [source,console]
  403. ----
  404. GET /sec_logs/_eql/search
  405. {
  406. "timestamp_field": "file.accessed",
  407. "query": """
  408. file where (file.size > 1 and file.type == "file")
  409. """
  410. }
  411. ----
  412. ====
  413. [discrete]
  414. [[eql-search-specify-a-sort-tiebreaker]]
  415. === Specify a sort tiebreaker
  416. By default, the EQL search API sorts matching events in the search response by
  417. timestamp. However, if two or more events share the same timestamp, a tiebreaker
  418. field is used to sort the events in ascending, lexicographic order.
  419. The EQL search API uses `event.sequence` as the default tiebreaker field. You
  420. can use the `tiebreaker_field` parameter to specify another field.
  421. .*Example*
  422. [%collapsible]
  423. ====
  424. The following request specifies `event.start` as the tiebreaker field.
  425. [source,console]
  426. ----
  427. GET /sec_logs/_eql/search
  428. {
  429. "tiebreaker_field": "event.id",
  430. "query": """
  431. process where process.name == "cmd.exe" and stringContains(process.path, "System32")
  432. """
  433. }
  434. ----
  435. // TEST[s/search/search\?filter_path\=\-\*\.events\.\*fields/]
  436. The API returns the following response.
  437. [source,console-result]
  438. ----
  439. {
  440. "is_partial": false,
  441. "is_running": false,
  442. "took": 34,
  443. "timed_out": false,
  444. "hits": {
  445. "total": {
  446. "value": 2,
  447. "relation": "eq"
  448. },
  449. "events": [
  450. {
  451. "_index": "sec_logs",
  452. "_id": "1",
  453. "_score": null,
  454. "_source": {
  455. "@timestamp": "2020-12-06T11:04:05.000Z",
  456. "agent": {
  457. "id": "8a4f500d"
  458. },
  459. "event": {
  460. "category": "process",
  461. "id": "edwCRnyD",
  462. "sequence": 1
  463. },
  464. "process": {
  465. "name": "cmd.exe",
  466. "path": "C:\\Windows\\System32\\cmd.exe"
  467. }
  468. }
  469. },
  470. {
  471. "_index": "sec_logs",
  472. "_id": "3",
  473. "_score": null,
  474. "_source": {
  475. "@timestamp": "2020-12-07T11:06:07.000Z",
  476. "agent": {
  477. "id": "8a4f500d"
  478. },
  479. "event": {
  480. "category": "process",
  481. "id": "cMyt5SZ2",
  482. "sequence": 3
  483. },
  484. "process": {
  485. "name": "cmd.exe",
  486. "path": "C:\\Windows\\System32\\cmd.exe"
  487. }
  488. }
  489. }
  490. ]
  491. }
  492. }
  493. ----
  494. // TESTRESPONSE[s/"took": 34/"took": $body.took/]
  495. ====
  496. [discrete]
  497. [[eql-search-filter-query-dsl]]
  498. === Filter using query DSL
  499. You can use the EQL search API's `filter` parameter to specify an additional
  500. query using <<query-dsl,query DSL>>. This query filters the documents on which
  501. the EQL query runs.
  502. .*Example*
  503. [%collapsible]
  504. ====
  505. The following request uses a `range` query to filter the `sec_logs`
  506. index down to only documents with a `file.size` value greater than `1` but less
  507. than `1000000` bytes. The EQL query in `query` parameter then runs on these
  508. filtered documents.
  509. [source,console]
  510. ----
  511. GET /sec_logs/_eql/search
  512. {
  513. "filter": {
  514. "range" : {
  515. "file.size" : {
  516. "gte" : 1,
  517. "lte" : 1000000
  518. }
  519. }
  520. },
  521. "query": """
  522. file where (file.type == "file" and file.name == "cmd.exe")
  523. """
  524. }
  525. ----
  526. ====
  527. [discrete]
  528. [[eql-search-async]]
  529. === Run an async EQL search
  530. EQL searches in {es} are designed to run on large volumes of data quickly,
  531. often returning results in milliseconds. Because of this, the EQL search API
  532. runs _synchronous_ searches by default. This means the search request waits for
  533. complete results before returning a response.
  534. However, complete results can take longer for searches across:
  535. * <<frozen-indices,Frozen indices>>
  536. * <<modules-cross-cluster-search,Multiple clusters>>
  537. * Many shards
  538. To avoid long waits, you can use the EQL search API's
  539. `wait_for_completion_timeout` parameter to run an _asynchronous_, or _async_,
  540. search.
  541. Set the `wait_for_completion_timeout` parameter to a duration you'd like to wait
  542. for complete search results. If the search request does not finish within this
  543. period, the search becomes an async search. The EQL search
  544. API returns a response that includes:
  545. * A search ID, which can be used to monitor the progress of the async search and
  546. retrieve complete results when it finishes.
  547. * An `is_partial` value of `true`, indicating the response does not contain
  548. complete search results.
  549. * An `is_running` value of `true`, indicating the search is async and ongoing.
  550. The async search continues to run in the background without blocking
  551. other requests.
  552. [%collapsible]
  553. .*Example*
  554. ====
  555. The following request searches the `frozen_sec_logs` index, which has been
  556. <<frozen-indices,frozen>> for storage and is rarely searched.
  557. Because searches on frozen indices are expected to take longer to complete, the
  558. request contains a `wait_for_completion_timeout` parameter value of `2s`
  559. (two seconds).
  560. If the request does not return complete results in two seconds, the search
  561. becomes an async search and a search ID is returned.
  562. [source,console]
  563. ----
  564. GET /frozen_sec_logs/_eql/search
  565. {
  566. "wait_for_completion_timeout": "2s",
  567. "query": """
  568. process where process.name == "cmd.exe"
  569. """
  570. }
  571. ----
  572. // TEST[s/frozen_sec_logs/sec_logs/]
  573. After two seconds, the request returns the following response. Note the
  574. `is_partial` and `is_running` properties are `true`, indicating an ongoing async
  575. search.
  576. [source,console-result]
  577. ----
  578. {
  579. "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  580. "is_partial": true,
  581. "is_running": true,
  582. "took": 2000,
  583. "timed_out": false,
  584. "hits": ...
  585. }
  586. ----
  587. // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
  588. // TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
  589. // TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
  590. // TESTRESPONSE[s/"took": 2000/"took": $body.took/]
  591. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  592. ====
  593. You can use the the returned search ID and the <<get-async-eql-search-api,get
  594. async EQL search API>> to check the progress of an ongoing async search.
  595. The get async EQL search API also accepts a `wait_for_completion_timeout` query
  596. parameter. Set the `wait_for_completion_timeout` parameter to a duration you'd
  597. like to wait for complete search results. If the request does not complete
  598. during this period, the response returns an `is_partial` value of `true` and no
  599. search results.
  600. [%collapsible]
  601. .*Example*
  602. ====
  603. The following get async EQL search API request checks the progress of the
  604. previous async EQL search. The request specifies a `wait_for_completion_timeout`
  605. query parameter value of `2s` (two seconds).
  606. [source,console]
  607. ----
  608. GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
  609. ----
  610. // TEST[skip: no access to search ID]
  611. The request returns the following response. Note the `is_partial` and
  612. `is_running` properties are `false`, indicating the async EQL search has
  613. finished and the search results in the `hits` property are complete.
  614. [source,console-result]
  615. ----
  616. {
  617. "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  618. "is_partial": false,
  619. "is_running": false,
  620. "took": 2000,
  621. "timed_out": false,
  622. "hits": ...
  623. }
  624. ----
  625. // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
  626. // TESTRESPONSE[s/"took": 2000/"took": $body.took/]
  627. // TESTRESPONSE[s/"_index": "frozen_sec_logs"/"_index": "sec_logs"/]
  628. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  629. ====
  630. [discrete]
  631. [[eql-search-store-async-eql-search]]
  632. === Change the search retention period
  633. By default, the EQL search API only stores async searches and their results for
  634. five days. After this period, any ongoing searches or saved results are deleted.
  635. You can use the EQL search API's `keep_alive` parameter to change the duration
  636. of this period.
  637. .*Example*
  638. [%collapsible]
  639. ====
  640. In the following EQL search API request, the `keep_alive` parameter is `2d` (two
  641. days). This means that if the search becomes async, its results
  642. are stored on the cluster for two days. After two days, the async
  643. search and its results are deleted, even if it's still ongoing.
  644. [source,console]
  645. ----
  646. GET /sec_logs/_eql/search
  647. {
  648. "keep_alive": "2d",
  649. "wait_for_completion_timeout": "2s",
  650. "query": """
  651. process where process.name == "cmd.exe"
  652. """
  653. }
  654. ----
  655. ====
  656. You can use the <<get-async-eql-search-api,get async EQL search API>>'s
  657. `keep_alive` query parameter to later change the retention period. The new
  658. retention period starts after the get async EQL search API request executes.
  659. .*Example*
  660. [%collapsible]
  661. ====
  662. The following get async EQL search API request sets the `keep_alive` query
  663. parameter to `5d` (five days). The async search and its results are deleted five
  664. days after the get async EQL search API request executes.
  665. [source,console]
  666. ----
  667. GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
  668. ----
  669. // TEST[skip: no access to search ID]
  670. ====
  671. You can use the <<delete-async-eql-search-api,delete async EQL search API>> to
  672. manually delete an async EQL search before the `keep_alive` period ends. If the
  673. search is still ongoing, this cancels the search request.
  674. .*Example*
  675. [%collapsible]
  676. ====
  677. The following delete async EQL search API request deletes an async EQL search
  678. and its results.
  679. [source,console]
  680. ----
  681. DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
  682. ----
  683. // TEST[skip: no access to search ID]
  684. ====
  685. [discrete]
  686. [[eql-search-store-sync-eql-search]]
  687. === Store synchronous EQL searches
  688. By default, the EQL search API only stores async searches that cannot be
  689. completed within the period set by the `wait_for_completion_timeout` parameter.
  690. To save the results of searches that complete during this period, set the
  691. `keep_on_completion` parameter to `true`.
  692. [%collapsible]
  693. .*Example*
  694. ====
  695. In the following EQL search API request, the `keep_on_completion` parameter is
  696. `true`. This means the search results are stored on the cluster, even if
  697. the search completes within the `2s` (two-second) period set by the
  698. `wait_for_completion_timeout` parameter.
  699. [source,console]
  700. ----
  701. GET /sec_logs/_eql/search
  702. {
  703. "keep_on_completion": true,
  704. "wait_for_completion_timeout": "2s",
  705. "query": """
  706. process where process.name == "cmd.exe"
  707. """
  708. }
  709. ----
  710. The API returns the following response. Note that a search ID is provided in the
  711. `id` property. The `is_partial` and `is_running` properties are `false`,
  712. indicating the EQL search was synchronous and returned complete search results.
  713. [source,console-result]
  714. ----
  715. {
  716. "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  717. "is_partial": false,
  718. "is_running": false,
  719. "took": 52,
  720. "timed_out": false,
  721. "hits": ...
  722. }
  723. ----
  724. // TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
  725. // TESTRESPONSE[s/"took": 52/"took": $body.took/]
  726. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  727. You can use the search ID and the <<get-async-eql-search-api,get async EQL
  728. search API>> to retrieve the same results later.
  729. [source,console]
  730. ----
  731. GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
  732. ----
  733. // TEST[skip: no access to search ID]
  734. ====
  735. Saved synchronous searches are still subject to the storage retention period set
  736. by the `keep_alive` parameter. After this period, the search and its saved
  737. results are deleted.
  738. You can also manually delete saved synchronous searches using the
  739. <<delete-async-eql-search-api,delete async EQL search API>>.
  740. [discrete]
  741. [[eql-search-case-sensitive]]
  742. === Run a case-sensitive EQL search
  743. By default, matching for EQL queries is case-insensitive. You can use the EQL
  744. search API's `case_sensitive` parameter to toggle case sensitivity on or off.
  745. .*Example*
  746. [%collapsible]
  747. ====
  748. The following search request contains a query that matches `process` events
  749. with a `process.path` containing `System32`.
  750. Because the `case_sensitive` parameter is `true`, this query only matches
  751. `process.path` values containing `System32` with the exact same capitalization.
  752. A `process.path` value containing `system32` or `SYSTEM32` would not match this
  753. query.
  754. [source,console]
  755. ----
  756. GET /sec_logs/_eql/search
  757. {
  758. "keep_on_completion": true,
  759. "case_sensitive": true,
  760. "query": """
  761. process where stringContains(process.path, "System32")
  762. """
  763. }
  764. ----
  765. ====