eql.asciidoc 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[eql]]
  4. = EQL search
  5. ++++
  6. <titleabbrev>EQL</titleabbrev>
  7. ++++
  8. Event Query Language (EQL) is a query language for event-based time series
  9. data, such as logs, metrics, and traces.
  10. [discrete]
  11. [[eql-advantages]]
  12. === Advantages of EQL
  13. * *EQL lets you express relationships between events.* +
  14. Many query languages allow you to match single events. EQL lets you match a
  15. sequence of events across different event categories and time spans.
  16. * *EQL has a low learning curve.* +
  17. <<eql-syntax,EQL syntax>> looks like other common query languages, such as SQL.
  18. EQL lets you write and read queries intuitively, which makes for quick,
  19. iterative searching.
  20. * *EQL is designed for security use cases.* +
  21. While you can use it for any event-based data, we created EQL for threat
  22. hunting. EQL not only supports indicator of compromise (IOC) searches but can
  23. describe activity that goes beyond IOCs.
  24. [discrete]
  25. [[eql-required-fields]]
  26. === Required fields
  27. To run an EQL search, the searched data stream or index must contain a
  28. _timestamp_ and _event category_ field. By default, EQL uses the `@timestamp`
  29. and `event.category` fields from the {ecs-ref}[Elastic Common Schema
  30. (ECS)]. To use a different timestamp or event category field, see
  31. <<specify-a-timestamp-or-event-category-field>>.
  32. TIP: While no schema is required to use EQL, we recommend using the
  33. {ecs-ref}[ECS]. EQL searches are designed to work with core ECS fields by
  34. default.
  35. [discrete]
  36. [[run-an-eql-search]]
  37. === Run an EQL search
  38. Use the <<eql-search-api,EQL search API>> to run a <<eql-basic-syntax,basic EQL
  39. query>>.
  40. ////
  41. [source,console]
  42. ----
  43. DELETE /_data_stream/*
  44. DELETE /_index_template/*
  45. ----
  46. // TEARDOWN
  47. ////
  48. [source,console]
  49. ----
  50. GET /my-data-stream/_eql/search
  51. {
  52. "query": """
  53. process where process.name == "regsvr32.exe"
  54. """
  55. }
  56. ----
  57. // TEST[setup:sec_logs]
  58. By default, basic EQL queries return the 10 most recent matching events in the
  59. `hits.events` property. These hits are sorted by timestamp, converted to
  60. milliseconds since the {wikipedia}/Unix_time[Unix epoch], in ascending order.
  61. [source,console-result]
  62. ----
  63. {
  64. "is_partial": false,
  65. "is_running": false,
  66. "took": 60,
  67. "timed_out": false,
  68. "hits": {
  69. "total": {
  70. "value": 2,
  71. "relation": "eq"
  72. },
  73. "events": [
  74. {
  75. "_index": ".ds-my-data-stream-2099.12.07-000001",
  76. "_id": "OQmfCaduce8zoHT93o4H",
  77. "_source": {
  78. "@timestamp": "2099-12-07T11:07:09.000Z",
  79. "event": {
  80. "category": "process",
  81. "id": "aR3NWVOs",
  82. "sequence": 4
  83. },
  84. "process": {
  85. "pid": 2012,
  86. "name": "regsvr32.exe",
  87. "command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
  88. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  89. }
  90. }
  91. },
  92. {
  93. "_index": ".ds-my-data-stream-2099.12.07-000001",
  94. "_id": "xLkCaj4EujzdNSxfYLbO",
  95. "_source": {
  96. "@timestamp": "2099-12-07T11:07:10.000Z",
  97. "event": {
  98. "category": "process",
  99. "id": "GTSmSqgz0U",
  100. "sequence": 6,
  101. "type": "termination"
  102. },
  103. "process": {
  104. "pid": 2012,
  105. "name": "regsvr32.exe",
  106. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  107. }
  108. }
  109. }
  110. ]
  111. }
  112. }
  113. ----
  114. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  115. // TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
  116. // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
  117. // TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]
  118. Use the `size` parameter to get a smaller or larger set of hits:
  119. [source,console]
  120. ----
  121. GET /my-data-stream/_eql/search
  122. {
  123. "query": """
  124. process where process.name == "regsvr32.exe"
  125. """,
  126. "size": 50
  127. }
  128. ----
  129. // TEST[setup:sec_logs]
  130. [discrete]
  131. [[eql-search-sequence]]
  132. === Search for a sequence of events
  133. Use EQL's <<eql-sequences,sequence syntax>> to search for a series of
  134. ordered events. List the event items in ascending chronological order,
  135. with the most recent event listed last:
  136. [source,console]
  137. ----
  138. GET /my-data-stream/_eql/search
  139. {
  140. "query": """
  141. sequence
  142. [ process where process.name == "regsvr32.exe" ]
  143. [ file where stringContains(file.name, "scrobj.dll") ]
  144. """
  145. }
  146. ----
  147. // TEST[setup:sec_logs]
  148. The response's `hits.sequences` property contains the 10 most recent matching
  149. sequences.
  150. [source,console-result]
  151. ----
  152. {
  153. "is_partial": false,
  154. "is_running": false,
  155. "took": 60,
  156. "timed_out": false,
  157. "hits": {
  158. "total": {
  159. "value": 1,
  160. "relation": "eq"
  161. },
  162. "sequences": [
  163. {
  164. "events": [
  165. {
  166. "_index": ".ds-my-data-stream-2099.12.07-000001",
  167. "_id": "OQmfCaduce8zoHT93o4H",
  168. "_source": {
  169. "@timestamp": "2099-12-07T11:07:09.000Z",
  170. "event": {
  171. "category": "process",
  172. "id": "aR3NWVOs",
  173. "sequence": 4
  174. },
  175. "process": {
  176. "pid": 2012,
  177. "name": "regsvr32.exe",
  178. "command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
  179. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  180. }
  181. }
  182. },
  183. {
  184. "_index": ".ds-my-data-stream-2099.12.07-000001",
  185. "_id": "yDwnGIJouOYGBzP0ZE9n",
  186. "_source": {
  187. "@timestamp": "2099-12-07T11:07:10.000Z",
  188. "event": {
  189. "category": "file",
  190. "id": "tZ1NWVOs",
  191. "sequence": 5
  192. },
  193. "process": {
  194. "pid": 2012,
  195. "name": "regsvr32.exe",
  196. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  197. },
  198. "file": {
  199. "path": "C:\\Windows\\System32\\scrobj.dll",
  200. "name": "scrobj.dll"
  201. }
  202. }
  203. }
  204. ]
  205. }
  206. ]
  207. }
  208. }
  209. ----
  210. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  211. // TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/]
  212. // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
  213. // TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
  214. Use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to constrain
  215. matching sequences to a timespan:
  216. [source,console]
  217. ----
  218. GET /my-data-stream/_eql/search
  219. {
  220. "query": """
  221. sequence with maxspan=1h
  222. [ process where process.name == "regsvr32.exe" ]
  223. [ file where stringContains(file.name, "scrobj.dll") ]
  224. """
  225. }
  226. ----
  227. // TEST[setup:sec_logs]
  228. Use the <<eql-by-keyword,`by` keyword>> to match events that share the
  229. same field values:
  230. [source,console]
  231. ----
  232. GET /my-data-stream/_eql/search
  233. {
  234. "query": """
  235. sequence with maxspan=1h
  236. [ process where process.name == "regsvr32.exe" ] by process.pid
  237. [ file where stringContains(file.name, "scrobj.dll") ] by process.pid
  238. """
  239. }
  240. ----
  241. // TEST[setup:sec_logs]
  242. If a field value should be shared across all events, use the `sequence by`
  243. keyword. The following query is equivalent to the previous one.
  244. [source,console]
  245. ----
  246. GET /my-data-stream/_eql/search
  247. {
  248. "query": """
  249. sequence by process.pid with maxspan=1h
  250. [ process where process.name == "regsvr32.exe" ]
  251. [ file where stringContains(file.name, "scrobj.dll") ]
  252. """
  253. }
  254. ----
  255. // TEST[setup:sec_logs]
  256. The `hits.sequences.join_keys` property contains the shared field values.
  257. [source,console-result]
  258. ----
  259. {
  260. "is_partial": false,
  261. "is_running": false,
  262. "took": 60,
  263. "timed_out": false,
  264. "hits": {
  265. "total": {
  266. "value": 1,
  267. "relation": "eq"
  268. },
  269. "sequences": [
  270. {
  271. "join_keys": [
  272. 2012
  273. ],
  274. "events": [
  275. {
  276. "_index": ".ds-my-data-stream-2099.12.07-000001",
  277. "_id": "OQmfCaduce8zoHT93o4H",
  278. "_source": {
  279. "@timestamp": "2099-12-07T11:07:09.000Z",
  280. "event": {
  281. "category": "process",
  282. "id": "aR3NWVOs",
  283. "sequence": 4
  284. },
  285. "process": {
  286. "pid": 2012,
  287. "name": "regsvr32.exe",
  288. "command_line": "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll",
  289. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  290. }
  291. }
  292. },
  293. {
  294. "_index": ".ds-my-data-stream-2099.12.07-000001",
  295. "_id": "yDwnGIJouOYGBzP0ZE9n",
  296. "_source": {
  297. "@timestamp": "2099-12-07T11:07:10.000Z",
  298. "event": {
  299. "category": "file",
  300. "id": "tZ1NWVOs",
  301. "sequence": 5
  302. },
  303. "process": {
  304. "pid": 2012,
  305. "name": "regsvr32.exe",
  306. "executable": "C:\\Windows\\System32\\regsvr32.exe"
  307. },
  308. "file": {
  309. "path": "C:\\Windows\\System32\\scrobj.dll",
  310. "name": "scrobj.dll"
  311. }
  312. }
  313. }
  314. ]
  315. }
  316. ]
  317. }
  318. }
  319. ----
  320. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  321. // TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/]
  322. // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
  323. // TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]
  324. Use the <<eql-until-keyword,`until` keyword>> to specify an expiration
  325. event for sequences. Matching sequences must end before this event.
  326. [source,console]
  327. ----
  328. GET /my-data-stream/_eql/search
  329. {
  330. "query": """
  331. sequence by process.pid with maxspan=1h
  332. [ process where process.name == "regsvr32.exe" ]
  333. [ file where stringContains(file.name, "scrobj.dll") ]
  334. until [ process where event.type == "termination" ]
  335. """
  336. }
  337. ----
  338. // TEST[setup:sec_logs]
  339. [discrete]
  340. [[retrieve-selected-fields]]
  341. === Retrieve selected fields
  342. By default, each hit in the search response includes the document `_source`,
  343. which is the entire JSON object that was provided when indexing the document.
  344. You can use the <<common-options-response-filtering,`filter_path`>> query
  345. parameter to filter the API response. For example, the following search returns
  346. only the timestamp and PID from the `_source` of each matching event.
  347. [source,console]
  348. ----
  349. GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits.events._source.process.pid
  350. {
  351. "query": """
  352. process where process.name == "regsvr32.exe"
  353. """
  354. }
  355. ----
  356. // TEST[setup:sec_logs]
  357. The API returns the following response.
  358. [source,console-result]
  359. ----
  360. {
  361. "hits": {
  362. "events": [
  363. {
  364. "_source": {
  365. "@timestamp": "2099-12-07T11:07:09.000Z",
  366. "process": {
  367. "pid": 2012
  368. }
  369. }
  370. },
  371. {
  372. "_source": {
  373. "@timestamp": "2099-12-07T11:07:10.000Z",
  374. "process": {
  375. "pid": 2012
  376. }
  377. }
  378. }
  379. ]
  380. }
  381. }
  382. ----
  383. You can also use the `fields` parameter to retrieve and format specific fields
  384. in the response. This field is identical to the search API's
  385. <<search-fields,`fields` parameter>>.
  386. include::{es-repo-dir}/search/search-your-data/retrieve-selected-fields.asciidoc[tag=fields-param-desc]
  387. The following EQL search uses the `fields` parameter to retrieve values for the
  388. `event.type` field, all fields starting with `process.`, and the `@timestamp`
  389. field. The request also uses the `filter_path` query parameter to exclude the
  390. `_source` of each hit.
  391. [source,console]
  392. ----
  393. GET /my-data-stream/_eql/search?filter_path=-hits.events._source
  394. {
  395. "query": """
  396. process where process.name == "regsvr32.exe"
  397. """,
  398. "fields": [
  399. "event.type",
  400. "process.*", <1>
  401. {
  402. "field": "@timestamp", <2>
  403. "format": "epoch_millis"
  404. }
  405. ]
  406. }
  407. ----
  408. // TEST[setup:sec_logs]
  409. include::{es-repo-dir}/search/search-your-data/retrieve-selected-fields.asciidoc[tag=fields-param-callouts]
  410. The values are returned as a flat list in the `fields` section of each hit:
  411. [source,console-result]
  412. ----
  413. {
  414. "is_partial": false,
  415. "is_running": false,
  416. "took": 60,
  417. "timed_out": false,
  418. "hits": {
  419. "total": {
  420. "value": 2,
  421. "relation": "eq"
  422. },
  423. "events": [
  424. {
  425. "_index": ".ds-my-data-stream-2099.12.07-000001",
  426. "_id": "OQmfCaduce8zoHT93o4H",
  427. "fields": {
  428. "process.name": [
  429. "regsvr32.exe"
  430. ],
  431. "process.name.keyword": [
  432. "regsvr32.exe"
  433. ],
  434. "@timestamp": [
  435. "4100324829000"
  436. ],
  437. "process.command_line": [
  438. "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll"
  439. ],
  440. "process.command_line.keyword": [
  441. "regsvr32.exe /s /u /i:https://...RegSvr32.sct scrobj.dll"
  442. ],
  443. "process.executable.keyword": [
  444. "C:\\Windows\\System32\\regsvr32.exe"
  445. ],
  446. "process.pid": [
  447. 2012
  448. ],
  449. "process.executable": [
  450. "C:\\Windows\\System32\\regsvr32.exe"
  451. ]
  452. }
  453. },
  454. {
  455. "_index": ".ds-my-data-stream-2099.12.07-000001",
  456. "_id": "xLkCaj4EujzdNSxfYLbO",
  457. "fields": {
  458. "process.name": [
  459. "regsvr32.exe"
  460. ],
  461. "process.name.keyword": [
  462. "regsvr32.exe"
  463. ],
  464. "@timestamp": [
  465. "4100324830000"
  466. ],
  467. "event.type": [
  468. "termination"
  469. ],
  470. "process.executable.keyword": [
  471. "C:\\Windows\\System32\\regsvr32.exe"
  472. ],
  473. "process.pid": [
  474. 2012
  475. ],
  476. "process.executable": [
  477. "C:\\Windows\\System32\\regsvr32.exe"
  478. ]
  479. }
  480. }
  481. ]
  482. }
  483. }
  484. ----
  485. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  486. // TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
  487. // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
  488. // TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]
  489. [discrete]
  490. [[eql-use-runtime-fields]]
  491. === Use runtime fields
  492. Use the `runtime_mappings` parameter to extract and create <<runtime,runtime
  493. fields>> during a search. Use the `fields` parameter to include runtime fields
  494. in the response.
  495. The following search creates a `day_of_week` runtime field from the `@timestamp`
  496. and returns it in the response.
  497. [source,console]
  498. ----
  499. GET /my-data-stream/_eql/search?filter_path=-hits.events._source
  500. {
  501. "runtime_mappings": {
  502. "day_of_week": {
  503. "type": "keyword",
  504. "script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())"
  505. }
  506. },
  507. "query": """
  508. process where process.name == "regsvr32.exe"
  509. """,
  510. "fields": [
  511. "@timestamp",
  512. "day_of_week"
  513. ]
  514. }
  515. ----
  516. // TEST[setup:sec_logs]
  517. The API returns:
  518. [source,console-result]
  519. ----
  520. {
  521. "is_partial": false,
  522. "is_running": false,
  523. "took": 60,
  524. "timed_out": false,
  525. "hits": {
  526. "total": {
  527. "value": 2,
  528. "relation": "eq"
  529. },
  530. "events": [
  531. {
  532. "_index": ".ds-my-data-stream-2099.12.07-000001",
  533. "_id": "OQmfCaduce8zoHT93o4H",
  534. "fields": {
  535. "@timestamp": [
  536. "2099-12-07T11:07:09.000Z"
  537. ],
  538. "day_of_week": [
  539. "MONDAY"
  540. ]
  541. }
  542. },
  543. ...
  544. ]
  545. }
  546. }
  547. ----
  548. // TESTRESPONSE[s/"took": 60/"took": $body.took/]
  549. // TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
  550. // TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
  551. // TESTRESPONSE[s/\.\.\./$body.hits.events.1/]
  552. [discrete]
  553. [[specify-a-timestamp-or-event-category-field]]
  554. === Specify a timestamp or event category field
  555. The EQL search API uses the `@timestamp` and `event.category` fields from the
  556. {ecs-ref}[ECS] by default. To specify different fields, use the
  557. `timestamp_field` and `event_category_field` parameters:
  558. [source,console]
  559. ----
  560. GET /my-data-stream/_eql/search
  561. {
  562. "timestamp_field": "file.accessed",
  563. "event_category_field": "file.type",
  564. "query": """
  565. file where (file.size > 1 and file.type == "file")
  566. """
  567. }
  568. ----
  569. // TEST[setup:sec_logs]
  570. The event category field must be mapped as a <<keyword,`keyword`>> family field
  571. type. The timestamp field should be mapped as a <<date,`date`>> field type.
  572. <<date_nanos,`date_nanos`>> timestamp fields are not supported. You cannot use a
  573. <<nested,`nested`>> field or the sub-fields of a `nested` field as the timestamp
  574. or event category field.
  575. [discrete]
  576. [[eql-search-specify-a-sort-tiebreaker]]
  577. === Specify a sort tiebreaker
  578. By default, the EQL search API returns matching hits by timestamp. If two or
  579. more events share the same timestamp, {es} uses a tiebreaker field value to sort
  580. the events in ascending order. {es} orders events with no
  581. tiebreaker value after events with a value.
  582. If you don't specify a tiebreaker field or the events also share the same
  583. tiebreaker value, {es} considers the events concurrent. Concurrent events cannot
  584. be part of the same sequence and may not be returned in a consistent sort order.
  585. To specify a tiebreaker field, use the `tiebreaker_field` parameter. If you use
  586. the {ecs-ref}[ECS], we recommend using `event.sequence` as the tiebreaker field.
  587. [source,console]
  588. ----
  589. GET /my-data-stream/_eql/search
  590. {
  591. "tiebreaker_field": "event.sequence",
  592. "query": """
  593. process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
  594. """
  595. }
  596. ----
  597. // TEST[setup:sec_logs]
  598. [discrete]
  599. [[eql-search-filter-query-dsl]]
  600. === Filter using Query DSL
  601. The `filter` parameter uses <<query-dsl,Query DSL>> to limit the documents on
  602. which an EQL query runs.
  603. [source,console]
  604. ----
  605. GET /my-data-stream/_eql/search
  606. {
  607. "filter": {
  608. "range": {
  609. "@timestamp": {
  610. "gte": "now-1d/d",
  611. "lt": "now/d"
  612. }
  613. }
  614. },
  615. "query": """
  616. file where (file.type == "file" and file.name == "cmd.exe")
  617. """
  618. }
  619. ----
  620. // TEST[setup:sec_logs]
  621. [discrete]
  622. [[eql-search-async]]
  623. === Run an async EQL search
  624. By default, EQL search requests are synchronous and wait for complete results
  625. before returning a response. However, complete results can take longer for
  626. searches across large data sets, <<data-tiers,cold>> or <<data-tiers,frozen>>
  627. data, or <<modules-cross-cluster-search,multiple clusters>>.
  628. To avoid long waits, run an async EQL search. Set `wait_for_completion_timeout`
  629. to a duration you'd like to wait for synchronous results.
  630. +
  631. [source,console]
  632. ----
  633. GET /my-data-stream/_eql/search
  634. {
  635. "wait_for_completion_timeout": "2s",
  636. "query": """
  637. process where process.name == "cmd.exe"
  638. """
  639. }
  640. ----
  641. // TEST[setup:sec_logs]
  642. If the request doesn't finish within the timeout period, the search becomes async
  643. and returns a response that includes:
  644. * A search ID
  645. * An `is_partial` value of `true`, indicating the search results are
  646. incomplete
  647. * An `is_running` value of `true`, indicating the search is ongoing
  648. The async search continues to run in the background without blocking other
  649. requests.
  650. [source,console-result]
  651. ----
  652. {
  653. "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  654. "is_partial": true,
  655. "is_running": true,
  656. "took": 2000,
  657. "timed_out": false,
  658. "hits": ...
  659. }
  660. ----
  661. // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
  662. // TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
  663. // TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
  664. // TESTRESPONSE[s/"took": 2000/"took": $body.took/]
  665. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  666. To check the progress of an async search, use the <<get-async-eql-search-api,get
  667. async EQL search API>> with the search ID. Specify how long you'd like for
  668. complete results in the `wait_for_completion_timeout` parameter.
  669. [source,console]
  670. ----
  671. GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
  672. ----
  673. // TEST[skip: no access to search ID]
  674. If the response's `is_running` value is `false`, the async search has finished.
  675. If the `is_partial` value is `false`, the returned search results are
  676. complete.
  677. [source,console-result]
  678. ----
  679. {
  680. "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  681. "is_partial": false,
  682. "is_running": false,
  683. "took": 2000,
  684. "timed_out": false,
  685. "hits": ...
  686. }
  687. ----
  688. // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
  689. // TESTRESPONSE[s/"took": 2000/"took": $body.took/]
  690. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  691. Another more lightweight way to check the progress of an async search is to use
  692. the <<get-async-eql-status-api,get async EQL status API>> with the search ID.
  693. [source,console]
  694. ----
  695. GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=
  696. ----
  697. // TEST[skip: no access to search ID]
  698. [source,console-result]
  699. ----
  700. {
  701. "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  702. "is_running": false,
  703. "is_partial": false,
  704. "expiration_time_in_millis": 1611690295000,
  705. "completion_status": 200
  706. }
  707. ----
  708. // TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
  709. // TESTRESPONSE[s/"expiration_time_in_millis": 1611690295000/"expiration_time_in_millis": $body.expiration_time_in_millis/]
  710. [discrete]
  711. [[eql-search-store-async-eql-search]]
  712. === Change the search retention period
  713. By default, the EQL search API stores async searches for five days. After this
  714. period, any searches and their results are deleted. Use the `keep_alive`
  715. parameter to change this retention period:
  716. [source,console]
  717. ----
  718. GET /my-data-stream/_eql/search
  719. {
  720. "keep_alive": "2d",
  721. "wait_for_completion_timeout": "2s",
  722. "query": """
  723. process where process.name == "cmd.exe"
  724. """
  725. }
  726. ----
  727. // TEST[setup:sec_logs]
  728. You can use the <<get-async-eql-search-api,get async EQL search API>>'s
  729. `keep_alive` parameter to later change the retention period. The new retention
  730. period starts after the get request runs.
  731. [source,console]
  732. ----
  733. GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
  734. ----
  735. // TEST[skip: no access to search ID]
  736. Use the <<delete-async-eql-search-api,delete async EQL search API>> to
  737. manually delete an async EQL search before the `keep_alive` period ends. If the
  738. search is still ongoing, {es} cancels the search request.
  739. [source,console]
  740. ----
  741. DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
  742. ----
  743. // TEST[skip: no access to search ID]
  744. [discrete]
  745. [[eql-search-store-sync-eql-search]]
  746. === Store synchronous EQL searches
  747. By default, the EQL search API only stores async searches. To save a synchronous
  748. search, set `keep_on_completion` to `true`:
  749. [source,console]
  750. ----
  751. GET /my-data-stream/_eql/search
  752. {
  753. "keep_on_completion": true,
  754. "wait_for_completion_timeout": "2s",
  755. "query": """
  756. process where process.name == "cmd.exe"
  757. """
  758. }
  759. ----
  760. // TEST[setup:sec_logs]
  761. The response includes a search ID. `is_partial` and `is_running` are `false`,
  762. indicating the EQL search was synchronous and returned complete results.
  763. [source,console-result]
  764. ----
  765. {
  766. "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  767. "is_partial": false,
  768. "is_running": false,
  769. "took": 52,
  770. "timed_out": false,
  771. "hits": ...
  772. }
  773. ----
  774. // TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
  775. // TESTRESPONSE[s/"took": 52/"took": $body.took/]
  776. // TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]
  777. Use the <<get-async-eql-search-api,get async EQL search API>> to get the
  778. same results later:
  779. [source,console]
  780. ----
  781. GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
  782. ----
  783. // TEST[skip: no access to search ID]
  784. Saved synchronous searches are still subject to the `keep_alive` parameter's
  785. retention period. When this period ends, the search and its results are deleted.
  786. You can also check only the status of the saved synchronous search without
  787. results by using <<get-async-eql-status-api,get async EQL status API>>.
  788. You can also manually delete saved synchronous searches using the
  789. <<delete-async-eql-search-api,delete async EQL search API>>.
  790. include::syntax.asciidoc[]
  791. include::functions.asciidoc[]
  792. include::pipes.asciidoc[]
  793. include::detect-threats-with-eql.asciidoc[]