syntax.asciidoc 31 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034
  1. [role="xpack"]
  2. [testenv="basic"]
  3. [[eql-syntax]]
  4. == EQL syntax reference
  5. ++++
  6. <titleabbrev>Syntax reference</titleabbrev>
  7. ++++
  8. [discrete]
  9. [[eql-basic-syntax]]
  10. === Basic syntax
  11. EQL queries require an event category and a matching condition. The `where`
  12. keyword connects them.
  13. [source,eql]
  14. ----
  15. event_category where condition
  16. ----
  17. For example, the following EQL query matches `process` events with a
  18. `process.name` field value of `svchost.exe`:
  19. [source,eql]
  20. ----
  21. process where process.name == "svchost.exe"
  22. ----
  23. [discrete]
  24. [[eql-syntax-event-categories]]
  25. === Event categories
  26. An event category is a valid, indexed value of the
  27. <<eql-required-fields,event category field>>. You can set the event category
  28. field using the `event_category_field` parameter of the EQL search API.
  29. [discrete]
  30. [[eql-syntax-match-any-event-category]]
  31. === Match any event category
  32. To match events of any category, use the `any` keyword. You can also use the
  33. `any` keyword to search for documents without a event category field.
  34. For example, the following EQL query matches any documents with a
  35. `network.protocol` field value of `http`:
  36. [source,eql]
  37. ----
  38. any where network.protocol == "http"
  39. ----
  40. [discrete]
  41. [[eql-syntax-escape-an-event-category]]
  42. === Escape an event category
  43. Use enclosing double quotes (`"`) or three enclosing double quotes (`"""`) to
  44. escape event categories that:
  45. * Contain a special character, such as a hyphen (`-`) or dot (`.`)
  46. * Contain a space
  47. * Start with a numeral
  48. [source,eql]
  49. ----
  50. ".my.event.category"
  51. "my-event-category"
  52. "my event category"
  53. "6eventcategory"
  54. """.my.event.category"""
  55. """my-event-category"""
  56. """my event category"""
  57. """6eventcategory"""
  58. ----
  59. [discrete]
  60. [[eql-syntax-escape-a-field-name]]
  61. === Escape a field name
  62. Use enclosing enclosing backticks (+++`+++) to escape field names that:
  63. * Contain a hyphen (`-`)
  64. * Contain a space
  65. * Start with a numeral
  66. [source,eql]
  67. ----
  68. `my-field`
  69. `my field`
  70. `6myfield`
  71. ----
  72. Use double backticks (+++``+++) to escape any backticks (+++`+++) in the field
  73. name.
  74. [source,eql]
  75. ----
  76. my`field -> `my``field`
  77. ----
  78. [discrete]
  79. [[eql-syntax-conditions]]
  80. === Conditions
  81. A condition consists of one or more criteria an event must match.
  82. You can specify and combine these criteria using the following operators. Most
  83. EQL operators are case-sensitive by default.
  84. [discrete]
  85. [[eql-syntax-comparison-operators]]
  86. === Comparison operators
  87. [source,eql]
  88. ----
  89. < <= == : != >= >
  90. ----
  91. `<` (less than)::
  92. Returns `true` if the value to the left of the operator is less than the value
  93. to the right. Otherwise returns `false`.
  94. `<=` (less than or equal) ::
  95. Returns `true` if the value to the left of the operator is less than or equal to
  96. the value to the right. Otherwise returns `false`.
  97. `==` (equal, case-sensitive)::
  98. Returns `true` if the values to the left and right of the operator are equal.
  99. Otherwise returns `false`. For strings, matching is case-sensitive. Wildcards
  100. are not supported.
  101. `:` (equal, case-insensitive)::
  102. Returns `true` if strings to the left and right of the operator are equal.
  103. Otherwise returns `false`. Matching is case-insensitive and can only be used to
  104. compare strings. Supports <<eql-syntax-wildcards,wildcards>> and
  105. <<eql-syntax-lookup-operators,list lookups>>.
  106. [IMPORTANT]
  107. ====
  108. Avoid using the `==` or `:` operators to perform exact matching on
  109. <<text,`text`>> field values.
  110. By default, {es} changes the values of `text` fields as part of <<analysis,
  111. analysis>>. This can make finding exact matches for `text` field values
  112. difficult.
  113. To search `text` fields, consider using a <<eql-search-filter-query-dsl,query
  114. DSL filter>> that contains a <<query-dsl-match-query,`match`>> query.
  115. ====
  116. `!=` (not equal, case-sensitive)::
  117. Returns `true` if the values to the left and right of the operator are not
  118. equal. Otherwise returns `false`. For strings, matching is case-sensitive.
  119. Wildcards are not supported.
  120. `>=` (greater than or equal) ::
  121. Returns `true` if the value to the left of the operator is greater than or equal
  122. to the value to the right. Otherwise returns `false`. When comparing strings,
  123. the operator uses a case-sensitive lexicographic order.
  124. `>` (greater than)::
  125. Returns `true` if the value to the left of the operator is greater than the
  126. value to the right. Otherwise returns `false`. When comparing strings,
  127. the operator uses a case-sensitive lexicographic order.
  128. NOTE: `=` is not supported as an equal operator. Use `==` or `:` instead.
  129. You cannot chain comparison operators. Instead, use a
  130. <<eql-syntax-logical-operators,logical operator>> between comparisons. For
  131. example, `foo < bar <= baz` is not supported. However, you can rewrite the
  132. expression as `foo < bar and bar <= baz`, which is supported.
  133. You also cannot use comparison operators to compare a field to another field.
  134. This applies even if the fields are changed using a <<eql-functions,function>>.
  135. *Example* +
  136. The following EQL query compares the `process.parent_name` field
  137. value to a static value, `foo`. This comparison is supported.
  138. However, the query also compares the `process.parent.name` field value to the
  139. `process.name` field. This comparison is not supported and will return an
  140. error for the entire query.
  141. [source,eql]
  142. ----
  143. process where process.parent.name == "foo" and process.parent.name == process.name
  144. ----
  145. Instead, you can rewrite the query to compare both the `process.parent.name`
  146. and `process.name` fields to static values.
  147. [source,eql]
  148. ----
  149. process where process.parent.name == "foo" and process.name == "foo"
  150. ----
  151. [discrete]
  152. [[eql-syntax-logical-operators]]
  153. === Logical operators
  154. [source,eql]
  155. ----
  156. and or not
  157. ----
  158. `and`::
  159. Returns `true` only if the condition to the left and right _both_ return `true`.
  160. Otherwise returns `false`.
  161. `or`::
  162. Returns `true` if one of the conditions to the left or right `true`.
  163. Otherwise returns `false`.
  164. `not`::
  165. Returns `true` if the condition to the right is `false`.
  166. [discrete]
  167. [[eql-syntax-lookup-operators]]
  168. === Lookup operators
  169. [source,eql]
  170. ----
  171. user.name in ("Administrator", "SYSTEM", "NETWORK SERVICE")
  172. user.name in~ ("administrator", "system", "network service")
  173. user.name not in ("Administrator", "SYSTEM", "NETWORK SERVICE")
  174. user.name not in~ ("administrator", "system", "network service")
  175. user.name : ("administrator", "system", "network service")
  176. ----
  177. `in` (case-sensitive)::
  178. Returns `true` if the value is contained in the provided list. For strings,
  179. matching is case-sensitive.
  180. `in~` (case-insensitive)::
  181. Returns `true` if the value is contained in the provided list. For strings,
  182. matching is case-insensitive.
  183. `not in` (case-sensitive)::
  184. Returns `true` if the value is not contained in the provided list. For strings,
  185. matching is case-sensitive.
  186. `not in~` (case-insensitive)::
  187. Returns `true` if the value is not contained in the provided list. For strings,
  188. matching is case-insensitive.
  189. `:` (case-insensitive)::
  190. Returns `true` if the value is contained in the provided list. Can only be used
  191. to compare strings.
  192. [discrete]
  193. [[eql-syntax-math-operators]]
  194. === Math operators
  195. [source,eql]
  196. ----
  197. + - * / %
  198. ----
  199. `+` (add)::
  200. Adds the values to the left and right of the operator.
  201. `-` (subtract)::
  202. Subtracts the value to the right of the operator from the value to the left.
  203. `*` (multiply)::
  204. Multiplies the values to the left and right of the operator.
  205. `/` (divide)::
  206. Divides the value to the left of the operator by the value to the right.
  207. +
  208. [[eql-divide-operator-float-rounding]]
  209. [WARNING]
  210. ====
  211. If both the dividend and divisor are integers, the divide (`\`) operation
  212. _rounds down_ any returned floating point numbers to the nearest integer. To
  213. avoid rounding, convert either the dividend or divisor to a float.
  214. *Example* +
  215. The `process.args_count` field is a <<number,`long`>> integer field containing a
  216. count of process arguments.
  217. A user might expect the following EQL query to only match events with a
  218. `process.args_count` value of `4`.
  219. [source,eql]
  220. ----
  221. process where ( 4 / process.args_count ) == 1
  222. ----
  223. However, the EQL query matches events with a `process.args_count` value of `3`
  224. or `4`.
  225. For events with a `process.args_count` value of `3`, the divide operation
  226. returns a float of `1.333...`, which is rounded down to `1`.
  227. To match only events with a `process.args_count` value of `4`, convert
  228. either the dividend or divisor to a float.
  229. The following EQL query changes the integer `4` to the equivalent float `4.0`.
  230. [source,eql]
  231. ----
  232. process where ( 4.0 / process.args_count ) == 1
  233. ----
  234. ====
  235. `%` (modulo)::
  236. Divides the value to the left of the operator by the value to the right. Returns only the remainder.
  237. [discrete]
  238. [[eql-syntax-match-any-condition]]
  239. === Match any condition
  240. To match events solely on event category, use the `where true` condition.
  241. For example, the following EQL query matches any `file` events:
  242. [source,eql]
  243. ----
  244. file where true
  245. ----
  246. To match any event, you can combine the `any` keyword with the `where true`
  247. condition:
  248. [source,eql]
  249. ----
  250. any where true
  251. ----
  252. [discrete]
  253. [[eql-syntax-strings]]
  254. === Strings
  255. Strings are enclosed in double quotes (`"`).
  256. [source,eql]
  257. ----
  258. "hello world"
  259. ----
  260. Strings enclosed in single quotes (`'`) are not supported.
  261. [discrete]
  262. [[eql-syntax-escape-characters]]
  263. === Escape characters in a string
  264. When used within a string, special characters, such as a carriage return or
  265. double quote (`"`), must be escaped with a preceding backslash (`\`).
  266. [source,eql]
  267. ----
  268. "example \r of \" escaped \n characters"
  269. ----
  270. [options="header"]
  271. |====
  272. | Escape sequence | Literal character
  273. |`\n` | A newline (linefeed) character
  274. |`\r` | A carriage return character
  275. |`\t` | A tab character
  276. |`\\` | A backslash (`\`) character
  277. |`\"` | A double quote (`"`) character
  278. |====
  279. IMPORTANT: The single quote (`'`) character is reserved for future use. You
  280. cannot use an escaped single quote (`\'`) for literal strings. Use an escaped
  281. double quote (`\"`) instead.
  282. [discrete]
  283. [[eql-syntax-raw-strings]]
  284. === Raw strings
  285. Raw strings treat special characters, such as backslashes (`\`), as literal
  286. characters. Raw strings are enclosed in three double quotes (`"""`).
  287. [source,eql]
  288. ----
  289. """Raw string with a literal double quote " and blackslash \ included"""
  290. ----
  291. A raw string cannot contain three consecutive double quotes (`"""`). Instead,
  292. use a regular string with the `\"` escape sequence.
  293. [source,eql]
  294. ----
  295. "String containing \"\"\" three double quotes"
  296. ----
  297. [discrete]
  298. [[eql-syntax-wildcards]]
  299. === Wildcards
  300. For string comparisons using the `:` operator, you can use the `*` and `?`
  301. wildcards to match specific patterns. The `*` wildcard matches zero or more
  302. characters:
  303. [source,eql]
  304. ----
  305. my_field : "doc*" // Matches "doc", "docs", or "document" but not "dos"
  306. my_field : "*doc" // Matches "adoc" or "asciidoc"
  307. my_field : "d*c" // Matches "doc" or "disc"
  308. ----
  309. The `?` wildcard matches exactly one character:
  310. [source,eql]
  311. ----
  312. my_field : "doc?" // Matches "docs" but not "doc", "document", or "dos"
  313. my_field : "?doc" // Matches "adoc" but not "asciidoc"
  314. my_field : "d?c" // Matches "doc" but not "disc"
  315. ----
  316. The `:` operator also supports wildcards in <<eql-syntax-lookup-operators,list
  317. lookups>>:
  318. [source,eql]
  319. ----
  320. my_field : ("doc*", "f*o", "ba?", "qux")
  321. ----
  322. [discrete]
  323. [[eql-sequences]]
  324. === Sequences
  325. You can use EQL sequences to describe and match an ordered series of events.
  326. Each item in a sequence is an event category and event condition,
  327. surrounded by square brackets (`[ ]`). Events are listed in ascending
  328. chronological order, with the most recent event listed last.
  329. [source,eql]
  330. ----
  331. sequence
  332. [ event_category_1 where condition_1 ]
  333. [ event_category_2 where condition_2 ]
  334. ...
  335. ----
  336. *Example* +
  337. The following EQL sequence query matches this series of ordered events:
  338. . Start with an event with:
  339. +
  340. --
  341. * An event category of `file`
  342. * A `file.extension` of `exe`
  343. --
  344. . Followed by an event with an event category of `process`
  345. [source,eql]
  346. ----
  347. sequence
  348. [ file where file.extension == "exe" ]
  349. [ process where true ]
  350. ----
  351. [discrete]
  352. [[eql-with-maxspan-keywords]]
  353. === `with maxspan` keywords
  354. You can use the `with maxspan` keywords to constrain a sequence to a specified
  355. timespan. All events in a matching sequence must occur within this duration,
  356. starting at the first event's timestamp.
  357. The `maxspan` keyword accepts <<time-units,time value>> arguments.
  358. [source,eql]
  359. ----
  360. sequence with maxspan=30s
  361. [ event_category_1 where condition_1 ] by field_baz
  362. [ event_category_2 where condition_2 ] by field_bar
  363. ...
  364. ----
  365. *Example* +
  366. The following sequence query uses a `maxspan` value of `15m` (15 minutes).
  367. Events in a matching sequence must occur within 15 minutes of the first event's
  368. timestamp.
  369. [source,eql]
  370. ----
  371. sequence with maxspan=15m
  372. [ file where file.extension == "exe" ]
  373. [ process where true ]
  374. ----
  375. [discrete]
  376. [[eql-by-keyword]]
  377. === `by` keyword
  378. You can use the `by` keyword with sequences to only match events that share the
  379. same field values. If a field value should be shared across all events, you
  380. can use `sequence by`.
  381. [source,eql]
  382. ----
  383. sequence by field_foo
  384. [ event_category_1 where condition_1 ] by field_baz
  385. [ event_category_2 where condition_2 ] by field_bar
  386. ...
  387. ----
  388. *Example* +
  389. The following sequence query uses the `by` keyword to constrain matching events
  390. to:
  391. * Events with the same `user.name` value
  392. * `file` events with a `file.path` value equal to the following `process`
  393. event's `process.path` value.
  394. [source,eql]
  395. ----
  396. sequence
  397. [ file where file.extension == "exe" ] by user.name, file.path
  398. [ process where true ] by user.name, process.path
  399. ----
  400. Because the `user.name` field is shared across all events in the sequence, it
  401. can be included using `sequence by`. The following sequence is equivalent to the
  402. prior one.
  403. [source,eql]
  404. ----
  405. sequence by user.name
  406. [ file where file.extension == "exe" ] by file.path
  407. [ process where true ] by process.path
  408. ----
  409. You can combine the `sequence by` and `with maxspan` keywords to constrain a
  410. sequence by both field values and a timespan.
  411. [source,eql]
  412. ----
  413. sequence by field_foo with maxspan=30s
  414. [ event_category_1 where condition_1 ] by field_baz
  415. [ event_category_2 where condition_2 ] by field_bar
  416. ...
  417. ----
  418. *Example* +
  419. The following sequence query uses the `sequence by` keyword and `with maxspan`
  420. keywords to match only a sequence of events that:
  421. * Share the same `user.name` field values
  422. * Occur within `15m` (15 minutes) of the first matching event
  423. [source,eql]
  424. ----
  425. sequence by user.name with maxspan=15m
  426. [ file where file.extension == "exe" ] by file.path
  427. [ process where true ] by process.path
  428. ----
  429. [discrete]
  430. [[eql-until-keyword]]
  431. === `until` keyword
  432. You can use the `until` keyword to specify an expiration event for a sequence.
  433. If this expiration event occurs _between_ matching events in a sequence, the
  434. sequence expires and is not considered a match. If the expiration event occurs
  435. _after_ matching events in a sequence, the sequence is still considered a
  436. match. The expiration event is not included in the results.
  437. [source,eql]
  438. ----
  439. sequence
  440. [ event_category_1 where condition_1 ]
  441. [ event_category_2 where condition_2 ]
  442. ...
  443. until [ event_category_3 where condition_3 ]
  444. ----
  445. *Example* +
  446. A dataset contains the following event sequences, grouped by shared IDs:
  447. [source,txt]
  448. ----
  449. A, B
  450. A, B, C
  451. A, C, B
  452. ----
  453. The following EQL query searches the dataset for sequences containing
  454. event `A` followed by event `B`. Event `C` is used as an expiration event.
  455. [source,eql]
  456. ----
  457. sequence by ID
  458. A
  459. B
  460. until C
  461. ----
  462. The query matches sequences `A, B` and `A, B, C` but not `A, C, B`.
  463. [TIP]
  464. ====
  465. The `until` keyword can be useful when searching for process sequences in
  466. Windows event logs.
  467. In Windows, a process ID (PID) is unique only while a process is running. After
  468. a process terminates, its PID can be reused.
  469. You can search for a sequence of events with the same PID value using the `by`
  470. and `sequence by` keywords.
  471. *Example* +
  472. The following EQL query uses the `sequence by` keyword to match a
  473. sequence of events that share the same `process.pid` value.
  474. [source,eql]
  475. ----
  476. sequence by process.pid
  477. [ process where event.type == "start" and process.name == "cmd.exe" ]
  478. [ process where file.extension == "exe" ]
  479. ----
  480. However, due to PID reuse, this can result in a matching sequence that
  481. contains events across unrelated processes. To prevent false positives, you can
  482. use the `until` keyword to end matching sequences before a process termination
  483. event.
  484. The following EQL query uses the `until` keyword to end sequences before
  485. `process` events with an `event.type` of `stop`. These events indicate a process
  486. has been terminated.
  487. [source,eql]
  488. ----
  489. sequence by process.pid
  490. [ process where event.type == "start" and process.name == "cmd.exe" ]
  491. [ process where file.extension == "exe" ]
  492. until [ process where event.type == "stop" ]
  493. ----
  494. ====
  495. [discrete]
  496. [[eql-functions]]
  497. === Functions
  498. You can use EQL functions to convert data types, perform math, manipulate
  499. strings, and more. For a list of supported functions, see <<eql-function-ref>>.
  500. [discrete]
  501. [[eql-case-insensitive-functions]]
  502. === Case-insensitive functions
  503. Most EQL functions are case-sensitive by default. To make a function
  504. case-insensitive, use the `~` operator after the function name:
  505. [source,eql]
  506. ----
  507. stringContains(process.name,".exe") // Matches ".exe" but not ".EXE" or ".Exe"
  508. stringContains~(process.name,".exe") // Matches ".exe", ".EXE", or ".Exe"
  509. ----
  510. [discrete]
  511. [[eql-how-functions-impact-search-performance]]
  512. === How functions impact search performance
  513. Using functions in EQL queries can result in slower search speeds. If you
  514. often use functions to transform indexed data, you can speed up search by making
  515. these changes during indexing instead. However, that often means slower index
  516. speeds.
  517. *Example* +
  518. An index contains the `file.path` field. `file.path` contains the full path to a
  519. file, including the file extension.
  520. When running EQL searches, users often use the `endsWith` function with the
  521. `file.path` field to match file extensions:
  522. [source,eql]
  523. ----
  524. file where endsWith(file.path,".exe") or endsWith(file.path,".dll")
  525. ----
  526. While this works, it can be repetitive to write and can slow search speeds. To
  527. speed up search, you can do the following instead:
  528. . <<indices-put-mapping,Add a new field>>, `file.extension`, to the index. The
  529. `file.extension` field will contain only the file extension from the
  530. `file.path` field.
  531. . Use an <<ingest,ingest pipeline>> containing the <<grok-processor,`grok`>>
  532. processor or another preprocessor tool to extract the file extension from the
  533. `file.path` field before indexing.
  534. . Index the extracted file extension to the `file.extension` field.
  535. These changes may slow indexing but allow for faster searches. Users
  536. can use the `file.extension` field instead of multiple `endsWith` function
  537. calls:
  538. [source,eql]
  539. ----
  540. file where file.extension in ("exe", "dll")
  541. ----
  542. We recommend testing and benchmarking any indexing changes before deploying them
  543. in production. See <<tune-for-indexing-speed>> and <<tune-for-search-speed>>.
  544. [discrete]
  545. [[eql-pipes]]
  546. === Pipes
  547. EQL pipes filter, aggregate, and post-process events returned by
  548. an EQL query. You can use pipes to narrow down EQL query results or make them
  549. more specific.
  550. Pipes are delimited using the pipe (`|`) character.
  551. [source,eql]
  552. ----
  553. event_category where condition | pipe
  554. ----
  555. *Example* +
  556. The following EQL query uses the `tail` pipe to return only the 10 most recent
  557. events matching the query.
  558. [source,eql]
  559. ----
  560. authentication where agent.id == 4624
  561. | tail 10
  562. ----
  563. You can pass the output of a pipe to another pipe. This lets you use multiple
  564. pipes with a single query.
  565. For a list of supported pipes, see <<eql-pipe-ref>>.
  566. [discrete]
  567. [[eql-syntax-limitations]]
  568. === Limitations
  569. EQL does not support the following features and syntax.
  570. [discrete]
  571. [[eql-compare-fields]]
  572. ==== Comparing fields
  573. You cannot use EQL comparison operators to compare a field to
  574. another field. This applies even if the fields are changed using a
  575. <<eql-functions,function>>.
  576. [discrete]
  577. [[eql-array-fields]]
  578. ==== Array field values are not supported
  579. EQL does not support <<array,array>> field values, also known as
  580. multi-value fields. EQL searches on array field values may return inconsistent
  581. results.
  582. [discrete]
  583. [[eql-nested-fields]]
  584. ==== EQL search on nested fields
  585. You cannot use EQL to search the values of a <<nested,`nested`>> field or the
  586. sub-fields of a `nested` field. However, data streams and indices containing
  587. `nested` field mappings are otherwise supported.
  588. [discrete]
  589. [[eql-unsupported-syntax]]
  590. ==== Differences from Endgame EQL syntax
  591. {es} EQL differs from the {eql-ref}/index.html[Elastic Endgame EQL syntax] as
  592. follows:
  593. * Most operators and functions in {es} EQL are case-sensitive by default. For
  594. case-insensitive equality comparisons, use the `:` operator. To make a function
  595. case-insensitive, use the `~` operator after the function name. See
  596. <<eql-case-insensitive-functions>>.
  597. * Comparisons using the `==` and `!=` operators do not expand wildcard
  598. characters. For example, `process_name == "cmd*.exe"` interprets `*` as a
  599. literal asterisk, not a wildcard. For case-sensitive wildcard matching, use the
  600. <<eql-fn-wildcard,`wildcard`>> function.
  601. * `=` cannot be substituted for the `==` operator.
  602. * Strings enclosed in single quotes (`'`) are not supported. Enclose strings in
  603. double quotes (`"`) instead.
  604. * `?"` and `?'` do not indicate raw strings. Enclose raw strings in
  605. three double quotes (`"""`) instead.
  606. * {es} EQL does not support:
  607. ** Array functions:
  608. *** {eql-ref}/functions.html#arrayContains[`arrayContains`]
  609. *** {eql-ref}/functions.html#arrayCount[`arrayCount`]
  610. *** {eql-ref}/functions.html#arraySearch[`arraySearch`]
  611. ** The {eql-ref}//functions.html#match[`match`] function
  612. ** {eql-ref}/joins.html[Joins]
  613. ** {eql-ref}/basic-syntax.html#event-relationships[Lineage-related keywords]:
  614. *** `child of`
  615. *** `descendant of`
  616. *** `event of`
  617. ** The following {eql-ref}/pipes.html[pipes]:
  618. *** {eql-ref}/pipes.html#count[`count`]
  619. *** {eql-ref}/pipes.html#filter[`filter`]
  620. *** {eql-ref}/pipes.html#sort[`sort`]
  621. *** {eql-ref}/pipes.html#unique[`unique`]
  622. *** {eql-ref}/pipes.html#unique-count[`unique_count`]
  623. [discrete]
  624. [[eql-how-sequence-queries-handle-matches]]
  625. ==== How sequence queries handle matches
  626. <<eql-sequences,Sequence queries>> don't find all potential matches for a
  627. sequence. This approach would be too slow and costly for large event data sets.
  628. Instead, a sequence query handles pending sequence matches as a
  629. {wikipedia}/Finite-state_machine[state machine]:
  630. * Each event item in the sequence query is a state in the machine.
  631. * Only one pending sequence can be in each state at a time.
  632. * If two pending sequences are in the same state at the same time, the most
  633. recent sequence overwrites the older one.
  634. * If the query includes <<eql-by-keyword,`by` fields>>, the query uses a
  635. separate state machine for each unique `by` field value.
  636. .*Example*
  637. [%collapsible]
  638. ====
  639. A data set contains the following `process` events in ascending chronological
  640. order:
  641. [source,js]
  642. ----
  643. { "index" : { "_id": "1" } }
  644. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  645. { "index" : { "_id": "2" } }
  646. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  647. { "index" : { "_id": "3" } }
  648. { "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
  649. { "index" : { "_id": "4" } }
  650. { "user": { "name": "root" }, "process": { "name": "bash" }, ...}
  651. { "index" : { "_id": "5" } }
  652. { "user": { "name": "root" }, "process": { "name": "bash" }, ...}
  653. { "index" : { "_id": "6" } }
  654. { "user": { "name": "elkbee" }, "process": { "name": "attrib" }, ...}
  655. { "index" : { "_id": "7" } }
  656. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  657. { "index" : { "_id": "8" } }
  658. { "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
  659. { "index" : { "_id": "9" } }
  660. { "user": { "name": "root" }, "process": { "name": "cat" }, ...}
  661. { "index" : { "_id": "10" } }
  662. { "user": { "name": "elkbee" }, "process": { "name": "cat" }, ...}
  663. { "index" : { "_id": "11" } }
  664. { "user": { "name": "root" }, "process": { "name": "cat" }, ...}
  665. ----
  666. // NOTCONSOLE
  667. An EQL sequence query searches the data set:
  668. [source,eql]
  669. ----
  670. sequence by user.name
  671. [process where process.name == "attrib"]
  672. [process where process.name == "bash"]
  673. [process where process.name == "cat"]
  674. ----
  675. The query's event items correspond to the following states:
  676. * State A: `[process where process.name == "attrib"]`
  677. * State B: `[process where process.name == "bash"]`
  678. * Complete: `[process where process.name == "cat"]`
  679. image::images/eql/sequence-state-machine.svg[align="center"]
  680. To find matching sequences, the query uses separate state machines for each
  681. unique `user.name` value. Based on the data set, you can expect two state
  682. machines: one for the `root` user and one for `elkbee`.
  683. image::images/eql/separate-state-machines.svg[align="center"]
  684. Pending sequence matches move through each machine's states as follows:
  685. [source,txt]
  686. ----
  687. { "index" : { "_id": "1" } }
  688. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  689. // Creates sequence [1] in state A for the "root" user.
  690. //
  691. // +------------------------"root"------------------------+
  692. // | +-----------+ +-----------+ +------------+ |
  693. // | | State A | | State B | | Complete | |
  694. // | +-----------+ +-----------+ +------------+ |
  695. // | | [1] | | | | | |
  696. // | +-----------+ +-----------+ +------------+ |
  697. // +------------------------------------------------------+
  698. { "index" : { "_id": "2" } }
  699. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  700. // Creates sequence [2] in state A for "root", overwriting sequence [1].
  701. //
  702. // +------------------------"root"------------------------+
  703. // | +-----------+ +-----------+ +------------+ |
  704. // | | State A | | State B | | Complete | |
  705. // | +-----------+ +-----------+ +------------+ |
  706. // | | [2] | | | | | |
  707. // | +-----------+ +-----------+ +------------+ |
  708. // +------------------------------------------------------+
  709. { "index" : { "_id": "3" } }
  710. { "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
  711. // Nothing happens. The "elkbee" user has no pending sequence to move
  712. // from state A to state B.
  713. //
  714. // +-----------------------"elkbee"-----------------------+
  715. // | +-----------+ +-----------+ +------------+ |
  716. // | | State A | | State B | | Complete | |
  717. // | +-----------+ +-----------+ +------------+ |
  718. // | | | | | | | |
  719. // | +-----------+ +-----------+ +------------+ |
  720. // +------------------------------------------------------+
  721. { "index" : { "_id": "4" } }
  722. { "user": { "name": "root" }, "process": { "name": "bash" }, ...}
  723. // Sequence [2] moves out of state A for "root".
  724. // State B for "root" now contains [2, 4].
  725. // State A for "root" is empty.
  726. //
  727. // +------------------------"root"------------------------+
  728. // | +-----------+ +-----------+ +------------+ |
  729. // | | State A | | State B | | Complete | |
  730. // | +-----------+ --> +-----------+ +------------+ |
  731. // | | | | [2, 4] | | | |
  732. // | +-----------+ +-----------+ +------------+ |
  733. // +------------------------------------------------------+
  734. { "index" : { "_id": "5" } }
  735. { "user": { "name": "root" }, "process": { "name": "bash" }, ...}
  736. // Nothing happens. State A is empty for "root".
  737. //
  738. // +------------------------"root"------------------------+
  739. // | +-----------+ +-----------+ +------------+ |
  740. // | | State A | | State B | | Complete | |
  741. // | +-----------+ +-----------+ +------------+ |
  742. // | | | | [2, 4] | | | |
  743. // | +-----------+ +-----------+ +------------+ |
  744. // +------------------------------------------------------+
  745. { "index" : { "_id": "6" } }
  746. { "user": { "name": "elkbee" }, "process": { "name": "attrib" }, ...}
  747. // Creates sequence [6] in state A for "elkbee".
  748. //
  749. // +-----------------------"elkbee"-----------------------+
  750. // | +-----------+ +-----------+ +------------+ |
  751. // | | State A | | State B | | Complete | |
  752. // | +-----------+ +-----------+ +------------+ |
  753. // | | [6] | | | | | |
  754. // | +-----------+ +-----------+ +------------+ |
  755. // +------------------------------------------------------+
  756. { "index" : { "_id": "7" } }
  757. { "user": { "name": "root" }, "process": { "name": "attrib" }, ...}
  758. // Creates sequence [7] in state A for "root".
  759. // Sequence [2, 4] remains in state B for "root".
  760. //
  761. // +------------------------"root"------------------------+
  762. // | +-----------+ +-----------+ +------------+ |
  763. // | | State A | | State B | | Complete | |
  764. // | +-----------+ +-----------+ +------------+ |
  765. // | | [7] | | [2, 4] | | | |
  766. // | +-----------+ +-----------+ +------------+ |
  767. // +------------------------------------------------------+
  768. { "index" : { "_id": "8" } }
  769. { "user": { "name": "elkbee" }, "process": { "name": "bash" }, ...}
  770. // Sequence [6, 8] moves to state B for "elkbee".
  771. // State A for "elkbee" is now empty.
  772. //
  773. // +-----------------------"elkbee"-----------------------+
  774. // | +-----------+ +-----------+ +------------+ |
  775. // | | State A | | State B | | Complete | |
  776. // | +-----------+ --> +-----------+ +------------+ |
  777. // | | | | [6, 8] | | | |
  778. // | +-----------+ +-----------+ +------------+ |
  779. // +------------------------------------------------------+
  780. { "index" : { "_id": "9" } }
  781. { "user": { "name": "root" }, "process": { "name": "cat" }, ...}
  782. // Sequence [2, 4, 9] is complete for "root".
  783. // State B for "root" is now empty.
  784. // Sequence [7] remains in state A.
  785. //
  786. // +------------------------"root"------------------------+
  787. // | +-----------+ +-----------+ +------------+ |
  788. // | | State A | | State B | | Complete | |
  789. // | +-----------+ +-----------+ --> +------------+ |
  790. // | | [7] | | | | [2, 4, 9] |
  791. // | +-----------+ +-----------+ +------------+ |
  792. // +------------------------------------------------------+
  793. { "index" : { "_id": "10" } }
  794. { "user": { "name": "elkbee" }, "process": { "name": "cat" }, ...}
  795. // Sequence [6, 8, 10] is complete for "elkbee".
  796. // State A and B for "elkbee" are now empty.
  797. //
  798. // +-----------------------"elkbee"-----------------------+
  799. // | +-----------+ +-----------+ +------------+ |
  800. // | | State A | | State B | | Complete | |
  801. // | +-----------+ +-----------+ --> +------------+ |
  802. // | | | | | | [6, 8, 10] |
  803. // | +-----------+ +-----------+ +------------+ |
  804. // +------------------------------------------------------+
  805. { "index" : { "_id": "11" } }
  806. { "user": { "name": "root" }, "process": { "name": "cat" }, ...}
  807. // Nothing happens.
  808. // The machines for "root" and "elkbee" remain the same.
  809. //
  810. // +------------------------"root"------------------------+
  811. // | +-----------+ +-----------+ +------------+ |
  812. // | | State A | | State B | | Complete | |
  813. // | +-----------+ +-----------+ +------------+ |
  814. // | | [7] | | | | [2, 4, 9] |
  815. // | +-----------+ +-----------+ +------------+ |
  816. // +------------------------------------------------------+
  817. //
  818. // +-----------------------"elkbee"-----------------------+
  819. // | +-----------+ +-----------+ +------------+ |
  820. // | | State A | | State B | | Complete | |
  821. // | +-----------+ +-----------+ +------------+ |
  822. // | | | | | | [6, 8, 10] |
  823. // | +-----------+ +-----------+ +------------+ |
  824. // +------------------------------------------------------+
  825. ----
  826. ====