elasticsearch/docs/reference/eql/eql.asciidoc

[role="xpack"]
[testenv="basic"]
[[eql]]
= EQL search
++++
<titleabbrev>EQL</titleabbrev>
++++

Event Query Language (EQL) is a query language for event-based time series
data, such as logs, metrics, and traces.

[discrete]
[[eql-advantages]]
=== Advantages of EQL

* *EQL lets you express relationships between events.* +
Many query languages allow you to match single events. EQL lets you match a
sequence of events across different event categories and time spans.

* *EQL has a low learning curve.* +
<<eql-syntax,EQL syntax>> looks like other common query languages, such as SQL.
EQL lets you write and read queries intuitively, which makes for quick,
iterative searching.

* *EQL is designed for security use cases.* +
While you can use it for any event-based data, we created EQL for threat
hunting. EQL not only supports indicator of compromise (IOC) searches but can
describe activity that goes beyond IOCs.

[discrete]
[[eql-required-fields]]
=== Required fields

To run an EQL search, the searched data stream or index must contain a
_timestamp_ and _event category_ field. By default, EQL uses the `@timestamp`
and `event.category` fields from the {ecs-ref}[Elastic Common Schema
(ECS)]. To use a different timestamp or event category field, see
<<specify-a-timestamp-or-event-category-field>>.

TIP: While no schema is required to use EQL, we recommend using the
{ecs-ref}[ECS]. EQL searches are designed to work with core ECS fields by
default.

[discrete]
[[run-an-eql-search]]
=== Run an EQL search

Use the <<eql-search-api,EQL search API>> to run a <<eql-basic-syntax,basic EQL
query>>.

////
[source,console]
----
DELETE /_data_stream/*
DELETE /_index_template/*
----
// TEARDOWN
////

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    process where process.name == "regsvr32.exe"
  """
}
----
// TEST[setup:sec_logs]

By default, basic EQL queries return the 10 most recent matching events in the
`hits.events` property. These hits are sorted by timestamp, converted to
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": ".ds-my-data-stream-2099.12.07-000001",
        "_id": "OQmfCaduce8zoHT93o4H",
        "_source": {
          "@timestamp": "2099-12-07T11:07:09.000Z",
          "event": {
            "category": "process",
            "id": "aR3NWVOs",
            "sequence": 4
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      },
      {
        "_index": ".ds-my-data-stream-2099.12.07-000001",
        "_id": "xLkCaj4EujzdNSxfYLbO",
        "_source": {
          "@timestamp": "2099-12-07T11:07:10.000Z",
          "event": {
            "category": "process",
            "id": "GTSmSqgz0U",
            "sequence": 6,
            "type": "termination"
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]

Use the `size` parameter to get a smaller or larger set of hits:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    process where process.name == "regsvr32.exe"
  """,
  "size": 50
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-sequence]]
=== Search for a sequence of events

Use EQL's <<eql-sequences,sequence syntax>> to search for a series of
ordered events. List the event items in ascending chronological order,
with the most recent event listed last:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    sequence
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

The response's `hits.sequences` property contains the 10 most recent matching
sequences.

[source,console-result]
----
{
  ...
  "hits": {
    "total": ...,
    "sequences": [
      {
        "events": [
          {
            "_index": ".ds-my-data-stream-2099.12.07-000001",
            "_id": "OQmfCaduce8zoHT93o4H",
            "_source": {
              "@timestamp": "2099-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 4
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              }
            }
          },
          {
            "_index": ".ds-my-data-stream-2099.12.07-000001",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_source": {
              "@timestamp": "2099-12-07T11:07:10.000Z",
              "event": {
                "category": "file",
                "id": "tZ1NWVOs",
                "sequence": 5
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              },
              "file": {
                "path": "C:\\Windows\\System32\\scrobj.dll",
                "name": "scrobj.dll"
              }
            }
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/  \.\.\.\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/]
// TESTRESPONSE[s/"total": \.\.\.,/"total": { "value": 1, "relation": "eq" },/]
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.sequences.0.events.0._index/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]

Use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to constrain
matching sequences to a timespan:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

Use the <<eql-by-keyword,`by` keyword>> to match events that share the
same field values:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ] by process.pid
      [ file where stringContains(file.name, "scrobj.dll") ] by process.pid
  """
}
----
// TEST[setup:sec_logs]

If a field value should be shared across all events, use the `sequence by`
keyword. The following query is equivalent to the previous one.

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

The `hits.sequences.join_keys` property contains the shared field values.

[source,console-result]
----
{
  ...
  "hits": ...,
    "sequences": [
      {
        "join_keys": [
          2012
        ],
        "events": ...
      }
    ]
  }
}
----
// TESTRESPONSE[s/  \.\.\.\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/]
// TESTRESPONSE[s/"hits": \.\.\.,/"hits": { "total": { "value": 1, "relation": "eq" },/]
// TESTRESPONSE[s/"events": \.\.\./"events": $body.hits.sequences.0.events/]

Use the <<eql-until-keyword,`until` keyword>> to specify an expiration
event for sequences. Matching sequences must end before this event.

[source,console]
----
GET /my-data-stream/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
    until [ process where event.type == "termination" ]
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[retrieve-selected-fields]]
=== Retrieve selected fields

By default, each hit in the search response includes the document `_source`,
which is the entire JSON object that was provided when indexing the document.

You can use the <<common-options-response-filtering,`filter_path`>> query
parameter to filter the API response. For example, the following search returns
only the timestamp and PID from the `_source` of each matching event.

[source,console]
----
GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits.events._source.process.pid
{
  "query": """
    process where process.name == "regsvr32.exe"
  """
}
----
// TEST[setup:sec_logs]

The API returns the following response.

[source,console-result]
----
{
  "hits": {
    "events": [
      {
        "_source": {
          "@timestamp": "2099-12-07T11:07:09.000Z",
          "process": {
            "pid": 2012
          }
        }
      },
      {
        "_source": {
          "@timestamp": "2099-12-07T11:07:10.000Z",
          "process": {
            "pid": 2012
          }
        }
      }
    ]
  }
}
----

You can also use the `fields` parameter to retrieve and format specific fields
in the response. This field is identical to the search API's
<<search-fields,`fields` parameter>>.

include::{es-repo-dir}/search/search-your-data/retrieve-selected-fields.asciidoc[tag=fields-param-desc]

The following search request uses the `fields` parameter to retrieve values for
the `event.type` field, all fields starting with `process.`, and the
`@timestamp` field. The request also uses the `filter_path` query parameter to
exclude the `_source` of each hit.

[source,console]
----
GET /my-data-stream/_eql/search?filter_path=-hits.events._source
{
  "query": """
    process where process.name == "regsvr32.exe"
  """,
  "fields": [
    "event.type",
    "process.*",                <1>
    {
      "field": "@timestamp",
      "format": "epoch_millis"  <2>
    }
  ]
}
----
// TEST[setup:sec_logs]

include::{es-repo-dir}/search/search-your-data/retrieve-selected-fields.asciidoc[tag=fields-param-callouts]

The response includes values as a flat list in the `fields` section for each
hit.

[source,console-result]
----
{
  ...
  "hits": {
    "total": ...,
    "events": [
      {
        "_index": ".ds-my-data-stream-2099.12.07-000001",
        "_id": "OQmfCaduce8zoHT93o4H",
        "fields": {
          "process.name": [
            "regsvr32.exe"
          ],
          "process.name.keyword": [
            "regsvr32.exe"
          ],
          "@timestamp": [
            "4100324829000"
          ],
          "process.command_line": [
            "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll"
          ],
          "process.command_line.keyword": [
            "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll"
          ],
          "process.executable.keyword": [
            "C:\\Windows\\System32\\regsvr32.exe"
          ],
          "process.pid": [
            2012
          ],
          "process.executable": [
            "C:\\Windows\\System32\\regsvr32.exe"
          ]
        }
      },
      ....
    ]
  }
}
----
// TESTRESPONSE[s/  \.\.\.\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/]
// TESTRESPONSE[s/"total": \.\.\.,/"total": { "value": 2, "relation": "eq" },/]
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/      \.\.\.\.\n/$body.hits.events.1/]

[discrete]
[[eql-use-runtime-fields]]
=== Use runtime fields

Use the `runtime_mappings` parameter to extract and create <<runtime,runtime
fields>> during a search. Use the `fields` parameter to include runtime fields
in the response.

The following search creates a `day_of_week` runtime field from the `@timestamp`
and returns it in the response.

[source,console]
----
GET /my-data-stream/_eql/search?filter_path=-hits.events._source
{
  "runtime_mappings": {
    "day_of_week": {
      "type": "keyword",
      "script": "emit(doc['@timestamp'].value.dayOfWeekEnum.toString())"
    }
  },
  "query": """
    process where process.name == "regsvr32.exe"
  """,
  "fields": [
    "@timestamp",
    "day_of_week"
  ]
}
----
// TEST[setup:sec_logs]

The API returns:

[source,console-result]
----
{
  ...
  "hits": {
    "total": ...,
    "events": [
      {
        "_index": ".ds-my-data-stream-2099.12.07-000001",
        "_id": "OQmfCaduce8zoHT93o4H",
        "fields": {
          "@timestamp": [
            "2099-12-07T11:07:09.000Z"
          ],
          "day_of_week": [
            "MONDAY"
          ]
        }
      },
      ....
    ]
  }
}
----
// TESTRESPONSE[s/  \.\.\.\n/"is_partial": false, "is_running": false, "took": $body.took, "timed_out": false,/]
// TESTRESPONSE[s/"total": \.\.\.,/"total": { "value": 2, "relation": "eq" },/]
// TESTRESPONSE[s/"_index": ".ds-my-data-stream-2099.12.07-000001"/"_index": $body.hits.events.0._index/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/      \.\.\.\.\n/$body.hits.events.1/]


[discrete]
[[specify-a-timestamp-or-event-category-field]]
=== Specify a timestamp or event category field

The EQL search API uses the `@timestamp` and `event.category` fields from the
{ecs-ref}[ECS] by default. To specify different fields, use the
`timestamp_field` and `event_category_field` parameters:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "timestamp_field": "file.accessed",
  "event_category_field": "file.type",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}
----
// TEST[setup:sec_logs]

The event category field must be mapped as a <<keyword,`keyword`>> family field
type. The timestamp field should be mapped as a <<date,`date`>> field type.
<<date_nanos,`date_nanos`>> timestamp fields are not supported. You cannot use a
<<nested,`nested`>> field or the sub-fields of a `nested` field as the timestamp
or event category field.

[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker

By default, the EQL search API returns matching hits by timestamp. If two or
more events share the same timestamp, {es} uses a tiebreaker field value to sort
the events in ascending order. {es} orders events with no
tiebreaker value after events with a value.

If you don't specify a tiebreaker field or the events also share the same
tiebreaker value, {es} considers the events concurrent and may
not return them in a consistent sort order.

To specify a tiebreaker field, use the `tiebreaker_field` parameter. If you use
the {ecs-ref}[ECS], we recommend using `event.sequence` as the tiebreaker field.

[source,console]
----
GET /my-data-stream/_eql/search
{
  "tiebreaker_field": "event.sequence",
  "query": """
    process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using Query DSL

The `filter` parameter uses <<query-dsl,Query DSL>> to limit the documents on
which an EQL query runs.

[source,console]
----
GET /my-data-stream/_eql/search
{
  "filter": {
    "range": {
      "@timestamp": {
        "gte": "now-1d/d",
        "lt": "now/d"
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-async]]
=== Run an async EQL search

By default, EQL search requests are synchronous and wait for complete results
before returning a response. However, complete results can take longer for
searches across large data sets or <<data-tiers,frozen>> data.

To avoid long waits, run an async EQL search. Set `wait_for_completion_timeout`
to a duration you'd like to wait for synchronous results.

[source,console]
----
GET /my-data-stream/_eql/search
{
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]

If the request doesn't finish within the timeout period, the search becomes async
and returns a response that includes:

* A search ID
* An `is_partial` value of `true`, indicating the search results are
  incomplete
* An `is_running` value of `true`, indicating the search is ongoing

The async search continues to run in the background without blocking other
requests.

[source,console-result]
----
{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": true,
  "is_running": true,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
// TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

To check the progress of an async search, use the <<get-async-eql-search-api,get
async EQL search API>> with the search ID. Specify how long you'd like for
complete results in the `wait_for_completion_timeout` parameter.
[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
----
// TEST[skip: no access to search ID]

If the response's `is_running` value is `false`, the async search has finished.
If the `is_partial` value is `false`, the returned search results are
complete.

[source,console-result]
----
{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": false,
  "is_running": false,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

Another more lightweight way to check the progress of an async search is to use
the <<get-async-eql-status-api,get async EQL status API>> with the search ID.

[source,console]
----
GET /_eql/search/status/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=
----
// TEST[skip: no access to search ID]

[source,console-result]
----
{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_running": false,
  "is_partial": false,
  "expiration_time_in_millis": 1611690295000,
  "completion_status": 200
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"expiration_time_in_millis": 1611690295000/"expiration_time_in_millis": $body.expiration_time_in_millis/]

[discrete]
[[eql-search-store-async-eql-search]]
=== Change the search retention period

By default, the EQL search API stores async searches for five days. After this
period, any searches and their results are deleted. Use the `keep_alive`
parameter to change this retention period:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "keep_alive": "2d",
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]

You can use the <<get-async-eql-search-api,get async EQL search API>>'s
`keep_alive` parameter to later change the retention period. The new retention
period starts after the get request runs.

[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]

Use the <<delete-async-eql-search-api,delete async EQL search API>> to
manually delete an async EQL search before the `keep_alive` period ends. If the
search is still ongoing, {es} cancels the search request.

[source,console]
----
DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]

[discrete]
[[eql-search-store-sync-eql-search]]
=== Store synchronous EQL searches

By default, the EQL search API only stores async searches. To save a synchronous
search, set `keep_on_completion` to `true`:

[source,console]
----
GET /my-data-stream/_eql/search
{
  "keep_on_completion": true,
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]

The response includes a search ID. `is_partial` and `is_running` are `false`,
indicating the EQL search was synchronous and returned complete results.

[source,console-result]
----
{
  "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  "is_partial": false,
  "is_running": false,
  "took": 52,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
// TESTRESPONSE[s/"took": 52/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

Use the <<get-async-eql-search-api,get async EQL search API>> to get the
same results later:

[source,console]
----
GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
----
// TEST[skip: no access to search ID]

Saved synchronous searches are still subject to the `keep_alive` parameter's
retention period. When this period ends, the search and its results are deleted.

You can also check only the status of the saved synchronous search without
results by using <<get-async-eql-status-api,get async EQL status API>>.

You can also manually delete saved synchronous searches using the
<<delete-async-eql-search-api,delete async EQL search API>>.

include::syntax.asciidoc[]
include::functions.asciidoc[]
include::pipes.asciidoc[]
include::detect-threats-with-eql.asciidoc[]