elasticsearch/docs/reference/eql/eql.asciidoc

[role="xpack"]
[testenv="basic"]
[[eql]]
= EQL search
++++
<titleabbrev>EQL</titleabbrev>
++++

beta::[]

Event Query Language (EQL) is a query language for event-based time series
data, such as logs, metrics, and traces.

[discrete]
[[eql-advantages]]
== Advantages of EQL

* *EQL lets you express relationships between events.* +
Many query languages allow you to match single events. EQL lets you match a
sequence of events across different event categories and time spans.

* *EQL has a low learning curve.* +
<<eql-syntax,EQL syntax>> looks like other common query languages, such as SQL.
EQL lets you write and read queries intuitively, which makes for quick,
iterative searching.

* *EQL is designed for security use cases.* +
While you can use it for any event-based data, we created EQL for threat
hunting. EQL not only supports indicator of compromise (IOC) searches but can
describe activity that goes beyond IOCs.

[discrete]
[[eql-required-fields]]
== Required fields

To run an EQL search, the searched data stream or index must contain a
_timestamp_ and _event category_ field. By default, EQL uses the `@timestamp`
and `event.category` fields from the {ecs-ref}[Elastic Common Schema
(ECS)]. To use a different timestamp or event category field, see
<<specify-a-timestamp-or-event-category-field>>.

TIP: While no schema is required to use EQL, we recommend using the
{ecs-ref}[ECS]. EQL searches are designed to work with core ECS fields by
default.

[discrete]
[[run-an-eql-search]]
== Run an EQL search

Use the <<eql-search-api,EQL search API>> to run a <<eql-basic-syntax,basic
EQL query>>:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    process where process.name == "regsvr32.exe"
  """
}
----
// TEST[setup:sec_logs]

By default, basic EQL queries return the top 10 matching events in the
`hits.events` property. These hits are sorted by timestamp, converted to
milliseconds since the {wikipedia}/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "my-index-000001",
        "_id": "OQmfCaduce8zoHT93o4H",
        "_source": {
          "@timestamp": "2099-12-07T11:07:09.000Z",
          "event": {
            "category": "process",
            "id": "aR3NWVOs",
            "sequence": 4
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      },
      {
        "_index": "my-index-000001",
        "_id": "xLkCaj4EujzdNSxfYLbO",
        "_source": {
          "@timestamp": "2099-12-07T11:07:10.000Z",
          "event": {
            "category": "process",
            "id": "GTSmSqgz0U",
            "sequence": 6,
            "type": "termination"
          },
          "process": {
            "pid": 2012,
            "name": "regsvr32.exe",
            "executable": "C:\\Windows\\System32\\regsvr32.exe"
          }
        }
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.events.0._id/]
// TESTRESPONSE[s/"_id": "xLkCaj4EujzdNSxfYLbO"/"_id": $body.hits.events.1._id/]

Use the `size` parameter to get a smaller or larger set of hits:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    process where process.name == "regsvr32.exe"
  """,
  "size": 50
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-sequence]]
=== Search for a sequence of events

Use EQL's <<eql-sequences,sequence syntax>> to search for a series of
ordered events. List the event items in ascending chronological order, 
with the most recent event listed last:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    sequence
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

Matching sequences are returned in the `hits.sequences` property.

[source,console-result]
----
{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "my-index-000001",
            "_id": "OQmfCaduce8zoHT93o4H",
            "_source": {
              "@timestamp": "2099-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 4
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_source": {
              "@timestamp": "2099-12-07T11:07:10.000Z",
              "event": {
                "category": "file",
                "id": "tZ1NWVOs",
                "sequence": 5
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              },
              "file": {
                "path": "C:\\Windows\\System32\\scrobj.dll",
                "name": "scrobj.dll"
              }
            }
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]

Use the <<eql-with-maxspan-keywords,`with maxspan` keywords>> to constrain
matching sequences to a timespan:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

Use the <<eql-by-keyword,`by` keyword>> to match events that share the
same field values:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    sequence with maxspan=1h
      [ process where process.name == "regsvr32.exe" ] by process.pid
      [ file where stringContains(file.name, "scrobj.dll") ] by process.pid
  """
}
----
// TEST[setup:sec_logs]

If a field value should be shared across all events, use the `sequence by`
keyword. The following query is equivalent to the previous one.

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
  """
}
----
// TEST[setup:sec_logs]

The `hits.sequences.join_keys` property contains the shared field values.

[source,console-result]
----
{
  "is_partial": false,
  "is_running": false,
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          2012
        ],
        "events": [
          {
            "_index": "my-index-000001",
            "_id": "OQmfCaduce8zoHT93o4H",
            "_source": {
              "@timestamp": "2099-12-07T11:07:09.000Z",
              "event": {
                "category": "process",
                "id": "aR3NWVOs",
                "sequence": 4
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "command_line": "regsvr32.exe  /s /u /i:https://...RegSvr32.sct scrobj.dll",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              }
            }
          },
          {
            "_index": "my-index-000001",
            "_id": "yDwnGIJouOYGBzP0ZE9n",
            "_source": {
              "@timestamp": "2099-12-07T11:07:10.000Z",
              "event": {
                "category": "file",
                "id": "tZ1NWVOs",
                "sequence": 5
              },
              "process": {
                "pid": 2012,
                "name": "regsvr32.exe",
                "executable": "C:\\Windows\\System32\\regsvr32.exe"
              },
              "file": {
                "path": "C:\\Windows\\System32\\scrobj.dll",
                "name": "scrobj.dll"
              }
            }
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
// TESTRESPONSE[s/"_id": "OQmfCaduce8zoHT93o4H"/"_id": $body.hits.sequences.0.events.0._id/]
// TESTRESPONSE[s/"_id": "yDwnGIJouOYGBzP0ZE9n"/"_id": $body.hits.sequences.0.events.1._id/]

Use the <<eql-until-keyword,`until` keyword>> to specify an expiration
event for sequences. Matching sequences must end before this event.

[source,console]
----
GET /my-index-000001/_eql/search
{
  "query": """
    sequence by process.pid with maxspan=1h
      [ process where process.name == "regsvr32.exe" ]
      [ file where stringContains(file.name, "scrobj.dll") ]
    until [ process where event.type == "termination" ]
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[specify-a-timestamp-or-event-category-field]]
=== Specify a timestamp or event category field

The EQL search API uses the `@timestamp` and `event.category` fields from the
{ecs-ref}[ECS] by default. To specify different fields, use the
`timestamp_field` and `event_category_field` parameters:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "timestamp_field": "file.accessed",
  "event_category_field": "file.type",
  "query": """
    file where (file.size > 1 and file.type == "file")
  """
}
----
// TEST[setup:sec_logs]

The event category field must be mapped as a <<keyword,`keyword`>> family field
type. The timestamp field should be mapped as a <<date,`date`>> field type.
<<date_nanos,`date_nanos`>> timestamp fields are not supported. You cannot use a
<<nested,`nested`>> field or the sub-fields of a `nested` field as the timestamp
or event category field.

[discrete]
[[eql-search-specify-a-sort-tiebreaker]]
=== Specify a sort tiebreaker

By default, the EQL search API returns matching events by timestamp. If two or
more events share the same timestamp, {es} uses a tiebreaker field value to sort
the events in ascending, lexicographic order.

`event.sequence` is the default tiebreaker field. To specify another tiebreaker
field, use the `tiebreaker_field` parameter:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "tiebreaker_field": "event.id",
  "query": """
    process where process.name == "cmd.exe" and stringContains(process.executable, "System32")
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-filter-query-dsl]]
=== Filter using query DSL

The `filter` parameter uses <<query-dsl,query DSL>> to limit the documents on
which an EQL query runs.

[source,console]
----
GET /my-index-000001/_eql/search
{
  "filter": {
    "range" : {
      "file.size" : {
        "gte" : 1,
        "lte" : 1000000
      }
    }
  },
  "query": """
    file where (file.type == "file" and file.name == "cmd.exe")
  """
}
----
// TEST[setup:sec_logs]

[discrete]
[[eql-search-async]]
=== Run an async EQL search

By default, EQL search requests are synchronous and wait for complete results
before returning a response. However, complete results can take longer for
searches across <<frozen-indices,frozen indices>> or
<<modules-cross-cluster-search,multiple clusters>>.

To avoid long waits, run an async EQL search. Set the
`wait_for_completion_timeout` parameter to a duration you'd like to wait for
synchronous results.

[source,console]
----
GET /frozen-my-index-000001/_eql/search
{
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]
// TEST[s/frozen-my-index-000001/my-index-000001/]

If the request doesn't finish within the timeout period, the search becomes async
and returns a response that includes:

* A search ID
* An `is_partial` value of `true`, indicating the search results are
  incomplete
* An `is_running` value of `true`, indicating the search is ongoing

The async search continues to run in the background without blocking other
requests.

[source,console-result]
----
{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": true,
  "is_running": true,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"is_partial": true/"is_partial": $body.is_partial/]
// TESTRESPONSE[s/"is_running": true/"is_running": $body.is_running/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

To check the progress of an async search, use the <<get-async-eql-search-api,get
async EQL search API>> with the search ID. Specify how long you'd like for
complete results in the `wait_for_completion_timeout` parameter.

[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?wait_for_completion_timeout=2s
----
// TEST[skip: no access to search ID]

If the response's `is_running` value is `false`, the async search has finished.
If the `is_partial` value is `false`, the returned search results are
complete.

[source,console-result]
----
{
  "id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
  "is_partial": false,
  "is_running": false,
  "took": 2000,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=/$body.id/]
// TESTRESPONSE[s/"took": 2000/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

[discrete]
[[eql-search-store-async-eql-search]]
=== Change the search retention period

By default, the EQL search API stores async searches for five days. After this
period, any searches and their results are deleted. Use the `keep_alive`
parameter to change this retention period:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "keep_alive": "2d",
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]

You can use the <<get-async-eql-search-api,get async EQL search API>>'s
`keep_alive` parameter to later change the retention period. The new retention
period starts after the get request runs.

[source,console]
----
GET /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]

Use the <<delete-async-eql-search-api,delete async EQL search API>> to
manually delete an async EQL search before the `keep_alive` period ends. If the
search is still ongoing, {es} cancels the search request.

[source,console]
----
DELETE /_eql/search/FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=?keep_alive=5d
----
// TEST[skip: no access to search ID]

[discrete]
[[eql-search-store-sync-eql-search]]
=== Store synchronous EQL searches

By default, the EQL search API only stores async searches that cannot be
completed within the period set by `wait_for_completion_timeout`.

To save the results for a synchronous search, set `keep_on_completion` to
`true`:

[source,console]
----
GET /my-index-000001/_eql/search
{
  "keep_on_completion": true,
  "wait_for_completion_timeout": "2s",
  "query": """
    process where process.name == "cmd.exe"
  """
}
----
// TEST[setup:sec_logs]

The response includes a search ID. `is_partial` and `is_running` are `false`,
indicating the EQL search was synchronous and returned complete results.

[source,console-result]
----
{
  "id": "FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=",
  "is_partial": false,
  "is_running": false,
  "took": 52,
  "timed_out": false,
  "hits": ...
}
----
// TESTRESPONSE[s/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=/$body.id/]
// TESTRESPONSE[s/"took": 52/"took": $body.took/]
// TESTRESPONSE[s/"hits": \.\.\./"hits": $body.hits/]

Use the <<get-async-eql-search-api,get async EQL search API>> to get the
same results later:

[source,console]
----
GET /_eql/search/FjlmbndxNmJjU0RPdExBTGg0elNOOEEaQk9xSjJBQzBRMldZa1VVQ2pPa01YUToxMDY=
----
// TEST[skip: no access to search ID]

Saved synchronous searches are still subject to the `keep_alive` parameter's
retention period. When this period ends, the search and its results are deleted.

You can also manually delete saved synchronous searches using the
<<delete-async-eql-search-api,delete async EQL search API>>.

include::syntax.asciidoc[]
include::functions.asciidoc[]
include::pipes.asciidoc[]
include::detect-threats-with-eql.asciidoc[]