elasticsearch/docs/reference/esql/esql-enrich-data.asciidoc

[[esql-enrich-data]]
=== Data enrichment

++++
<titleabbrev>Data enrichment</titleabbrev>
++++

The {esql} <<esql-enrich>> processing command combines, at query-time, data from
one or more source indexes with field-value combinations found in {es} enrich
indexes.

For example, you can use `ENRICH` to:

* Identify web services or vendors based on known IP addresses
* Add product information to retail orders based on product IDs
* Supplement contact information based on an email address

<<esql-enrich>> is similar to <<esql-lookup-join,LOOKUP join>> in the fact that they both help you join data together. You should use `ENRICH` when:

* Enrichment data doesn't change frequently
* You can accept index-time overhead
* You can accept having multiple matches combined into multi-values
* You can accept being limited to predefined match fields
* You do not need fine-grained security: There are no restrictions to specific enrich policies or document and field level security.
* You want to match using ranges or spatial relations

[discrete]
[[esql-how-enrich-works]]
==== How the `ENRICH` command works

The `ENRICH` command adds new columns to a table, with data from {es} indices.
It requires a few special components:

image::images/esql/esql-enrich.png[align="center"]


[[esql-enrich-policy]]
Enrich policy::
+
--
A set of configuration options used to add the right enrich data to the input
table.

An enrich policy contains:

include::../ingest/enrich.asciidoc[tag=enrich-policy-fields]

After <<esql-create-enrich-policy,creating a policy>>, it must be
<<esql-execute-enrich-policy,executed>> before it can be used. Executing an
enrich policy uses data from the policy's source indices to create a streamlined
system index called the _enrich index_. The `ENRICH` command uses this index to
match and enrich an input table.
--

[[esql-source-index]]
Source index::
An index which stores enrich data that the `ENRICH` command can add to input
tables. You can create and manage these indices just like a regular {es} index.
You can use multiple source indices in an enrich policy. You also can use the
same source index in multiple enrich policies.

[[esql-enrich-index]]
Enrich index::
+
--
A special system index tied to a specific enrich policy.

Directly matching rows from input tables to documents in source indices could be
slow and resource intensive. To speed things up, the `ENRICH` command uses an
enrich index.

include::../ingest/enrich.asciidoc[tag=enrich-index]
--

[discrete]
[[esql-set-up-enrich-policy]]
==== Set up an enrich policy

To start using `ENRICH`, follow these steps:

. Check the <<enrich-prereqs, prerequisites>>.
. <<esql-create-enrich-source-index>>.
. <<esql-create-enrich-policy>>.
. <<esql-execute-enrich-policy>>.
. <<esql-use-enrich>>

Once you have enrich policies set up, you can <<esql-update-enrich-data,update
your enrich data>> and <<esql-update-enrich-policies, update your enrich
policies>>.

[discrete]
[IMPORTANT]
====
The `ENRICH` command performs several operations and may impact the speed of
your query.
[discrete]
====

[discrete]
[[esql-enrich-prereqs]]
==== Prerequisites

include::{es-ref-dir}/ingest/apis/enrich/put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]

[discrete]
[[esql-create-enrich-source-index]]
==== Add enrich data

include::../ingest/enrich.asciidoc[tag=create-enrich-source-index]

[discrete]
[[esql-create-enrich-policy]]
==== Create an enrich policy

include::../ingest/enrich.asciidoc[tag=create-enrich-policy]

[discrete]
[[esql-execute-enrich-policy]]
==== Execute the enrich policy

include::../ingest/enrich.asciidoc[tag=execute-enrich-policy1]

image::images/esql/esql-enrich-policy.png[align="center"]

include::../ingest/enrich.asciidoc[tag=execute-enrich-policy2]

[discrete]
[[esql-use-enrich]]
==== Use the enrich policy

After the policy has been executed, you can use the <<esql-enrich,`ENRICH`
command>> to enrich your data.

image::images/esql/esql-enrich-command.png[align="center",width=50%]

include::processing-commands/enrich.asciidoc[tag=examples]

[discrete]
[[esql-update-enrich-data]]
==== Update an enrich index

include::{es-ref-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=update-enrich-index]

[discrete]
[[esql-update-enrich-policies]]
==== Update an enrich policy

include::../ingest/enrich.asciidoc[tag=update-enrich-policy]

==== Enrich Policy Types and Limitations
The {esql} `ENRICH` command supports all three enrich policy types:

`geo_match`::
Matches enrich data to incoming documents based on a <<query-dsl-geo-shape-query,`geo_shape` query>>.
For an example, see <<geo-match-enrich-policy-type>>.

`match`::
Matches enrich data to incoming documents based on a <<query-dsl-term-query,`term` query>>.
For an example, see <<match-enrich-policy-type>>.

`range`::
Matches a number, date, or IP address in incoming documents to a range in the
enrich index based on a <<query-dsl-term-query,`term` query>>. For an example,
see <<range-enrich-policy-type>>.

// tag::limitations[]
While all three enrich policy types are supported, there are some limitations to be aware of:

* The `geo_match` enrich policy type only supports the `intersects` spatial relation.
* It is required that the `match_field` in the `ENRICH` command is of the correct type.
For example, if the enrich policy is of type `geo_match`, the `match_field` in the `ENRICH`
command must be of type `geo_point` or `geo_shape`.
Likewise, a `range` enrich policy requires a `match_field` of type `integer`, `long`, `date`, or `ip`,
depending on the type of the range field in the original enrich index.
* However, this constraint is relaxed for `range` policies when the `match_field` is of type `KEYWORD`.
In this case the field values will be parsed during query execution, row by row.
If any value fails to parse, the output values for that row will be set to `null`,
an appropriate warning will be produced and the query will continue to execute.
// end::limitations[]