|
|
@@ -0,0 +1,215 @@
|
|
|
+[[esql-for-search]]
|
|
|
+=== Using {esql} for search
|
|
|
+
|
|
|
+preview::[]
|
|
|
+
|
|
|
+This page provides an overview of how to use {esql} for search use cases.
|
|
|
+
|
|
|
+// [TIP]
|
|
|
+// ====
|
|
|
+// Prefer to get started with a hands-on tutorial? Check out <<esql-search-tutorial>>.
|
|
|
+// ====
|
|
|
+
|
|
|
+The following table summarizes the key search features available in {esql} and when they were introduced.
|
|
|
+
|
|
|
+[cols="1,1,2", options="header"]
|
|
|
+|===
|
|
|
+|Feature |Available since |Description
|
|
|
+
|
|
|
+|<<esql-search-functions,Full text search functions>>
|
|
|
+|8.17
|
|
|
+|Perform basic text searches with <<esql-match, match function>> and <<esql-search-operators,match operator (`:`)>>
|
|
|
+
|
|
|
+|<<esql-for-search-query-string,Query string function>>
|
|
|
+|8.17
|
|
|
+|Execute complex queries with <<esql-qstr,`qstr`>> using Query String syntax
|
|
|
+
|
|
|
+|<<esql-for-search-scoring,Relevance scoring>>
|
|
|
+|8.18/9.0
|
|
|
+|Calculate and sort by relevance with `METADATA _score`
|
|
|
+
|
|
|
+|Enhanced match options
|
|
|
+|8.18/9.0
|
|
|
+|Configure text searches with additional parameters for the `match` function
|
|
|
+
|
|
|
+|<<esql-for-search-kql,Kibana Query Language>>
|
|
|
+|8.18/9.0
|
|
|
+|Use Kibana Query Language with <<esql-kql,`kql`>> function
|
|
|
+
|
|
|
+|<<esql-for-search-semantic,Semantic search>>
|
|
|
+|8.18/9.0
|
|
|
+|Perform semantic searches on `semantic_text` field types
|
|
|
+
|
|
|
+|<<esql-for-search-hybrid,Hybrid search>>
|
|
|
+|8.18/9.0
|
|
|
+|Combine lexical and semantic search approaches with custom weights
|
|
|
+|===
|
|
|
+
|
|
|
+[[esql-filtering-vs-searching]]
|
|
|
+==== Filtering vs. searching
|
|
|
+
|
|
|
+{esql} can be used for both simple filtering and relevance-based searching:
|
|
|
+
|
|
|
+* **Filtering** removes non-matching documents without calculating relevance scores
|
|
|
+* **Searching** both filters documents and ranks them by how well they match the query
|
|
|
+
|
|
|
+Note that filtering is faster than searching, because it doesn't require score calculations.
|
|
|
+
|
|
|
+[[esql-for-search-scoring]]
|
|
|
+===== Relevance scoring
|
|
|
+
|
|
|
+To get the most relevant results first, you need to use `METADATA _score` and sort by score. For example:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM books METADATA _score
|
|
|
+| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
|
|
|
+| SORT _score DESC
|
|
|
+----
|
|
|
+
|
|
|
+[[esql-for-search-how-scoring-works]]
|
|
|
+===== How `_score` works
|
|
|
+
|
|
|
+When working with relevance scoring in ES|QL:
|
|
|
+
|
|
|
+* If you don't include `METADATA _score` in your query, this only performs filtering operations with no relevance calculation.
|
|
|
+* When you include `METADATA _score`, any search function included in `WHERE` conditions contribute to the relevance score. This means that every occurrence of `MATCH`, `QSTR` and `KQL` will affect the score.
|
|
|
+* Filtering operations that are not search functions, like range conditions and exact matches, don't affect the score.
|
|
|
+* Including `METADATA _score` doesn't automatically sort your results by relevance. You must explicitly use `SORT _score DESC` or `SORT _score ASC` to order your results by relevance.
|
|
|
+
|
|
|
+[[esql-for-search-full-text]]
|
|
|
+==== Full text search
|
|
|
+
|
|
|
+[[esql-for-search-match-function-operator]]
|
|
|
+===== Match function and operator
|
|
|
+
|
|
|
+ES|QL offers two syntax options for `match`, which replicate the functionality of <<query-dsl-match-query, `match`>> queries in Query DSL.
|
|
|
+
|
|
|
+Use the compact operator syntax (`:`) for simple text matching with default parameters.
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM logs | WHERE match(message, "connection error")
|
|
|
+----
|
|
|
+
|
|
|
+Use the `match()` function syntax when you need to pass additional parameters:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM products | WHERE match(name, "laptop", { "boost": 2.0 })
|
|
|
+----
|
|
|
+
|
|
|
+These full-text functions address several key limitations that existed for text filtering in {esql}:
|
|
|
+
|
|
|
+* They work directly on multivalued fields, returning results when any value in a multivalued field matches the query
|
|
|
+* They leverage analyzers, ensuring the query is analyzed with the same process as the indexed data (enabling case-insensitive matching, ASCII folding, stopword removal, and synonym support)
|
|
|
+* They are highly performant, using Lucene index structures rather than pattern matching or regular expressions to locate terms in your data
|
|
|
+
|
|
|
+Refer to this blog for more context: https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL].
|
|
|
+
|
|
|
+[TIP]
|
|
|
+====
|
|
|
+See <<match-field-params,Match field parameters>> for more advanced options using match.
|
|
|
+====
|
|
|
+
|
|
|
+[IMPORTANT]
|
|
|
+====
|
|
|
+These queries match documents but don't automatically sort by relevance. To get the most relevant results first, you need to use `METADATA _score` and sort by score. See <<esql-for-search-scoring,Relevance scoring>> for more information.
|
|
|
+====
|
|
|
+
|
|
|
+[[esql-for-search-query-string]]
|
|
|
+===== Query string function (`QSTR`)
|
|
|
+
|
|
|
+The <<esql-qstr,`qstr` function>> provides the same functionality as the Query DSL's `query_string` query. This is for advanced use cases, such as wildcard searches, searches across multiple fields, and more.
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM articles METADATA _score
|
|
|
+| WHERE QSTR("(new york city) OR (big apple)")
|
|
|
+| SORT _score DESC
|
|
|
+| LIMIT 10
|
|
|
+----
|
|
|
+
|
|
|
+For complete details, refer to the <<query-dsl-query-string-query, Query DSL `query_string` docs>>.
|
|
|
+
|
|
|
+[[esql-for-search-kql]]
|
|
|
+===== Kibana Query Language function (`KQL`)
|
|
|
+
|
|
|
+Use the <<esql-kql,KQL function>> to use the {kibana-ref}/kuery-query.html[Kibana Query Language] in your ES|QL queries:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM logs*
|
|
|
+| WHERE KQL("http.request.method:GET AND agent.type:filebeat")
|
|
|
+----
|
|
|
+
|
|
|
+The `kql` function is useful when transitioning queries from Kibana's Discover, Dashboard, or other interfaces that use KQL. This will allow you to gradually migrate queries to ES|QL without needing to rewrite them all at once.
|
|
|
+
|
|
|
+[[esql-for-search-semantic]]
|
|
|
+==== Semantic search
|
|
|
+
|
|
|
+You can perform semantic searches over <<semantic-text, `semantic_text`>> field types using the same match syntax as full-text search.
|
|
|
+
|
|
|
+This example uses the match operator `:`:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM articles METADATA _score
|
|
|
+| WHERE semantic_content:"What are the impacts of climate change on agriculture?"
|
|
|
+| SORT _score DESC
|
|
|
+----
|
|
|
+
|
|
|
+This example uses the match function:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM articles METADATA _score
|
|
|
+| WHERE match(semantic_content, "What are the impacts of climate change on agriculture?")
|
|
|
+| SORT _score DESC
|
|
|
+----
|
|
|
+
|
|
|
+[[esql-for-search-hybrid]]
|
|
|
+==== Hybrid search
|
|
|
+
|
|
|
+Combine traditional and semantic search with custom weights:
|
|
|
+
|
|
|
+[source,esql]
|
|
|
+----
|
|
|
+FROM books METADATA _score
|
|
|
+| WHERE match(semantic_title, "fantasy adventure", { "boost": 0.75 })
|
|
|
+ OR match(title, "fantasy adventure", { "boost": 0.25 })
|
|
|
+| SORT _score DESC
|
|
|
+----
|
|
|
+
|
|
|
+[[esql-for-search-limitations]]
|
|
|
+==== Limitations
|
|
|
+
|
|
|
+Refer to <<esql-limitations-full-text-search, {esql} limitations>> for a list of known limitations.
|
|
|
+
|
|
|
+[[esql-for-search-next-steps]]
|
|
|
+==== Next steps
|
|
|
+
|
|
|
+[[esql-for-search-tutorials]]
|
|
|
+===== Tutorials and how-to guides
|
|
|
+
|
|
|
+// TODO * <<esql-search-tutorial>>: Hands-on tutorial for getting started with search tools in {esql}
|
|
|
+* <<semantic-search-semantic-text>>: Learn how to use the `semantic_text` field type
|
|
|
+
|
|
|
+[[esql-for-search-reference]]
|
|
|
+===== Technical reference
|
|
|
+
|
|
|
+* <<esql-search-functions>>: Complete reference for all search functions
|
|
|
+* <<esql-limitations-full-text-search, Limitations>>: Current limitations for search in ES|QL
|
|
|
+
|
|
|
+[[esql-for-search-concepts]]
|
|
|
+===== Background concepts
|
|
|
+
|
|
|
+* <<analysis>>: Learn how text is processed for full-text search
|
|
|
+* <<semantic-search>>: Get an overview of semantic search in {es}
|
|
|
+* <<query-filter-context>>: Understand the difference between query and filter contexts in {es}
|
|
|
+
|
|
|
+[[esql-for-search-blogs]]
|
|
|
+===== Related blog posts
|
|
|
+
|
|
|
+// TODO* https://www.elastic.co/blog/esql-you-know-for-search-scoring-semantic-search[ES|QL, you know for Search]: Introducing scoring and semantic search
|
|
|
+* https://www.elastic.co/search-labs/blog/filtering-in-esql-full-text-search-match-qstr[Introducing full text filtering in ES|QL]: Overview of text filtering capabilities
|
Liam Thompson