analyzer.asciidoc 3.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107
  1. [[analyzer]]
  2. === `analyzer`
  3. [IMPORTANT]
  4. ====
  5. Only <<text,`text`>> fields support the `analyzer` mapping parameter.
  6. ====
  7. The `analyzer` parameter specifies the <<analyzer-anatomy,analyzer>> used for
  8. <<analysis,text analysis>> when indexing or searching a `text` field.
  9. Unless overridden with the <<search-analyzer,`search_analyzer`>> mapping
  10. parameter, this analyzer is used for both <<analysis-index-search-time,index and
  11. search analysis>>. See <<specify-analyzer>>.
  12. [TIP]
  13. ====
  14. We recommend testing analyzers before using them in production. See
  15. <<test-analyzer>>.
  16. ====
  17. [[search-quote-analyzer]]
  18. ==== `search_quote_analyzer`
  19. The `search_quote_analyzer` setting allows you to specify an analyzer for phrases, this is particularly useful when dealing with disabling
  20. stop words for phrase queries.
  21. To disable stop words for phrases a field utilising three analyzer settings will be required:
  22. 1. An `analyzer` setting for indexing all terms including stop words
  23. 2. A `search_analyzer` setting for non-phrase queries that will remove stop words
  24. 3. A `search_quote_analyzer` setting for phrase queries that will not remove stop words
  25. [source,console]
  26. --------------------------------------------------
  27. PUT my_index
  28. {
  29. "settings":{
  30. "analysis":{
  31. "analyzer":{
  32. "my_analyzer":{ <1>
  33. "type":"custom",
  34. "tokenizer":"standard",
  35. "filter":[
  36. "lowercase"
  37. ]
  38. },
  39. "my_stop_analyzer":{ <2>
  40. "type":"custom",
  41. "tokenizer":"standard",
  42. "filter":[
  43. "lowercase",
  44. "english_stop"
  45. ]
  46. }
  47. },
  48. "filter":{
  49. "english_stop":{
  50. "type":"stop",
  51. "stopwords":"_english_"
  52. }
  53. }
  54. }
  55. },
  56. "mappings":{
  57. "properties":{
  58. "title": {
  59. "type":"text",
  60. "analyzer":"my_analyzer", <3>
  61. "search_analyzer":"my_stop_analyzer", <4>
  62. "search_quote_analyzer":"my_analyzer" <5>
  63. }
  64. }
  65. }
  66. }
  67. PUT my_index/_doc/1
  68. {
  69. "title":"The Quick Brown Fox"
  70. }
  71. PUT my_index/_doc/2
  72. {
  73. "title":"A Quick Brown Fox"
  74. }
  75. GET my_index/_search
  76. {
  77. "query":{
  78. "query_string":{
  79. "query":"\"the quick brown fox\"" <6>
  80. }
  81. }
  82. }
  83. --------------------------------------------------
  84. <1> `my_analyzer` analyzer which tokens all terms including stop words
  85. <2> `my_stop_analyzer` analyzer which removes stop words
  86. <3> `analyzer` setting that points to the `my_analyzer` analyzer which will be used at index time
  87. <4> `search_analyzer` setting that points to the `my_stop_analyzer` and removes stop words for non-phrase queries
  88. <5> `search_quote_analyzer` setting that points to the `my_analyzer` analyzer and ensures that stop words are not removed from phrase queries
  89. <6> Since the query is wrapped in quotes it is detected as a phrase query therefore the `search_quote_analyzer` kicks in and ensures the stop words
  90. are not removed from the query. The `my_analyzer` analyzer will then return the following tokens [`the`, `quick`, `brown`, `fox`] which will match one
  91. of the documents. Meanwhile term queries will be analyzed with the `my_stop_analyzer` analyzer which will filter out stop words. So a search for either
  92. `The quick brown fox` or `A quick brown fox` will return both documents since both documents contain the following tokens [`quick`, `brown`, `fox`].
  93. Without the `search_quote_analyzer` it would not be possible to do exact matches for phrase queries as the stop words from phrase queries would be
  94. removed resulting in both documents matching.