lowercase-tokenizer.asciidoc 577 B

123456789101112131415
  1. [[analysis-lowercase-tokenizer]]
  2. === Lowercase Tokenizer
  3. A tokenizer of type `lowercase` that performs the function of
  4. <<analysis-letter-tokenizer,Letter
  5. Tokenizer>> and
  6. <<analysis-lowercase-tokenfilter,Lower
  7. Case Token Filter>> together. It divides text at non-letters and converts
  8. them to lower case. While it is functionally equivalent to the
  9. combination of
  10. <<analysis-letter-tokenizer,Letter
  11. Tokenizer>> and
  12. <<analysis-lowercase-tokenfilter,Lower
  13. Case Token Filter>>, there is a performance advantage to doing the two
  14. tasks at once, hence this (redundant) implementation.