123456789101112131415 |
- [[analysis-lowercase-tokenizer]]
- === Lowercase Tokenizer
- A tokenizer of type `lowercase` that performs the function of
- <<analysis-letter-tokenizer,Letter
- Tokenizer>> and
- <<analysis-lowercase-tokenfilter,Lower
- Case Token Filter>> together. It divides text at non-letters and converts
- them to lower case. While it is functionally equivalent to the
- combination of
- <<analysis-letter-tokenizer,Letter
- Tokenizer>> and
- <<analysis-lowercase-tokenfilter,Lower
- Case Token Filter>>, there is a performance advantage to doing the two
- tasks at once, hence this (redundant) implementation.
|