thai-tokenizer.asciidoc 290 B

1234567
  1. [[analysis-thai-tokenizer]]
  2. === Thai Tokenizer
  3. A tokenizer of type `thai` that segments Thai text into words. This tokenizer
  4. uses the built-in Thai segmentation algorithm included with Java to divide
  5. up Thai text. Text in other languages in general will be treated the same
  6. as `standard`.