elasticsearch/docs/plugins/analysis-nori.asciidoc

[[analysis-nori]]
=== Korean (nori) analysis plugin

The Korean (nori) Analysis plugin integrates Lucene nori analysis
module into elasticsearch. It uses the https://bitbucket.org/eunjeon/mecab-ko-dic[mecab-ko-dic dictionary]
to perform morphological analysis of Korean texts.

:plugin_name: analysis-nori
include::install_remove.asciidoc[]

[[analysis-nori-analyzer]]
==== `nori` analyzer

The `nori` analyzer consists of the following tokenizer and token filters:

* <<analysis-nori-tokenizer,`nori_tokenizer`>>
* <<analysis-nori-speech,`nori_part_of_speech`>> token filter
* <<analysis-nori-readingform,`nori_readingform`>> token filter
* {ref}/analysis-lowercase-tokenfilter.html[`lowercase`] token filter

It supports the `decompound_mode` and `user_dictionary` settings from
<<analysis-nori-tokenizer,`nori_tokenizer`>> and the `stoptags` setting from
<<analysis-nori-speech,`nori_part_of_speech`>>.

[[analysis-nori-tokenizer]]
==== `nori_tokenizer`

The `nori_tokenizer` accepts the following settings:

`decompound_mode`::
+
--

The decompound mode determines how the tokenizer handles compound tokens.
It can be set to:

`none`::

    No decomposition for compounds. Example output:

    가거도항
    가곡역

`discard`::

    Decomposes compounds and discards the original form (*default*). Example output:

    가곡역 => 가곡, 역

`mixed`::

    Decomposes compounds and keeps the original form. Example output:

    가곡역 => 가곡역, 가곡, 역
--

`discard_punctuation`::

    Whether punctuation should be discarded from the output. Defaults to `true`.

`lenient`::

    Whether the `user_dictionary` should be deduplicated on the provided `text`.
    False by default causing duplicates to generate an error.

`user_dictionary`::
+
--
The Nori tokenizer uses the https://bitbucket.org/eunjeon/mecab-ko-dic[mecab-ko-dic dictionary] by default.
A `user_dictionary` with custom nouns (`NNG`) may be appended to the default dictionary.
The dictionary should have the following format:

[source,txt]
-----------------------
<token> [<token 1> ... <token n>]
-----------------------

The first token is mandatory and represents the custom noun that should be added in
the dictionary. For compound nouns the custom segmentation can be provided
after the first token (`[<token 1> ... <token n>]`). The segmentation of the
custom compound nouns is controlled by the `decompound_mode` setting.


As a demonstration of how the user dictionary can be used, save the following
dictionary to `$ES_HOME/config/userdict_ko.txt`:

[source,txt]
-----------------------
c++                 <1>
C쁠쁠
세종
세종시 세종 시        <2>
-----------------------

<1> A simple noun
<2> A compound noun (`세종시`) followed by its decomposition: `세종` and `시`.

Then create an analyzer as follows:

[source,console]
--------------------------------------------------
PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "tokenizer": {
          "nori_user_dict": {
            "type": "nori_tokenizer",
            "decompound_mode": "mixed",
            "discard_punctuation": "false",
            "user_dictionary": "userdict_ko.txt",
            "lenient": "true"
          }
        },
        "analyzer": {
          "my_analyzer": {
            "type": "custom",
            "tokenizer": "nori_user_dict"
          }
        }
      }
    }
  }
}

GET nori_sample/_analyze
{
  "analyzer": "my_analyzer",
  "text": "세종시"  <1>
}
--------------------------------------------------

<1> Sejong city

The above `analyze` request returns the following:

[source,console-result]
--------------------------------------------------
{
  "tokens" : [ {
    "token" : "세종시",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "word",
    "position" : 0,
    "positionLength" : 2    <1>
  }, {
    "token" : "세종",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "word",
    "position" : 0
  }, {
    "token" : "시",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "word",
    "position" : 1
   }]
}
--------------------------------------------------

<1> This is a compound token that spans two positions (`mixed` mode).
--

`user_dictionary_rules`::
+
--

You can also inline the rules directly in the tokenizer definition using
the `user_dictionary_rules` option:

[source,console]
--------------------------------------------------
PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "tokenizer": {
          "nori_user_dict": {
            "type": "nori_tokenizer",
            "decompound_mode": "mixed",
            "user_dictionary_rules": ["c++", "C쁠쁠", "세종", "세종시 세종 시"]
          }
        },
        "analyzer": {
          "my_analyzer": {
            "type": "custom",
            "tokenizer": "nori_user_dict"
          }
        }
      }
    }
  }
}
--------------------------------------------------
--

The `nori_tokenizer` sets a number of additional attributes per token that are used by token filters
to modify the stream.
You can view all these additional attributes with the following request:

[source,console]
--------------------------------------------------
GET _analyze
{
  "tokenizer": "nori_tokenizer",
  "text": "뿌리가 깊은 나무는",   <1>
  "attributes" : ["posType", "leftPOS", "rightPOS", "morphemes", "reading"],
  "explain": true
}
--------------------------------------------------

<1> A tree with deep roots

Which responds with:

[source,console-result]
--------------------------------------------------
{
  "detail": {
    "custom_analyzer": true,
    "charfilters": [],
    "tokenizer": {
      "name": "nori_tokenizer",
      "tokens": [
        {
          "token": "뿌리",
          "start_offset": 0,
          "end_offset": 2,
          "type": "word",
          "position": 0,
          "leftPOS": "NNG(General Noun)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "NNG(General Noun)"
        },
        {
          "token": "가",
          "start_offset": 2,
          "end_offset": 3,
          "type": "word",
          "position": 1,
          "leftPOS": "J(Ending Particle)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "J(Ending Particle)"
        },
        {
          "token": "깊",
          "start_offset": 4,
          "end_offset": 5,
          "type": "word",
          "position": 2,
          "leftPOS": "VA(Adjective)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "VA(Adjective)"
        },
        {
          "token": "은",
          "start_offset": 5,
          "end_offset": 6,
          "type": "word",
          "position": 3,
          "leftPOS": "E(Verbal endings)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "E(Verbal endings)"
        },
        {
          "token": "나무",
          "start_offset": 7,
          "end_offset": 9,
          "type": "word",
          "position": 4,
          "leftPOS": "NNG(General Noun)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "NNG(General Noun)"
        },
        {
          "token": "는",
          "start_offset": 9,
          "end_offset": 10,
          "type": "word",
          "position": 5,
          "leftPOS": "J(Ending Particle)",
          "morphemes": null,
          "posType": "MORPHEME",
          "reading": null,
          "rightPOS": "J(Ending Particle)"
        }
      ]
    },
    "tokenfilters": []
  }
}
--------------------------------------------------

[[analysis-nori-speech]]
==== `nori_part_of_speech` token filter

The `nori_part_of_speech` token filter removes tokens that match a set of
part-of-speech tags. The list of supported tags and their meanings can be found here:
{lucene-core-javadoc}/../analysis/nori/org/apache/lucene/analysis/ko/POS.Tag.html[Part of speech tags]

It accepts the following setting:

`stoptags`::

    An array of part-of-speech tags that should be removed.

and defaults to:

[source,js]
--------------------------------------------------
"stoptags": [
    "E",
    "IC",
    "J",
    "MAG", "MAJ", "MM",
    "SP", "SSC", "SSO", "SC", "SE",
    "XPN", "XSA", "XSN", "XSV",
    "UNA", "NA", "VSV"
]
--------------------------------------------------
// NOTCONSOLE

For example:

[source,console]
--------------------------------------------------
PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "nori_tokenizer",
            "filter": [
              "my_posfilter"
            ]
          }
        },
        "filter": {
          "my_posfilter": {
            "type": "nori_part_of_speech",
            "stoptags": [
              "NR"   <1>
            ]
          }
        }
      }
    }
  }
}

GET nori_sample/_analyze
{
  "analyzer": "my_analyzer",
  "text": "여섯 용이"  <2>
}
--------------------------------------------------

<1> Korean numerals should be removed (`NR`)
<2> Six dragons

Which responds with:

[source,console-result]
--------------------------------------------------
{
  "tokens" : [ {
    "token" : "용",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "word",
    "position" : 1
  }, {
    "token" : "이",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "word",
    "position" : 2
  } ]
}
--------------------------------------------------


[[analysis-nori-readingform]]
==== `nori_readingform` token filter

The `nori_readingform` token filter rewrites tokens written in Hanja to their Hangul form.

[source,console]
--------------------------------------------------
PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "nori_tokenizer",
            "filter": [ "nori_readingform" ]
          }
        }
      }
    }
  }
}

GET nori_sample/_analyze
{
  "analyzer": "my_analyzer",
  "text": "鄕歌"      <1>
}
--------------------------------------------------

<1> A token written in Hanja: Hyangga

Which responds with:

[source,console-result]
--------------------------------------------------
{
  "tokens" : [ {
    "token" : "향가",     <1>
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "word",
    "position" : 0
  }]
}
--------------------------------------------------

<1> The Hanja form is replaced by the Hangul translation.


[[analysis-nori-number]]
==== `nori_number` token filter

The `nori_number` token filter normalizes Korean numbers
to regular Arabic decimal numbers in half-width characters.

Korean numbers are often written using a combination of Hangul and Arabic numbers with various kinds of punctuation.
For example, ３．２천 means 3200.
This filter does this kind of normalization and allows a search for 3200 to match ３．２천 in text,
but can also be used to make range facets based on the normalized numbers and so on.

[NOTE]
====
Notice that this analyzer uses a token composition scheme and relies on punctuation tokens
being found in the token stream.
Please make sure your `nori_tokenizer` has `discard_punctuation` set to false.
In case punctuation characters, such as U+FF0E(．), is removed from the token stream,
this filter would find input tokens ３ and ２천 and give outputs 3 and 2000 instead of 3200,
which is likely not the intended result.

If you want to remove punctuation characters from your index that are not part of normalized numbers,
add a `stop` token filter with the punctuation you wish to remove after `nori_number` in your analyzer chain.
====
Below are some examples of normalizations this filter supports.
The input is untokenized text and the result is the single term attribute emitted for the input.

- 영영칠 -> 7
- 일영영영 -> 1000
- 삼천2백2십삼 -> 3223
- 조육백만오천일 -> 1000006005001
- ３.２천 ->  3200
- １.２만３４５.６７ -> 12345.67
- 4,647.100 -> 4647.1
- 15,7 -> 157 (be aware of this weakness)

For example:

[source,console]
--------------------------------------------------
PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "tokenizer_discard_puncuation_false",
            "filter": [
              "part_of_speech_stop_sp", "nori_number"
            ]
          }
        },
        "tokenizer": {
          "tokenizer_discard_puncuation_false": {
            "type": "nori_tokenizer",
            "discard_punctuation": "false"
          }
        },
        "filter": {
            "part_of_speech_stop_sp": {
                "type": "nori_part_of_speech",
                "stoptags": ["SP"]
            }
        }
      }
    }
  }
}

GET nori_sample/_analyze
{
  "analyzer": "my_analyzer",
  "text": "십만이천오백과 ３.２천"
}
--------------------------------------------------

Which results in:

[source,console-result]
--------------------------------------------------
{
  "tokens" : [{
    "token" : "102500",
    "start_offset" : 0,
    "end_offset" : 6,
    "type" : "word",
    "position" : 0
  }, {
    "token" : "과",
    "start_offset" : 6,
    "end_offset" : 7,
    "type" : "word",
    "position" : 1
  }, {
    "token" : "3200",
    "start_offset" : 8,
    "end_offset" : 12,
    "type" : "word",
    "position" : 2
  }]
}
--------------------------------------------------