JAPANESE Tokenizing

turutosiya

Hi.

I'm a niewbie at RapidMiner.

I'm trying to mining some webpages with "GetPage", "Extract Content" And "Process Documents".
It seems work well for ENGLIUSH pages, but for JAPANESE pages, tokenizer doesn't work well,

Japanese tokenize is not supported?

land

Hi,
not really and as I'm not an expert on Japanese, I don't have a clue how we should do this, they don't have whitespaces, do they?
How is determined where a word ends?

Greetings,
  Sebastian

el_chief

you will probably want to tokenize using the regular expression mode, with the regular expression matching all characters. this should tokenize the document on every character, which i believe is what you want with japanese and chinese.

you should also try the Text Processing > Transformation > Generate n-Grams (Characters) operator

karlrb

If I can be of any help, I would be happy to look into any specific questions on this subject.  My wife is Japanese and I'm in the process of learning Japanese - amazingly complex.

Karl Bergerson
Seattle WA USA
karl.bergerson@gmail.com

land

Hi Karl,
you are very welcome if you can come up with a good algorithm for japanese tokenization!

With kind regards,
  Sebastian

turutosiya

Hi All.

It's beeeeeen a really long time to start this proj. at last, I have time to try.

I'm looking for document which describing API spec for Tokenizer.
does anyone know?

I'm trying to implement a JapaneseTokenizer which work with morphological analysis engine, such as Chasen / Mecab.