Text Mining- Select Token based on Dictionary File

mbjasser

Hi every one,

I'm trying to work on a text mining workflow to filter specific language contents based on  a specific language dictionary (TXT file most probably).

I was able to filter stopwords using the operator "Filter Stopwords (Dictionary)" to filter the content depending on a dictionary, but I'm still trying to select tokens based also on a dictionary, but it seems that the only operator offered is Filter Tokens (by contents) (which enables selecting tokens based on a regular expression, there is no option for selecting tokens based on a dictionary file).

I need your support if you have an idea if there exists any operator to do that, or if I'm missing something.

Thank you in advance

JEdward

Seconded. 

See my post here: http://rapid-i.com/rapidforum/index.php/topic,8008.msg27328.html#msg27328 suggesting that a group of us band together to pay for RM to improve language support in the text extension.  Particularly for more difficult languages such as Arabic, Chinese, Indonesian or even TxtSpk. 

fras

There is a operator "Filter Stopwords" where you can place a
txt file with one stopword per line.

mbjasser

Thanks fras, I'm already using the operator of Filter Stop words based on dictionary, but this operator is filtering words, I want to select tokens based on dictionary not filtering (removing). I'm not able to find any up to now. The only option offered is filter tokens by content using the regular option field.

mbjasser

it's god idea JEdward, hopefully it'll work

JEdward

Yes, definitely something I'm going to follow up on in the next couple of weeks.