Tokenize operator of Text processing extension stopped working after upgrading to 7.3

Screen Shot 2016-12-06 at 12.30.32 PM.png

This is a screenshot of the results tab, when putting a breakpoint after the 'Tokenize' operator. The input if a text attribute called 'description' and the output is the same plain attribute instead of the tokenized version.

Rapidminer Studio version 7.3, extension is also up-to-date (7.3.0)

It was working correctly when I was using Studio 7.0 . Is there a way to downgrade? Didn't find older versions on sourcefourge either.

Find more posts tagged with

AI Studio

Extensions

Text Mining + NLP

Accepted answers

aliweb

In my process I was reading from a mysql table and the column type in mysql was 'Text', but RapidMiner was seeing it as 'nominal' attribute. Now after using the nominal to text operator, it works.

p.s: This was not an issue in version 7.0 as I was not using this type convertor operator.

Thanks

All comments

Thomas_Ott

With version 7.3 there's a limit to how much data you can process (10,000 rows). What is your license limit? Could be related to that.

aliweb

My license has unlimited data rows and in this process I only have 2000 data rows.

zprekopcsak

Hi,

I have just tested this on 7.3 and I see proper results at the breakpoint. Please check if you have anything meaningful on the input and not just empty text. Also, you can enable the Log panel to look for any potential errors. In case you can reproduce this and can share a demo process and demo data then we can investigate deeper.

Thanks, Zoltan

aliweb

In my process I was reading from a mysql table and the column type in mysql was 'Text', but RapidMiner was seeing it as 'nominal' attribute. Now after using the nominal to text operator, it works.

p.s: This was not an issue in version 7.0 as I was not using this type convertor operator.

Thanks