Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Home
Discussions
Community Q&A

"Loop over Tokens in RapidMiner"

Hello,

is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But then the rest of my intended operations are very complicated.

I'm doing this to simulate the creation of a language model like Katz Backoff and so on. Maybe there is a better way to produce language models but in the moment i can only think of iterating the tokens and calculate the values and store them into an example set.

Thank you

Andreas

Find more posts tagged with

AI Studio

Loops + Branches

Accepted answers

All comments

RWingerter

Hi Andreas,

maybe you can use the "Generate n-Grams (Terms)" operator.

Roland

MariusHelf

Hi,

to loop over the tokens you can use the Loop Attributes operator, since each token is represented by a single attribute.

Best regards,
Marius

Quick Links

All Categories
Recent Discussions
Activity
Unanswered
日本語 (Japanese)
한국어(Korean)
Groups