Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Problem with stopwords(dictionary)

hi everyone,

i want to filter some txt files and remove some useless words.i use the process filterstopwords Dictionary(greek words).but the problem is that the words that i want to remove are there after the filtering.I use utf 8 for encoding and all the txt files are in utf 8. firstly, my txt files were in ANSI encode and the stopwords were removed but the wordlist contained incomprehensible words.Now the word list (with utf8) is correct but the stopwords are still there.sorry for my Engish.

Thanks!!

Find more posts tagged with

AI Studio

Text Mining + NLP

Accepted answers

All comments

nery

I'm having exactly the same problem (with Portuguese text). Have you found a solution yet? Thanks, n.

kersor

No i didn;t find a solution.

A part solution is to tranform the portuguese letters into English.(with replace tokens)

for example the greek word συμφωνώ transformed into simfono.

With this the problem solved.

ut you have to do this again in the classification problems.If you want any further information just tell me

Regards