Why UTF-8 is not working?
Today I converted a pdf to txt, and I'm trying to analyse some therms frequency in the text. Despite the txt is in UTF-8 and I've already changed the program's encoding into the default (SYSTEM) or into 'UTF-8' before tokenizing, generating n_grams, it keeps showing incorrect words. For example, the word should've been 'abrangência' inetead of 'abrangãºncia'.
Find more posts tagged with
Sort by:
1 - 4 of
41
My txt file has correct words, it only happends when I run operators in RapidMiner. And I'm using operators for tokenizing, Transform Cases, Generate n-Grams, Filter Tokens and Filter StopWords. But the problem begins since the first operator wich is Tokenize...
I would also like to know how to must be the stop words list format. Since there is no Portuguese stop words operator, I made a list document, but I don't know if it accepts list format or if it should be dictionary or something else.
Altair EmployeeHi there,
what operator do you use to read the text file? It should have a setting as well.
Cheers,
Martin
Exactly, I was changing enconde in the settings > preferences. But in fact I should've done it on the operator settings. Thanks!
Sort by:
1 - 1 of
11
Altair EmployeeHi there,
what operator do you use to read the text file? It should have a setting as well.
Cheers,
Martin