Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Why UTF-8 is not working?

Today I converted a pdf to txt, and I'm trying to analyse some therms frequency in the text. Despite the txt is in UTF-8 and I've already changed the program's encoding into the default (SYSTEM) or into 'UTF-8' before tokenizing, generating n_grams, it keeps showing incorrect words. For example, the word should've been 'abrangência' inetead of 'abrangãºncia'.

Find more posts tagged with

AI Studio

Accepted answers

MartinLiebig

Hi there,

what operator do you use to read the text file? It should have a setting as well.

Cheers,

Martin

All comments

heron_oliveira

My txt file has correct words, it only happends when I run operators in RapidMiner. And I'm using operators for tokenizing, Transform Cases, Generate n-Grams, Filter Tokens and Filter StopWords. But the problem begins since the first operator wich is Tokenize...

heron_oliveira

I would also like to know how to must be the stop words list format. Since there is no Portuguese stop words operator, I made a list document, but I don't know if it accepts list format or if it should be dictionary or something else.

MartinLiebig

Hi there,

what operator do you use to read the text file? It should have a setting as well.

Cheers,

Martin

heron_oliveira

Exactly, I was changing enconde in the settings > preferences. But in fact I should've done it on the operator settings. Thanks!