🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Removing StopWords using Dictionary

User: "Hyram"
New Altair Community Member
Updated by Jocelyn
Hi
I am using my own dictionary to remove Stopwords. On close analysis, words like "is" are not being removed, although they are in the dictionary. Any clue as to why this is happening?
Thanks,
Hyram

Find more posts tagged with

Sort by:
1 - 2 of 21
    User: "kayman"
    New Altair Community Member
    Accepted Answer
    The process flow seems correct at first glance, so just some additional questions 

    - How do you do word tokenization? If this one is incorrect you might still take full sentences as a token
    - Do you transform to upper or lower case? since you are looking for 'is' I assume lower case  
    - Next you filter by length, as 'is' only contains 2 characters I assume you filter everything that's at least 2 characters. If not than 'is' should be stripped already here so again linked to how you do your word tokenization.
    - How is your dictionary constructed? Every stopword on a new line without any spaces? As you are using the NLTK list it may contain additional characters that RM doesn't like to use.

    You can also use the out of the box 'filter stopwords (english)', it's very similar to the NLTK one as far as I know.
    User: "kayman"
    New Altair Community Member
    Accepted Answer
    Yeah, it's a bit tricky sometimes. A word as 'like' can have a big impact on for instance sentiment analysis, so I personally wouldn't consider it as a generic stopword.

    What I typically do is combine both out of the box stop words and a personal addition. 
    Anyway, if it was gone when using the out of the box option, but remains with the NLTK doc there is indeed probably something with the format used and how it's read.

    Easiest would be to just save it as a plain and simple txt file rather than a docx file, this way you're sure there is nothing missed or added