Overfitting - Sentiment Analysis

SilkeKoch

Hi together,
I am not very experienced. I did use the sentiment template and created a model with about 83 % accuracy. But the model does not predict the sentiments of my unseen data well. The confidence average is about 50 to 60% only. What can I do to get a model which generalizes better? And is there an opportunity to compare my labeled data with the unlabeled data to see if the bad confidence is really so so bad.
 my training data is balanced about 1000 positive / 1000 negative. And I applied the model to about 100 unlabeled data.

Thank you very much for your help Silke

Telcontar120

It's hard to answer a general question like this without knowing more about the details of the analysis.
For example, How did you construct your model?  Did you use cross-validation?  What ML algorithm are you using?
Are the new 100 cases that you are validating similar to the original 2000 that you built the model on?
Did you ensure they were all having their text data processed in the same way that you did on your original model development sample?  You should be using a pre-built wordlist when you use this approach, and this is something that many users forget to do.  There are some good tutorials and lessons on text mining in the RapidMiner Academy that you might want to check out. 

SilkeKoch

I did use crossvalidation with SVM (dot.kernel). the 100 cases are similar to the original as it is from the same customer review dataset. I used exactly the same preprocessing.  I tried pruning but it doesn' t help. Should I do pruning? 

lionelderkrikor

HI @SilkeKoch,

Maybe SVM is not the most relevant model for your data.
Have you tried to submit your dataset to AutoModel to determine how some other models perform ?

Regards,

Lionel

SilkeKoch

Thank you, I  did not try that yet, but I will.