"good repformance almost zero accuracy"

Question

With this training process, the accuracy exceeds 75% . However, when we test a single document, with the same preprocessing steps, the accuracy is almost zero. Is there an easy way to map the initial vector obtained from the training phase to the new vector that corresponds to the new, unseen document without having to write it in code?If anyone wants to help please let me know and sent all the data to a mail. Best regards

kersor · Answer

Dear Dan, thank you for the remark.

However, the X-fold cross validation is performing what you suggest, obtainig a final model, which we save for real-time classification.

Therefore, the new, single document is inserted to the classifier (2nd process) and the classifier reads the model trained above and predicts the label. The problem is that the vectors for training and testing are not the same.

dan_agape · Answer

Hi,

Just a couple of remarks. when we test a single document, with the same preprocessing steps, the accuracy is almost zero.

Note that a document would correspond to a single row/example in the dataset, and it is rather improper to test a model with a single example. Normally evaluating a classification model with one example only leads to an accuracy of 1 or 0. To diminish the variance of the estimates for the accuracy (that is, to improve the estimate you get), more examples are to be used for testing, obviously.

Another remark - you may wish to try also to use TF-IDF when building the vectors - this scheme is commonly used with text mining as it takes into account (in addition to the frequency) how specific tokens/words are for some of the documents to classify, so this may often lead to a better performance of the built model.

Regards
Dan