"Same accuracy...different predictions"

blueearth

Hi i had a multi labels original data set which was weighted by two models gini index and uncertainty ...i collected attributes which gained weight more than .5 and made two databases....one based on gini index weighting attributes and the other one by uncertainty weighting attributes.
i ran these two data sets with a x-validation which trained by neural network operator...the achieved accuracy was same for both data sets :99.45%
but when i applied this model on an unknown database once with gini index attributes and once with uncertainty attributes ...the achieved prediction was completely different...whats the problem ? did i go wrong somewhere?

MariusHelf

Hi,

the accuracy represents only the probability that a new, unseen example drawn from the same distribution as the training set is classified correctly. This leaves room for different predictions on unseen data.

To tell if you did anything wrong we need more detailed information on what you did.
For example the data on which you apply a model must have the same attributes as the training data. So you can't apply a model trained on attribute set A on an example set with attribute set B and expect sensible results. From your description we can't see though what exactly you have done.

Best, Marius