Creating Charts for Performance Measurement Parameters

ozgeozyazar

Hi All !,

I have 2 phases question which I need advice. I have imbalanced data set and according to literature, accuracy is not accurate parameter for these kind of data for performance measurement of classification algorithm. 

I applied parameter optimization for decision tree algorithm with both cross validation for binomial and classification performance. (I am currently using three types feature selection algorithm and try to find out their effect on performance of classification). firstly, ı need to visualize roc curve which shows different results for same algorithm (for example fist graph will indicate result of DT main creation: accuracy, depth:100; confidence:0,050 and another result DT main reation: gain ratio, depth:50; confidence:0,200). 

Second question is, how I can compare with graphics other parameters like kappa, NPV, PPV ..

I hope I clearly express my question. 

Your valuable contribution will highly appreciated.

Özge 

SGolbert

ozgeozyazar,

I think I understand your question. The part about accuracy in inbalanced dataset refers to the fact that you can have a very good accuracy value but have a horrible recall for the minority class. In that case measures like the f-score could be better, but there is no universal best performance measure.

During the optimization you can log all of the performance measures that you use, you have to activate the "log all criteria option":

You can then decide which parameter combination is best using a combination of criteria (not an easy task, as it is a multicriteria optimization and there may be multiple "optimal" points).

Regarding the visualization, you can maybe plot 3 columns at the time, but as in other multivariate problem there is no universal visualization to represent the whole results.

Let me know if it helps, kind regards

Sebastian