Outlier Distance n

dennis

Hey,

I am new to Rapid Miner. I was wondering how one determines the number of outliners Rapid Miner should look for in a dataset of for example 100 datapoints.

 

Thanks in advance.

 

Den

Telcontar120

Hi Dennis, unfortunately there isn't a simple one-size-fits-all answer to the question of outlier detection.  It is very contextual to your dataset.  So here are a few ideas:

1.  In the standard outlier detection operator, you can try setting the threshold for outliers at 5, and then visually examine the outliers that it selects.  Ask yourself, are these examples really outliers in the context of my dataset?  Then try changing the threshold again (either up or down) and seeing what examples are added or subtracted.  Sometimes this will help get a sense of the appropriate threshold.
2. You should download the free anomaly detection extension from the marketplace.  This contains operators that do not require you to specify in advance how many outliers there should be.  You can then look at multiple methods of identifying outliers based on similarity measures.  If you don't have too many attributes, then the histogram-based outlier score (HBOS) is a very useful one.  If you have too many attributes, you will be better off with a nearest-neighbor approach such as the local outlier factor (LOF) approach.

I hope this is helpful.

Regards,