Outlier detection (in real-time)

Chris_J_

Hello RapidMiner-Community,

I'm quite new to the topic of machine learning algorithm. I usually detect outliers simply by calculating a mean value and then use a 2 or 3-fold standard deviation to decide if a data instance is normal or an outlier.

Now, I'd like to add and test some other attributes to detect outliers and so the mentioned method above doesn't do well anymore. The condition is that the algorithms can detect outliers in real time (or with a short delay). The model itself can be calculated during the day.

As I read in Chandolas paper "Anomaly detection: A survey" classification algorithm like one-class SVM could be used, because the testing phase is fast after the model is trained. Also clustering mechanism may be possible because each new instance has to be tested against a few clusters.

Now, I would be very thankful if you could give me some technical advice what algorithm in RapidMiner could actually work to detect outliers in real time, especially with spatio-temporal datasets.

Thanks for your help!

Greetings,

Chis

rakirk

So an idea that I have used in the past was to use the k-nearest neighbors algorithm to assemble a sort of 'state space' of features that cluster together. Then, in real time, the algorithm could look for proximity of existing elements given what it knows about the features of the new event.

Let me know how it turns out,

rk

ThomasM

Hi,

Did you try Cobweb clustering ? It is sensitive to examples' ordering, thus taking into account time parameter.

Thomas.

ThomasM

I would add a notice to what I have just said : "state space" means for me "Kalman" ; thus, Does RapidMiner have an implementation of Kalman Filtering ?

Thomas.

arunpushkar

Any body knows how to use anomaly detection operators in rapid miner an can some body explain how to use LOF please.