noise detection in RM?

Question

Hi Forum,

I have a question about noise detection in RM. I dont want to use complex methods like wavelet filter or something like that.  I just want to omit examples which have a low values in most of their attributes. Should I extend a new operator or the existing operators can do this? By the way, how does RM do in noise detection?

I should mention that I have seen the "Filter Example" but I couldn't find how I should use it to fulfill my request.

thanks in advance.

siamak_want · Answer

Hi Marius,

I got the Idea, Thanks in advance:)

MariusHelf · Answer

siamak_want  wrote:But I just want to remove examples which satisfy a simple condition (most their features have a low value)

If you can write the condition yourself, you could use Generate Attributes and Filter Attributes as described here: http://rapid-i.com/rapidforum/index.php/topic,5073.0.html

Best, Marius

siamak_want · Answer

Hi again Marius,Maybe you can try some kind of outlier detection?

I am not sure but I think outliers are something different from noise. Also there is a problem: Detecting outliers from high dimensional data (text data in my case) is very time consuming due to complex algorithms of this area (LOF,COF, ...). But I just want to remove examples which satisfy a simple condition (most their features have a low value) with scanning all the data for one time. I think I need to extend a new operator to remove these examples. Am I right Marius?

Again thanks.