Out of spec data

Process_Intern

Hi all, 

I am newbie on RM. I was wondering if there is a way on Rapid Miner to sort out the data. I am more advanced on Python and Excel, but comparing to an Excel, I would like RM to check my table on a certain column. In that column, if I have a value out of a specification that I give to my program, I would like Rapid Miner to give me a hint on that, maybe raise a flag or something ttelling me that I have a value out of specs. 

Thanks

Telcontar120

You can do this a number of different ways in RapidMiner.  In addition to the ways Martin already mentioned, for example you could use Filter Examples and then specify a value range for given attributes and have it send the matching examples to another output file for your review.
You could use Generate Attributes using logical tests for as many attributes as you like.  You could then specify different actions to be taken based on the values of these test attributes (typically 0/1 flags) such as when you get a certain number of them.
You could Loop Examples and have similar logical tests determine whether to keep or remove that example from your dataset. The possibilities are endless.

MartinLiebig

Hi,
what do you call "out of specification"? A simple Range? Or something like: It should be real, but is nominal?

For Outliers i would look for example at Tukey test. For the Conformance i would check "Check Model Conformance".

Cheers,
Martin

Process_Intern

An out of spec in my case is a number out of a simple range. AS an example, I want RM to notice me when my att1 value is under 200. A thing to know is that my att1 is Real type. I tried to generate a macro with an "If" formula but it didn't work. The macro was telling me that it wasn't knowing what my att1 was but I had clearly defined it previously. 

Telcontar120

You can do this a number of different ways in RapidMiner.  In addition to the ways Martin already mentioned, for example you could use Filter Examples and then specify a value range for given attributes and have it send the matching examples to another output file for your review.
You could use Generate Attributes using logical tests for as many attributes as you like.  You could then specify different actions to be taken based on the values of these test attributes (typically 0/1 flags) such as when you get a certain number of them.
You could Loop Examples and have similar logical tests determine whether to keep or remove that example from your dataset. The possibilities are endless.

Process_Intern

Thank you. 

I've figured out with I will do, thank you!

MartinLiebig

Hi @Process_Intern

one neat trick I've learned is that you can use Normalize on the training set, apply the normalize on the application set and check for [0,1]. This way you can easily check all attributes for [0,1] instead of arbitrary ranges.

~Martin