Rule extraction form neural networks

christian1983

Hi everybody,

did anyone of you make experiences in terms of extracting rules from  neural networks by using rapid miner?
The goal is to make the output of neural networks being compared to decision trees more accurate more comprehensible for humans.

Thank you.

RalfKlinkenberg

christian1983 wrote:
did anyone of you make experiences in terms of extracting rules from  neural networks by using rapid miner?

Hi Christian,

instead of first generating a numeric model like a neural network and then transforming it to symbolic human-readable rules, you can directly generate human-readable rules from your data. RapidMiner offers a wide range of rule learners (see operator group "Modelling" -> "Classification" -> "Rule Learners"). Or do you have any reason to believe that rules generated from a neural network are more readable or more accurate than rules directly generated from the data?

For comparing the accuracy of rules, decision trees, neural networks, and other models, using a validation scheme like cross-validation is the standard way of comparing model performance.

Best regards,
Ralf

dan_agape

Hi Christian,

Actually, although very useful in machine learning/data mining problems, neural nets do not have the explicative power of other models as those based on rules (including decision trees). They do the jobs of classification, prediction, estimation quite well in many situations, but they do not let you understand how (as rule based models do). They can be regarded rather as black boxes.

That's why you cannot really naturally extract rules from neural nets.

In general the most meaning you can get from a neural net is to check what influence the input attributes have on the output attribute. This is done via the so called sensitivity analysis. Not sure if RM incorporated this yet.

Intuitively, after a neural net is learned, a representative instance from the dataset is built (which is, roughly speaking, built as the instance of average values for all the attributes); then a perturbation is applied on each attribute value of this instance, by maintaining all the other values constant, and the effect on the output is measured. Larger the effect - more sensitive is the output to that input attribute - so you get an explanations regarding which attributes influence the most the outcome, but you are far from getting rules from the neural net. For those interested, technicalities are described in http://en.wikipedia.org/wiki/Sensitivity_analysis

Regards
Dan

 

christian1983

Hi Mr. Klinkenberg,

First of all thanks to your quick reply.
I´m aware of the wide range of rule learners RM 5.0 provides, but i aim to apply rule learners on data sets with numerical labels. Of course this is actually a task of regression, but i want to get rules from the results, so i look for a CART tree, which can handle numerical labels. Unfortunately there is no siutable one, so i want to use neural networks having according to literature a higher performance than decision trees and  get rules from the neural network results. So i´m looking for an approach with rapid miner to get them.

So i hope you can help me.
Maybe you could propose a reasonable way to discretizice the numerical label to nominal for a decision tree solution.

Thank you. 

land

Hi,
you could of course discretize the label. Take a look at the various discretization operators. What's reasonable depends on your task and unless you want us to do consulting for you, we are not familiar with your problem.
And of course there are various ways of adding CART functionality to RapidMiner...

Greetings,
  Sebastian

christian1983

Hi Sebastian,

Thanks for your reply.
But maybe you could give me an hint how to add CART functionality to RM in general?

Greetings,
Christian

land

Hi Christian,
in general there are existing two ways of adding CART to RapidMiner: Do it yourself or let it do someone else.
But telling you how to write extensions for RapidMiner could fill books. In fact, it does fill a book, which is (surprisingly) available in our shop. It gives you an introduction how to extend RapidMiner, which are the most important data structures and classes and how they work.
If you don't feel familiar enough with Java programming to do it yourself, or you can't expend the time it would take, you always could ask us for a quote for a certain extension.

Greetings,
  Sebastian