"Weighted average of two models, with weights based on other attributes"
keith
New Altair Community Member
I have a data set that contains a numerical label, the predictions from two other models (not generated in RM), and several attributes (mostly numerical, but a few binomial or polynominal).
I suspect that certain values of attributes determine whether one model or the other is more accurate for a given instance, and so I would like to created a weighted average of the two models' predictions, where the weights are based on the other attributes. The relationship between the attributes and weights is what I want RM to learn.
In other words, I have data with attributes: actual, model1, model2, att1
and I want to model this as:
actual = f(att1) * model1 + (1 - f(att1)) * model2
I'm actually open to other approaches that can combine two (or more) predictions based on other attributes. This was just the simplest place to begin.
Two questions:
1) How can we get RM to learn a model of this type (assuming it's possible)?
2) Given that the values (model1, model2) are themselves predictions, is there any way to get RM to see them that way (which might open up Bagging or Boosting as ways to combine them)? There doesn't seem to be a way to define a basic learner that says "predict X to be the value of Y". DefaultLearner almost does this, but instead of the mean, median, or constant, I want it to be the value of another attribute.
The following RM process creates some data that has the properties I described. The predictions from two models are given, but one model is more accurate, depending on the value of some third attribute.
Thanks,
Keith
I suspect that certain values of attributes determine whether one model or the other is more accurate for a given instance, and so I would like to created a weighted average of the two models' predictions, where the weights are based on the other attributes. The relationship between the attributes and weights is what I want RM to learn.
In other words, I have data with attributes: actual, model1, model2, att1
and I want to model this as:
actual = f(att1) * model1 + (1 - f(att1)) * model2
I'm actually open to other approaches that can combine two (or more) predictions based on other attributes. This was just the simplest place to begin.
Two questions:
1) How can we get RM to learn a model of this type (assuming it's possible)?
2) Given that the values (model1, model2) are themselves predictions, is there any way to get RM to see them that way (which might open up Bagging or Boosting as ways to combine them)? There doesn't seem to be a way to define a basic learner that says "predict X to be the value of Y". DefaultLearner almost does this, but instead of the mean, median, or constant, I want it to be the value of another attribute.
The following RM process creates some data that has the properties I described. The predictions from two models are given, but one model is more accurate, depending on the value of some third attribute.
Thanks,
Keith
<operator name="Root" class="Process" expanded="yes">
<operator name="ExampleSetGenerator" class="ExampleSetGenerator">
<parameter key="target_function" value="sum"/>
<parameter key="number_of_attributes" value="2"/>
<parameter key="attributes_lower_bound" value="0.0"/>
<parameter key="attributes_upper_bound" value="100.0"/>
</operator>
<operator name="Rename to Model1" class="ChangeAttributeName">
<parameter key="old_name" value="att1"/>
<parameter key="new_name" value="model1"/>
</operator>
<operator name="Rename to Model2" class="ChangeAttributeName">
<parameter key="old_name" value="att2"/>
<parameter key="new_name" value="model2"/>
</operator>
<operator name="AttributeConstruction" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="att1" value="rand()"/>
<parameter key="actual" value="sin(att1*pi/2)*model1 + (1-sin(att1*pi/2))*model2 + (rand()-0.5)*5"/>
<parameter key="diff1" value="abs(actual-model1)/abs(model1-model2)"/>
<parameter key="diff2" value="abs(actual-model2)/abs(model1-model2)"/>
</list>
</operator>
<operator name="ChangeAttributeRole" class="ChangeAttributeRole">
<parameter key="name" value="actual"/>
<parameter key="target_role" value="label"/>
</operator>
</operator>
0
Answers
-
Hi Keith,
Mmm, interesting question, just as a first point...
You just switch the attribute to a prediction.. like thisThere doesn't seem to be a way to define a basic learner that says "predict X to be the value of Y". <operator name="Root" class="Process" expanded="yes">
And here is code to exploit that...
<operator name="ExampleSetGenerator" class="ExampleSetGenerator">
<parameter key="target_function" value="sum"/>
<parameter key="number_of_attributes" value="2"/>
<parameter key="attributes_lower_bound" value="0.0"/>
<parameter key="attributes_upper_bound" value="100.0"/>
</operator>
<operator name="Rename to Model1" class="ChangeAttributeName">
<parameter key="old_name" value="att1"/>
<parameter key="new_name" value="model1"/>
</operator>
<operator name="Rename to Model2" class="ChangeAttributeName">
<parameter key="old_name" value="att2"/>
<parameter key="new_name" value="model2"/>
</operator>
<operator name="AttributeConstruction" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="att1" value="rand()"/>
<parameter key="actual" value="sin(att1*pi/2)*model1 + (1-sin(att1*pi/2))*model2 + (rand()-0.5)*5"/>
</list>
</operator>
<operator name="ChangeAttributeRole" class="ChangeAttributeRole">
<parameter key="name" value="actual"/>
<parameter key="target_role" value="label"/>
</operator>
<operator name="ChangeAttributeRole (2)" class="ChangeAttributeRole">
<parameter key="name" value="model1"/>
<parameter key="target_role" value="prediction"/>
</operator>
<operator name="RegressionPerformance" class="RegressionPerformance">
<parameter key="root_mean_squared_error" value="true"/>
</operator>
</operator><operator name="Root" class="Process" expanded="yes">
<operator name="ExampleSetGenerator" class="ExampleSetGenerator">
<parameter key="target_function" value="sum"/>
<parameter key="number_of_attributes" value="2"/>
<parameter key="attributes_lower_bound" value="0.0"/>
<parameter key="attributes_upper_bound" value="100.0"/>
</operator>
<operator name="Rename to Model1" class="ChangeAttributeName">
<parameter key="old_name" value="att1"/>
<parameter key="new_name" value="model1"/>
</operator>
<operator name="Rename to Model2" class="ChangeAttributeName">
<parameter key="old_name" value="att2"/>
<parameter key="new_name" value="model2"/>
</operator>
<operator name="AttributeConstruction" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="att1" value="rand()"/>
<parameter key="actual" value="sin(att1*pi/2)*model1 + (1-sin(att1*pi/2))*model2 + (rand()-0.5)*5"/>
</list>
</operator>
<operator name="ChangeAttributeRole" class="ChangeAttributeRole">
<parameter key="name" value="actual"/>
<parameter key="target_role" value="label"/>
</operator>
<operator name="GridParameterOptimization" class="GridParameterOptimization" expanded="yes">
<list key="parameters">
<parameter key="SingleMacroDefinition.value" value="0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1,1.1,1.2,1.3,1.4,1.5"/>
</list>
<operator name="SingleMacroDefinition" class="SingleMacroDefinition">
<parameter key="macro" value="Balance"/>
<parameter key="value" value="1.5"/>
</operator>
<operator name="AttributeConstruction (2)" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="combo" value="model1*(att1*%{Balance})+model2*att1*((1-%{Balance}))"/>
</list>
</operator>
<operator name="ChangeAttributeRole (2)" class="ChangeAttributeRole">
<parameter key="name" value="combo"/>
<parameter key="target_role" value="prediction"/>
</operator>
<operator name="RegressionPerformance" class="RegressionPerformance">
<parameter key="root_mean_squared_error" value="true"/>
</operator>
<operator name="ProcessLog" class="ProcessLog">
<list key="log">
<parameter key="Threshold" value="operator.SingleMacroDefinition.value.macro_value"/>
<parameter key="Performance" value="operator.RegressionPerformance.value.performance"/>
</list>
</operator>
</operator>
</operator>
It is poets' day so brain clunking very slowly, if anything more emerges I'll let you know..
Good weekend to all.
0 -
Very cool. Thanks, haddock.
I was actually thinking of a slightly different scenario. I wanted to be able to define the "attribute X is a prediction of label Y" as a very simple learner, i.e. something that can be turned into a model object, so it could be used in a meta-learner scenario like Bagging, as a possible way towards my ultimate goal of combining predictions.
In the process below, I've got a Bagging meta-learner with three learners (JMySVMLearner, NearestNeighbor, LinearRegression), with another LinearRegression learner as the method for how models are bagged. I've got a disabled OperatorChain called "Pseudo-learner" which tries to embed your idea (changing the role from attribute to prediction) to try to have it act as a 4th learner in the Bagging. If you enable Pseudo-learner and try to run the process, you'll get an error because Pseudo-learner doesn't return a Model object, which is what Bagging expects.
I think if DefaultLearner could be enhanced to allow specifying the value of another attribute as the prediction, I think this would work, and might be the most straightforward option. Maybe I'll pop over to the Feature Requests subforum and suggest it. :-)
<operator name="Root" class="Process" expanded="yes">
<operator name="ExampleSetGenerator" class="ExampleSetGenerator">
<parameter key="target_function" value="sum"/>
<parameter key="number_of_attributes" value="2"/>
<parameter key="attributes_lower_bound" value="0.0"/>
<parameter key="attributes_upper_bound" value="100.0"/>
</operator>
<operator name="Rename to Model1" class="ChangeAttributeName">
<parameter key="old_name" value="att1"/>
<parameter key="new_name" value="model1"/>
</operator>
<operator name="Rename to Model2" class="ChangeAttributeName">
<parameter key="old_name" value="att2"/>
<parameter key="new_name" value="model2"/>
</operator>
<operator name="AttributeConstruction" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="att1" value="rand()"/>
<parameter key="actual" value="sin(att1*pi/2)*model1 + (1-sin(att1*pi/2))*model2 + (rand()-0.5)*5"/>
</list>
</operator>
<operator name="ChangeAttributeRole" class="ChangeAttributeRole">
<parameter key="name" value="actual"/>
<parameter key="target_role" value="label"/>
</operator>
<operator name="Stacking" class="Stacking" expanded="yes">
<parameter key="keep_example_set" value="true"/>
<operator name="LinearRegression" class="LinearRegression">
</operator>
<operator name="Pseudo-learner" class="OperatorChain" activated="no" expanded="no">
<operator name="SingleMacroDefinition" class="SingleMacroDefinition">
<parameter key="macro" value="Balance"/>
<parameter key="value" value="1.5"/>
</operator>
<operator name="AttributeConstruction (2)" class="AttributeConstruction">
<list key="function_descriptions">
<parameter key="combo" value="model1*(att1*%{Balance})+model2*att1*((1-%{Balance}))"/>
</list>
</operator>
<operator name="ChangeAttributeRole (2)" class="ChangeAttributeRole">
<parameter key="name" value="combo"/>
<parameter key="target_role" value="prediction"/>
</operator>
</operator>
<operator name="JMySVMLearner" class="JMySVMLearner">
</operator>
<operator name="NearestNeighbors" class="NearestNeighbors">
<parameter key="k" value="5"/>
</operator>
<operator name="LinearRegression (2)" class="LinearRegression">
</operator>
</operator>
<operator name="ModelApplier" class="ModelApplier">
<parameter key="keep_model" value="true"/>
<list key="application_parameters">
</list>
</operator>
<operator name="RegressionPerformance" class="RegressionPerformance">
<parameter key="root_mean_squared_error" value="true"/>
</operator>
</operator>0