How to calculate inter-model correlation for an ensemble
djafarsidik
New Altair Community Member
Dear experts,
I have read that empirically, stacking ensemble tend to yield better results when there is a significant diversity among the models, one way to examine that is looking for a low correlation between the predictions of the classifier.
In R there is function modelCor (caret) which can be used to check it, Is there any same function/method/operator in Rapidminer ?
Thank you very much for your help and response.
Tagged:
0
Best Answer
-
Hello @djafarsidik
Please find the attached process and do let me know if you have any questions. I added comments under the operator in this process. To use this, you need to download the attached .rmp file and import it into your RM by selecting FILE --> IMport Process. Run it and you can see the correlations. I think I can simplify but the provided solution works.3
Answers
-
Hello @djafarsidik
Did you build a stacking ensemble in RM? if so, why not extract predictions from each model and use "correlation matrix" operator to find a correlation between the predictions coming from different models in a stacked ensemble.
If you can share your process .rmp file we can have a look.2 -
Hi @varunm1 ,Thank you very much for your response,Could you please give me simple sample for what you have explain ?,Actually I have thinked about using correlation matrix, but still cannot find how to use it as your suggestion.Below is something similar to my process, I am using naive bayes and decission tree as base learner and other naive bayes for meta learner, I would like to compare the performance between each base classifier with stacking, and I need to check correlation between each base classifier (inter-model).Thank you very much.
<?xml version="1.0" encoding="UTF-8"?><process version="9.5.001"><br> <context><br> <input/><br> <output/><br> <macros/><br> </context><br> <operator activated="true" class="process" compatibility="9.5.001" expanded="true" name="Process"><br> <parameter key="logverbosity" value="init"/><br> <parameter key="random_seed" value="2001"/><br> <parameter key="send_mail" value="never"/><br> <parameter key="notification_email" value=""/><br> <parameter key="process_duration_for_mail" value="30"/><br> <parameter key="encoding" value="SYSTEM"/><br> <process expanded="true"><br> <operator activated="true" class="retrieve" compatibility="9.5.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="45" y="136"><br> <parameter key="repository_entry" value="//Samples/data/Iris"/><br> </operator><br> <operator activated="true" class="multiply" compatibility="9.5.001" expanded="true" height="124" name="Multiply" width="90" x="179" y="136"/><br> <operator activated="true" class="concurrency:cross_validation" compatibility="9.5.001" expanded="true" height="145" name="Cross Validation DT" width="90" x="380" y="34"><br> <parameter key="split_on_batch_attribute" value="false"/><br> <parameter key="leave_one_out" value="false"/><br> <parameter key="number_of_folds" value="10"/><br> <parameter key="sampling_type" value="linear sampling"/><br> <parameter key="use_local_random_seed" value="false"/><br> <parameter key="local_random_seed" value="1992"/><br> <parameter key="enable_parallel_execution" value="true"/><br> <process expanded="true"><br> <operator activated="true" class="concurrency:parallel_decision_tree" compatibility="9.5.001" expanded="true" height="103" name="Decision Tree" width="90" x="112" y="85"><br> <parameter key="criterion" value="gain_ratio"/><br> <parameter key="maximal_depth" value="10"/><br> <parameter key="apply_pruning" value="true"/><br> <parameter key="confidence" value="0.1"/><br> <parameter key="apply_prepruning" value="true"/><br> <parameter key="minimal_gain" value="0.01"/><br> <parameter key="minimal_leaf_size" value="2"/><br> <parameter key="minimal_size_for_split" value="4"/><br> <parameter key="number_of_prepruning_alternatives" value="3"/><br> </operator><br> <connect from_port="training set" to_op="Decision Tree" to_port="training set"/><br> <connect from_op="Decision Tree" from_port="model" to_port="model"/><br> <portSpacing port="source_training set" spacing="0"/><br> <portSpacing port="sink_model" spacing="0"/><br> <portSpacing port="sink_through 1" spacing="0"/><br> </process><br> <process expanded="true"><br> <operator activated="true" class="apply_model" compatibility="9.5.001" expanded="true" height="82" name="Apply Model (5)" width="90" x="45" y="34"><br> <list key="application_parameters"/><br> <parameter key="create_view" value="false"/><br> </operator><br> <operator activated="true" class="performance_classification" compatibility="9.5.001" expanded="true" height="82" name="Performance DT" width="90" x="179" y="187"><br> <parameter key="main_criterion" value="accuracy"/><br> <parameter key="accuracy" value="true"/><br> <parameter key="classification_error" value="false"/><br> <parameter key="kappa" value="true"/><br> <parameter key="weighted_mean_recall" value="false"/><br> <parameter key="weighted_mean_precision" value="false"/><br> <parameter key="spearman_rho" value="false"/><br> <parameter key="kendall_tau" value="false"/><br> <parameter key="absolute_error" value="false"/><br> <parameter key="relative_error" value="false"/><br> <parameter key="relative_error_lenient" value="false"/><br> <parameter key="relative_error_strict" value="false"/><br> <parameter key="normalized_absolute_error" value="false"/><br> <parameter key="root_mean_squared_error" value="true"/><br> <parameter key="root_relative_squared_error" value="false"/><br> <parameter key="squared_error" value="false"/><br> <parameter key="correlation" value="false"/><br> <parameter key="squared_correlation" value="false"/><br> <parameter key="cross-entropy" value="false"/><br> <parameter key="margin" value="false"/><br> <parameter key="soft_margin_loss" value="false"/><br> <parameter key="logistic_loss" value="false"/><br> <parameter key="skip_undefined_labels" value="true"/><br> <parameter key="use_example_weights" value="true"/><br> <list key="class_weights"/><br> </operator><br> <connect from_port="model" to_op="Apply Model (5)" to_port="model"/><br> <connect from_port="test set" to_op="Apply Model (5)" to_port="unlabelled data"/><br> <connect from_op="Apply Model (5)" from_port="labelled data" to_op="Performance DT" to_port="labelled data"/><br> <connect from_op="Performance DT" from_port="performance" to_port="performance 1"/><br> <connect from_op="Performance DT" from_port="example set" to_port="test set results"/><br> <portSpacing port="source_model" spacing="0"/><br> <portSpacing port="source_test set" spacing="0"/><br> <portSpacing port="source_through 1" spacing="0"/><br> <portSpacing port="sink_test set results" spacing="0"/><br> <portSpacing port="sink_performance 1" spacing="0"/><br> <portSpacing port="sink_performance 2" spacing="0"/><br> </process><br> </operator><br> <operator activated="true" class="concurrency:cross_validation" compatibility="9.5.001" expanded="true" height="145" name="Cross Validation NB" width="90" x="380" y="340"><br> <parameter key="split_on_batch_attribute" value="false"/><br> <parameter key="leave_one_out" value="false"/><br> <parameter key="number_of_folds" value="10"/><br> <parameter key="sampling_type" value="linear sampling"/><br> <parameter key="use_local_random_seed" value="false"/><br> <parameter key="local_random_seed" value="1992"/><br> <parameter key="enable_parallel_execution" value="true"/><br> <process expanded="true"><br> <operator activated="true" class="naive_bayes" compatibility="9.5.001" expanded="true" height="82" name="Naive Bayes" width="90" x="112" y="34"><br> <parameter key="laplace_correction" value="true"/><br> </operator><br> <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/><br> <connect from_op="Naive Bayes" from_port="model" to_port="model"/><br> <portSpacing port="source_training set" spacing="0"/><br> <portSpacing port="sink_model" spacing="0"/><br> <portSpacing port="sink_through 1" spacing="0"/><br> </process><br> <process expanded="true"><br> <operator activated="true" class="apply_model" compatibility="9.5.001" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34"><br> <list key="application_parameters"/><br> <parameter key="create_view" value="false"/><br> </operator><br> <operator activated="true" class="performance_classification" compatibility="9.5.001" expanded="true" height="82" name="Performance NB" width="90" x="179" y="187"><br> <parameter key="main_criterion" value="accuracy"/><br> <parameter key="accuracy" value="true"/><br> <parameter key="classification_error" value="false"/><br> <parameter key="kappa" value="true"/><br> <parameter key="weighted_mean_recall" value="false"/><br> <parameter key="weighted_mean_precision" value="false"/><br> <parameter key="spearman_rho" value="false"/><br> <parameter key="kendall_tau" value="false"/><br> <parameter key="absolute_error" value="false"/><br> <parameter key="relative_error" value="false"/><br> <parameter key="relative_error_lenient" value="false"/><br> <parameter key="relative_error_strict" value="false"/><br> <parameter key="normalized_absolute_error" value="false"/><br> <parameter key="root_mean_squared_error" value="true"/><br> <parameter key="root_relative_squared_error" value="false"/><br> <parameter key="squared_error" value="false"/><br> <parameter key="correlation" value="false"/><br> <parameter key="squared_correlation" value="false"/><br> <parameter key="cross-entropy" value="false"/><br> <parameter key="margin" value="false"/><br> <parameter key="soft_margin_loss" value="false"/><br> <parameter key="logistic_loss" value="false"/><br> <parameter key="skip_undefined_labels" value="true"/><br> <parameter key="use_example_weights" value="true"/><br> <list key="class_weights"/><br> </operator><br> <connect from_port="model" to_op="Apply Model" to_port="model"/><br> <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/><br> <connect from_op="Apply Model" from_port="labelled data" to_op="Performance NB" to_port="labelled data"/><br> <connect from_op="Performance NB" from_port="performance" to_port="performance 1"/><br> <connect from_op="Performance NB" from_port="example set" to_port="test set results"/><br> <portSpacing port="source_model" spacing="0"/><br> <portSpacing port="source_test set" spacing="0"/><br> <portSpacing port="source_through 1" spacing="0"/><br> <portSpacing port="sink_test set results" spacing="0"/><br> <portSpacing port="sink_performance 1" spacing="0"/><br> <portSpacing port="sink_performance 2" spacing="0"/><br> </process><br> </operator><br> <operator activated="true" class="concurrency:cross_validation" compatibility="9.5.001" expanded="true" height="145" name="Cross Validation stacking" width="90" x="380" y="187"><br> <parameter key="split_on_batch_attribute" value="false"/><br> <parameter key="leave_one_out" value="false"/><br> <parameter key="number_of_folds" value="10"/><br> <parameter key="sampling_type" value="linear sampling"/><br> <parameter key="use_local_random_seed" value="false"/><br> <parameter key="local_random_seed" value="1992"/><br> <parameter key="enable_parallel_execution" value="true"/><br> <process expanded="true"><br> <operator activated="true" class="stacking" compatibility="9.5.001" expanded="true" height="68" name="Stacking (2)" width="90" x="179" y="85"><br> <parameter key="keep_all_attributes" value="true"/><br> <parameter key="keep_confidences" value="false"/><br> <process expanded="true"><br> <operator activated="true" class="naive_bayes" compatibility="9.5.001" expanded="true" height="82" name="Naive Bayes (3)" width="90" x="112" y="85"><br> <parameter key="laplace_correction" value="true"/><br> </operator><br> <operator activated="true" class="concurrency:parallel_decision_tree" compatibility="9.5.001" expanded="true" height="103" name="Decision Tree (4)" width="90" x="112" y="238"><br> <parameter key="criterion" value="gain_ratio"/><br> <parameter key="maximal_depth" value="10"/><br> <parameter key="apply_pruning" value="true"/><br> <parameter key="confidence" value="0.1"/><br> <parameter key="apply_prepruning" value="true"/><br> <parameter key="minimal_gain" value="0.01"/><br> <parameter key="minimal_leaf_size" value="2"/><br> <parameter key="minimal_size_for_split" value="4"/><br> <parameter key="number_of_prepruning_alternatives" value="3"/><br> </operator><br> <connect from_port="training set 1" to_op="Naive Bayes (3)" to_port="training set"/><br> <connect from_port="training set 2" to_op="Decision Tree (4)" to_port="training set"/><br> <connect from_op="Naive Bayes (3)" from_port="model" to_port="base model 1"/><br> <connect from_op="Decision Tree (4)" from_port="model" to_port="base model 2"/><br> <portSpacing port="source_training set 1" spacing="0"/><br> <portSpacing port="source_training set 2" spacing="0"/><br> <portSpacing port="source_training set 3" spacing="0"/><br> <portSpacing port="sink_base model 1" spacing="0"/><br> <portSpacing port="sink_base model 2" spacing="0"/><br> <portSpacing port="sink_base model 3" spacing="0"/><br> </process><br> <process expanded="true"><br> <operator activated="true" class="naive_bayes" compatibility="9.5.001" expanded="true" height="82" name="Naive Bayes (7)" width="90" x="179" y="85"><br> <parameter key="laplace_correction" value="true"/><br> </operator><br> <connect from_port="stacking examples" to_op="Naive Bayes (7)" to_port="training set"/><br> <connect from_op="Naive Bayes (7)" from_port="model" to_port="stacking model"/><br> <portSpacing port="source_stacking examples" spacing="0"/><br> <portSpacing port="sink_stacking model" spacing="0"/><br> </process><br> </operator><br> <connect from_port="training set" to_op="Stacking (2)" to_port="training set"/><br> <connect from_op="Stacking (2)" from_port="model" to_port="model"/><br> <portSpacing port="source_training set" spacing="0"/><br> <portSpacing port="sink_model" spacing="0"/><br> <portSpacing port="sink_through 1" spacing="0"/><br> </process><br> <process expanded="true"><br> <operator activated="true" class="apply_model" compatibility="9.5.001" expanded="true" height="82" name="Apply Model (4)" width="90" x="45" y="34"><br> <list key="application_parameters"/><br> <parameter key="create_view" value="false"/><br> </operator><br> <operator activated="true" class="performance_classification" compatibility="9.5.001" expanded="true" height="82" name="Performance Stack-nb" width="90" x="179" y="187"><br> <parameter key="main_criterion" value="accuracy"/><br> <parameter key="accuracy" value="true"/><br> <parameter key="classification_error" value="false"/><br> <parameter key="kappa" value="true"/><br> <parameter key="weighted_mean_recall" value="false"/><br> <parameter key="weighted_mean_precision" value="false"/><br> <parameter key="spearman_rho" value="false"/><br> <parameter key="kendall_tau" value="false"/><br> <parameter key="absolute_error" value="false"/><br> <parameter key="relative_error" value="false"/><br> <parameter key="relative_error_lenient" value="false"/><br> <parameter key="relative_error_strict" value="false"/><br> <parameter key="normalized_absolute_error" value="false"/><br> <parameter key="root_mean_squared_error" value="true"/><br> <parameter key="root_relative_squared_error" value="false"/><br> <parameter key="squared_error" value="false"/><br> <parameter key="correlation" value="false"/><br> <parameter key="squared_correlation" value="false"/><br> <parameter key="cross-entropy" value="false"/><br> <parameter key="margin" value="false"/><br> <parameter key="soft_margin_loss" value="false"/><br> <parameter key="logistic_loss" value="false"/><br> <parameter key="skip_undefined_labels" value="true"/><br> <parameter key="use_example_weights" value="true"/><br> <list key="class_weights"/><br> </operator><br> <connect from_port="model" to_op="Apply Model (4)" to_port="model"/><br> <connect from_port="test set" to_op="Apply Model (4)" to_port="unlabelled data"/><br> <connect from_op="Apply Model (4)" from_port="labelled data" to_op="Performance Stack-nb" to_port="labelled data"/><br> <connect from_op="Performance Stack-nb" from_port="performance" to_port="performance 1"/><br> <connect from_op="Performance Stack-nb" from_port="example set" to_port="test set results"/><br> <portSpacing port="source_model" spacing="0"/><br> <portSpacing port="source_test set" spacing="0"/><br> <portSpacing port="source_through 1" spacing="0"/><br> <portSpacing port="sink_test set results" spacing="0"/><br> <portSpacing port="sink_performance 1" spacing="0"/><br> <portSpacing port="sink_performance 2" spacing="0"/><br> </process><br> </operator><br> <connect from_op="Retrieve Iris" from_port="output" to_op="Multiply" to_port="input"/><br> <connect from_op="Multiply" from_port="output 1" to_op="Cross Validation stacking" to_port="example set"/><br> <connect from_op="Multiply" from_port="output 2" to_op="Cross Validation NB" to_port="example set"/><br> <connect from_op="Multiply" from_port="output 3" to_op="Cross Validation DT" to_port="example set"/><br> <connect from_op="Cross Validation DT" from_port="test result set" to_port="result 6"/><br> <connect from_op="Cross Validation DT" from_port="performance 1" to_port="result 5"/><br> <connect from_op="Cross Validation NB" from_port="test result set" to_port="result 3"/><br> <connect from_op="Cross Validation NB" from_port="performance 1" to_port="result 2"/><br> <connect from_op="Cross Validation stacking" from_port="test result set" to_port="result 4"/><br> <connect from_op="Cross Validation stacking" from_port="performance 1" to_port="result 1"/><br> <portSpacing port="source_input 1" spacing="0"/><br> <portSpacing port="sink_result 1" spacing="0"/><br> <portSpacing port="sink_result 2" spacing="0"/><br> <portSpacing port="sink_result 3" spacing="0"/><br> <portSpacing port="sink_result 4" spacing="0"/><br> <portSpacing port="sink_result 5" spacing="0"/><br> <portSpacing port="sink_result 6" spacing="0"/><br> <portSpacing port="sink_result 7" spacing="0"/><br> </process><br> </operator><br></process>
0 -
Hello @djafarsidik
Can you provide a .rmp file? you can export the process as .rmp by going to FILE --> Export Process in rapidminer. The XML you copied didnt have full code as it shows only the first line of process.0 -
-
Hello @djafarsidik
Please find the attached process and do let me know if you have any questions. I added comments under the operator in this process. To use this, you need to download the attached .rmp file and import it into your RM by selecting FILE --> IMport Process. Run it and you can see the correlations. I think I can simplify but the provided solution works.3