Performance operator - export results

Hello @Serek91

There is a "performance to data" operator, you can connect this to the performance operator and this will help you connect to write excel or CSV operator. But it will save the accuracy and other parameters and not the confusion matrix.

I see that the confusion matrix can be saved using report generation extension, there are two reporting extensions, one is an old one and the new one is "Advanced reporting extension" (3rd Party). This is a paid one but not much expensive if you want to consider (Click_Here).

I tried the free one "Reporting Extension", this is free but old and I don't think it works well, I tried saving as pdf but the confusion matrices are saved in the second page. Sample XML is given below with titanic dataset. You need to have the mentioned extension to run this code.

Hello @mschmitz

Do we have any document that specifies extensions and their relevant operators? It would be good if there is one.

Thanks

Thanks for your response. I tried searching confusion in marketplace and it didn't return anything, I think the keyword is not searchable in this case.

Either we can do text (string) matching in market place, but my concern is it returns lot of operators with that string and may not be feasible. I am thinking something like an excel with extension name and their relevant operator with a one line description if possible. When ever a new extension is created, this should be updates with relevant information. 

There may be other efficient ways, need to give some thought on this one.

Thanks

Hi @varunm1

Pretty strange and looks like some connection problem (?)... I personally never had any problem with search on the marketplace:



Same results are achievable also on the marketplace web site: https://marketplace.rapidminer.com/UpdateServer/faces/index.xhtml

Ohh, that's interesting. Thanks @kypexin some how it didn't work for me when I looked for it before answering this. I will try again.

This is fine, this resolves my question @mschmitz

I will update if there are any issues.

Hi @Serek91

There is an approximative solution to extract (in your case) as exampleset via the Performance to Data operator : 
 -  the weighted mean recall
 -  the weighted mean precision

NB : These performances metrics are initialy calculated by the Performance (Classification) operator inside the CV operator (so you have to check these 2 performance metrics in the parameters of the ¨Performance operator).

Then you can "join" the exampleset with these 2 perf. metrics with the exampleset with your confusion matrix...
The resulting exampleset looks like that : (here for the Iris Dataset) :

 
The process : 

<?xml version="1.0" encoding="UTF-8"?><process version="9.4.000-BETA">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.4.000-BETA" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.4.000-BETA" expanded="true" height="68" name="Retrieve Iris" width="90" x="45" y="85">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="concurrency:cross_validation" compatibility="9.4.000-BETA" expanded="true" height="145" name="Cross Validation" width="90" x="179" y="85">
        <parameter key="split_on_batch_attribute" value="false"/>
        <parameter key="leave_one_out" value="false"/>
        <parameter key="number_of_folds" value="10"/>
        <parameter key="sampling_type" value="automatic"/>
        <parameter key="use_local_random_seed" value="false"/>
        <parameter key="local_random_seed" value="1992"/>
        <parameter key="enable_parallel_execution" value="true"/>
        <process expanded="true">
          <operator activated="true" class="concurrency:parallel_decision_tree" compatibility="9.4.000-BETA" expanded="true" height="103" name="Decision Tree" width="90" x="112" y="34">
            <parameter key="criterion" value="gain_ratio"/>
            <parameter key="maximal_depth" value="10"/>
            <parameter key="apply_pruning" value="true"/>
            <parameter key="confidence" value="0.1"/>
            <parameter key="apply_prepruning" value="true"/>
            <parameter key="minimal_gain" value="0.01"/>
            <parameter key="minimal_leaf_size" value="2"/>
            <parameter key="minimal_size_for_split" value="4"/>
            <parameter key="number_of_prepruning_alternatives" value="3"/>
          </operator>
          <connect from_port="training set" to_op="Decision Tree" to_port="training set"/>
          <connect from_op="Decision Tree" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="9.4.000-BETA" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
            <list key="application_parameters"/>
            <parameter key="create_view" value="false"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="9.4.000-BETA" expanded="true" height="82" name="Performance" width="90" x="179" y="34">
            <parameter key="main_criterion" value="first"/>
            <parameter key="accuracy" value="true"/>
            <parameter key="classification_error" value="false"/>
            <parameter key="kappa" value="false"/>
            <parameter key="weighted_mean_recall" value="true"/>
            <parameter key="weighted_mean_precision" value="true"/>
            <parameter key="spearman_rho" value="false"/>
            <parameter key="kendall_tau" value="false"/>
            <parameter key="absolute_error" value="false"/>
            <parameter key="relative_error" value="false"/>
            <parameter key="relative_error_lenient" value="false"/>
            <parameter key="relative_error_strict" value="false"/>
            <parameter key="normalized_absolute_error" value="false"/>
            <parameter key="root_mean_squared_error" value="false"/>
            <parameter key="root_relative_squared_error" value="false"/>
            <parameter key="squared_error" value="false"/>
            <parameter key="correlation" value="false"/>
            <parameter key="squared_correlation" value="false"/>
            <parameter key="cross-entropy" value="false"/>
            <parameter key="margin" value="false"/>
            <parameter key="soft_margin_loss" value="false"/>
            <parameter key="logistic_loss" value="false"/>
            <parameter key="skip_undefined_labels" value="true"/>
            <parameter key="use_example_weights" value="true"/>
            <list key="class_weights"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="performance 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_test set results" spacing="0"/>
          <portSpacing port="sink_performance 1" spacing="0"/>
          <portSpacing port="sink_performance 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="multiply" compatibility="9.4.000-BETA" expanded="true" height="103" name="Multiply" width="90" x="313" y="142"/>
      <operator activated="true" class="converters:confusionmatrix_2_example_set" compatibility="0.5.000" expanded="true" height="82" name="Confusion Matrix to ExampleSet" width="90" x="447" y="136"/>
      <operator activated="true" class="performance_to_data" compatibility="9.4.000-BETA" expanded="true" height="82" name="Performance to Data" width="90" x="447" y="238"/>
      <operator activated="true" class="select_attributes" compatibility="9.4.000-BETA" expanded="true" height="82" name="Select Attributes" width="90" x="581" y="238">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="Criterion|Value"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
      </operator>
      <operator activated="true" class="filter_example_range" compatibility="9.4.000-BETA" expanded="true" height="82" name="Filter Example Range" width="90" x="715" y="238">
        <parameter key="first_example" value="2"/>
        <parameter key="last_example" value="3"/>
        <parameter key="invert_filter" value="false"/>
      </operator>
      <operator activated="true" class="transpose" compatibility="9.4.000-BETA" expanded="true" height="82" name="Transpose" width="90" x="849" y="238"/>
      <operator activated="true" class="rename_by_example_values" compatibility="9.4.000-BETA" expanded="true" height="82" name="Rename by Example Values" width="90" x="983" y="238">
        <parameter key="row_number" value="1"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="9.4.000-BETA" expanded="true" height="82" name="Select Attributes (2)" width="90" x="1117" y="187">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value="id"/>
        <parameter key="attributes" value="id|Criterion"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="true"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="operator_toolbox:merge" compatibility="2.1.000" expanded="true" height="103" name="Merge Attributes" width="90" x="1251" y="85">
        <parameter key="handling_of_duplicate_attributes" value="rename"/>
        <parameter key="handling_of_special_attributes" value="keep_first_special_other_regular"/>
        <parameter key="handling_of_duplicate_annotations" value="rename"/>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Cross Validation" to_port="example set"/>
      <connect from_op="Cross Validation" from_port="example set" to_port="result 1"/>
      <connect from_op="Cross Validation" from_port="performance 1" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Confusion Matrix to ExampleSet" to_port="per"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Performance to Data" to_port="performance vector"/>
      <connect from_op="Confusion Matrix to ExampleSet" from_port="exa" to_op="Merge Attributes" to_port="example set 1"/>
      <connect from_op="Performance to Data" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Filter Example Range" to_port="example set input"/>
      <connect from_op="Filter Example Range" from_port="example set output" to_op="Transpose" to_port="example set input"/>
      <connect from_op="Transpose" from_port="example set output" to_op="Rename by Example Values" to_port="example set input"/>
      <connect from_op="Rename by Example Values" from_port="example set output" to_op="Select Attributes (2)" to_port="example set input"/>
      <connect from_op="Select Attributes (2)" from_port="example set output" to_op="Merge Attributes" to_port="example set 2"/>
      <connect from_op="Merge Attributes" from_port="merged set" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

In you example Weighted Mean Recall has the same value as Accuracy. In my process, Weighted Mean Recall has exactly 10x smaller value than Accuracy. Why?

Hi @Serek91,

Mmmh, at first sight it is strange (but interesting) ....
I'm considering the picture you shared in your first post : 
 - accuracy = 46,49 %
 - When I average the different class recall in your results (52 %, 46%, 26% etc.) I obtain weighted_mean_recall = 46,49 % (weights = 1, 1,1 etc.) = accuracy.
Anyways to understand what's going on, can you share please : 
 - your data
 - your process (XML)

Regards,

Lionel