"[Solved] How to work on Correlation Matrix results"

TomTom

Hello,

I have made a correlation matrix within my process and would like to use the Matrix results in this process. Unfortunately the only operators I've found which can use "mat" as input are the ones in Reporting addon. So I can write an Excel File, for example. I couldn't find a way to use standard operators like "Filter examples" or  "Select" with the Matrix as input. Is there a way to filter the matrix for certain values without writing it to a file and reading the file within the process (This wouldn't be fast enough, because I have very very large data sets)?

I've tried also another way: I've used "Remove correlated attributes" instead "Correlation Matrix", and set filter relation to "less", to get attribute pairs which are correlating each other, but the results are confusing me:

Sometimes the result of "Remove correlated attributes" is a resultset with just one column. If I have a Result Set with two attributes: A and B and also some other attributes and column A has a high correlation to another column. Why is "Remove correlated attributes" just returning one of both columns? I would expect it to return both columns, because Correlation is a bidirectional relationship.

It would be really great, if anyone could help on this issue.

TomTom

As I haven't found a clean solution, I have chosen to write Correlation Matrix to file and reload it again. Very dirty, but it works. Please let me know, if there's a cleaner solution.

Here is an example process of what I have done:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.000" expanded="true" name="Process">
    <process expanded="true" height="656" width="681">
      <operator activated="true" class="generate_data" compatibility="5.3.000" expanded="true" height="60" name="Generate Data" width="90" x="45" y="30">
        <parameter key="number_of_attributes" value="20"/>
      </operator>
      <operator activated="true" class="correlation_matrix" compatibility="5.3.000" expanded="true" height="94" name="Correlation Matrix" width="90" x="179" y="30"/>
      <operator activated="true" class="write_as_text" compatibility="5.3.000" expanded="true" height="76" name="Write as Text" width="90" x="313" y="30">
        <parameter key="result_file" value="C:\TEMP\test.csv"/>
      </operator>
      <operator activated="true" class="read_csv" compatibility="5.3.000" expanded="true" height="60" name="Read CSV" width="90" x="179" y="165">
        <parameter key="csv_file" value="C:\TEMP\test.csv"/>
        <parameter key="column_separators" value="\s"/>
        <parameter key="comment_characters" value=""/>
        <parameter key="first_row_as_names" value="false"/>
        <list key="annotations"/>
        <parameter key="encoding" value="windows-1252"/>
        <list key="data_set_meta_data_information">
          <parameter key="0" value="att1.true.polynominal.attribute"/>
          <parameter key="1" value="att2.true.polynominal.attribute"/>
          <parameter key="2" value="att3.true.polynominal.attribute"/>
          <parameter key="3" value="att4.true.polynominal.attribute"/>
          <parameter key="4" value="att5.true.polynominal.attribute"/>
          <parameter key="5" value="att6.true.polynominal.attribute"/>
          <parameter key="6" value="att7.true.polynominal.attribute"/>
          <parameter key="7" value="att8.true.polynominal.attribute"/>
          <parameter key="8" value="att9.true.polynominal.attribute"/>
          <parameter key="9" value="att10.true.polynominal.attribute"/>
        </list>
      </operator>
      <operator activated="true" class="filter_example_range" compatibility="5.3.000" expanded="true" height="76" name="Filter Example Range" width="90" x="313" y="165">
        <parameter key="first_example" value="3"/>
        <parameter key="last_example" value="12"/>
      </operator>
      <connect from_op="Generate Data" from_port="output" to_op="Correlation Matrix" to_port="example set"/>
      <connect from_op="Correlation Matrix" from_port="matrix" to_op="Write as Text" to_port="input 1"/>
      <connect from_op="Write as Text" from_port="input 1" to_port="result 1"/>
      <connect from_op="Read CSV" from_port="output" to_op="Filter Example Range" to_port="example set input"/>
      <connect from_op="Filter Example Range" from_port="example set output" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>