Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

A bug in normalize operator and in bugzilla

First bug is related to Bugzilla, when I've try to file a bug a receive following error:

Software error:

Cannot determine local time zone
For help, please send mail to the webmaster (webmaster@bugs.rapid-i.com), giving this error message and the time and date of the error.

So I decided to submit it here, as it looks like an important bug. It is related to normalize operator and independence of its output ports.
The process below shows the problem. The process simply loads the data and performs normalization. As a result only data received from res0 should be normalized, and the data received from res1 should contain the original data, but in fact both outputs are normalized. The funny thing is that when I connect "ori" output port to the process output everything works fine.
The process:


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.0.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="53" y="202">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="7.0.001" expanded="true" height="103" name="Multiply" width="90" x="199" y="197"/>
      <operator activated="true" class="normalize" compatibility="7.0.001" expanded="true" height="103" name="Normalize" width="90" x="351" y="34">
        <parameter key="method" value="range transformation"/>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Multiply" from_port="output 2" to_port="result 2"/>
      <connect from_op="Normalize" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Best

Marcin

Find more posts tagged with

AI Studio

Bug Report

Accepted answers

All comments

JEdward

That's due to the way RapidMiner handles data to use less memory.
Normally when you are building a RapidMiner process each operator is applied as rules to the data line (for example 'Generate Attributes'). This makes it efficient when running the process as only one set of the data needs to be stored in-memory. Multiply doesn't create a new copy of the underlying data, it just separates the streams.

Some operations such as Normalise & Obfuscate change the underlying data in-memory so to resolve this use the Materialize Data operator to create a new exampleset in-memory. See below for your example process with the Materialize Data operator added.
This is pretty much what the Ori output port is doing in the normalise operator.

You can find an indepth explanation on how this works in the How to Extend RapidMiner guides, but I agree that this could be made clearer by RapidMiner itself.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.0.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="53" y="202">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="7.0.001" expanded="true" height="103" name="Multiply" width="90" x="199" y="197"/>
      <operator activated="true" class="materialize_data" compatibility="7.0.001" expanded="true" height="82" name="Materialize Data" width="90" x="246" y="34"/>
      <operator activated="true" class="normalize" compatibility="7.0.001" expanded="true" height="103" name="Normalize" width="90" x="380" y="34">
        <parameter key="method" value="range transformation"/>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Materialize Data" to_port="example set input"/>
      <connect from_op="Multiply" from_port="output 2" to_port="result 2"/>
      <connect from_op="Materialize Data" from_port="example set output" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Normalize" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="168"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

marcin_blachnik

Thank you for your answer but it is a bug.
My code (independence of outputs) works perfectly on RapidMiner 5 and RapidMiner 6, it just doesn’t work on RapidMiner 7.
There is a rule which says that whenever an operator modifies a data it creates a new attribute in the exampleTable and in the exampleSet makes a switch such that the exampleSet reference to the new modified attribute (switch a view).
I just guess that the developer didn’t clone the exampleSet before applying modifications so all of the modification are propagated in every copy of the dataset.

MartinLiebig

Hey,

do you know about Materialize Data?

~Martin

marcin_blachnik

Well

I know what is Materialize data and how for my own purpose solve this provlem,
but please explain me why
1) In RM7.0 this issue appear, and in RM 6.5 everything works ok. Does it mean that RM7 is not compatible with RM 6.5 and processes have to be rechecked
2) There is no line in the help that says about this change, and about the requirement of materialization
3) Why Normalization operator works differently according to its output port connection. It works ok when you connect "ori" output somewhere else and then the data on the input side is not being modified, (in other words also the data on the output of the "multiply" operator is not normalized), but when the "ori" output is not connected then Normalization modifies also the exampleset on the input side.

To show what I'm talking about in 3) take a look at this process:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.0.001" expanded="true" height="68" name="Retrieve Iris" width="90" x="69" y="201">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="7.0.001" expanded="true" height="103" name="Multiply" width="90" x="216" y="202"/>
      <operator activated="true" class="normalize" compatibility="7.0.001" expanded="true" height="103" name="Normalize" width="90" x="383" y="135">
        <parameter key="method" value="range transformation"/>
      </operator>
      <operator activated="true" class="parallel_decision_tree" compatibility="7.0.001" expanded="true" height="82" name="Decision Tree" width="90" x="386" y="259"/>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Decision Tree" to_port="training set"/>
      <connect from_op="Normalize" from_port="example set output" to_port="result 3"/>
      <connect from_op="Normalize" from_port="original" to_port="result 1"/>
      <connect from_op="Decision Tree" from_port="model" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
    </process>
  </operator>
</process>

Now everything works perfect and the tree has correct values on edges, but when you remove the top most process output (connection between ori and process output) you obtain completely different tree. Even more funny thing is that this issue don't appear on Golf dataset it's just a problem with Iris (both taken from the "samples" repository) I guess that this is because in iris all regular features are numeric.