Write CSV: append mode?

dcubed

I am trying to use a WriteCSV process to write the output of an ApplyModel process within a Validation process.
I find that each ApplyModel output overwrites the content of my CSV file.
Is there a recommended mechanism to have WriteCSV create OR append to its output file?

If writing to a CSV or database isn't the best way of collecting the predictions generated by the ApplyModel during testing, what is suggested?

      <operator activated="true" class="series:sliding_window_validation" compatibility="5.1.002" expanded="true" height="112" name="Validation (2)" width="90" x="246" y="165">
        <parameter key="training_window_step_size" value="1"/>
        <parameter key="test_window_width" value="1"/>
        <parameter key="parallelize_training" value="true"/>
        <parameter key="parallelize_testing" value="true"/>
        <process expanded="true" height="466" width="373">
          <operator activated="true" class="neural_net" compatibility="5.2.003" expanded="true" height="76" name="Neural Net" width="90" x="42" y="85">
            <list key="hidden_layers"/>
          </operator>
          <connect from_port="training" to_op="Neural Net" to_port="training set"/>
          <connect from_op="Neural Net" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="466" width="373">
          <operator activated="true" class="apply_model" compatibility="5.2.003" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="5.2.003" expanded="true" height="94" name="Multiply" width="90" x="67" y="113"/>
          <operator activated="true" class="write_csv" compatibility="5.2.003" expanded="true" height="76" name="Write CSV" width="90" x="246" y="165">
            <parameter key="csv_file" value="c:\temp\p1.csv"/>
            <parameter key="column_separator" value=","/>
          </operator>
          <operator activated="true" class="series:forecasting_performance" compatibility="5.1.002" expanded="true" height="76" name="Performance" width="90" x="209" y="30">
            <parameter key="horizon" value="1"/>
          </operator>
          <operator activated="true" class="write_access" compatibility="5.2.003" expanded="true" height="76" name="Write Access" width="90" x="56" y="273">
            <parameter key="database_file" value="c:\temp\RapidMiner.mdb"/>
            <parameter key="table_name" value="p1"/>
            <parameter key="overwrite_mode" value="append"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Multiply" from_port="output 2" to_op="Write CSV" to_port="input"/>
          <connect from_op="Write CSV" from_port="through" to_op="Write Access" to_port="input"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>

MariusHelf

Hi,

for csv files it's currently not possible to append contents. Does Write Access also overwrite the database file, even if overwrite_mode is set to "append"?

Why don't you want to collect the predictions in the process? That would be the best option, if your data is not too large to fit into memory.

Btw, it would be more helpful if you posted your complete process xml. Xml of single operators can't be easily pasted into RapidMiner.

Best, Marius