[SOLVED] Read CSV BUG: Two Consecutive Missing Columns Causes Read Error

dragoljub

Hi Rapid Miner Gurus,

Using RapidMiner 5.2.006x64 on Windows 7x64 I am running into a problem with the Read CSV operator. I do not recall this being a problem in the past versions but somehow it has cropped into the code.

I have a simple example data set test.csv that has 4 columns, two that are missing and next to each other as the last two columns. This problem occurs when there are two or more consecutive missing column's at the right end of the table. Excel reads this CSV fine and so do most other data analysis tools. However RapidMiner throws an error.  ???

A,B,C,D
1,2,,
3,4,,
5,6,,

SEVERE: Process failed: An attribute att4 was specified for column 3, but this column does not exist in input data.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.006">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.006" expanded="true" name="Process">
    <process expanded="true" height="145" width="145">
      <operator activated="true" class="read_csv" compatibility="5.2.006" expanded="true" height="60" name="Read CSV" width="90" x="45" y="30">
        <parameter key="csv_file" value="C:\test.csv"/>
        <parameter key="column_separators" value=","/>
        <parameter key="use_quotes" value="false"/>
        <list key="annotations"/>
        <list key="data_set_meta_data_information"/>
        <parameter key="read_not_matching_values_as_missings" value="false"/>
      </operator>
      <connect from_op="Read CSV" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

NOTE: This problem can be somewhat avoided if you EXPLICITLY set the column index and type for every attribute in the csv file. Unfortunately this is not an option because most of the time you are reading a data set that either has too many columns to manually set or has a changing number of columns some of which may be missing. RapidMiner handled missing columns in CSVs very well before so I hope it just a little bug.  ;D

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.006">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.006" expanded="true" name="Process">
    <process expanded="true" height="145" width="145">
      <operator activated="true" class="read_csv" compatibility="5.2.006" expanded="true" height="60" name="Read CSV" width="90" x="45" y="30">
        <parameter key="csv_file" value="C:\Users\ddrmanac\Desktop\test.csv"/>
        <parameter key="column_separators" value=","/>
        <parameter key="use_quotes" value="false"/>
        <list key="annotations"/>
        <list key="data_set_meta_data_information">
          <parameter key="0" value=".true.nominal.attribute"/>
          <parameter key="1" value=".true.integer.attribute"/>
          <parameter key="2" value=".true.integer.attribute"/>
          <parameter key="3" value=".true.integer.attribute"/>
        </list>
        <parameter key="read_not_matching_values_as_missings" value="false"/>
      </operator>
      <connect from_op="Read CSV" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Nils_Woehler

Hi,

thank you for your detailed report. This bug was already fixed in the latest version 5.2.008.
Please update your RapidMiner to this version and reading CSV files with missings at the end of a line shouldn't be a problem anymore.

Best,
Nils

dragoljub

Great! I am just about to upgrade! Thanks for the amazing work RM Team!  ;D

I can confirm that the upgrading to 5.2.008 fixed the issue.

Thanks,
-Gagi

jaybusiness86

Hi Team,

 

I get the attribute error while running the process.

 

Attached the snapshot, additionally the error is "Th attribute att2specified in the old_name parameter doesnot exist in the input data."

 

 

sgenzer

hello @jaybusiness86 - welcome to the community.  This is an old thread.  I would highly recommend reading the info in the "Getting Started Forum" and reposting there with your process etc...

 

Good luck!

 

Scott