how to import multiple files

Hi,
you can use "loop files" operator. Than you will probably need "append" operator to merge all example set from collection to one.

Best,
Vaclav

Can you post a simple example which reads 3 csv files from a single directory and then appends them into one file and stores it.

This process reads three csv files from the directory /Users/Ernesto/Desktop/files and appends them.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
    <process expanded="true" height="400" width="659">
      <operator activated="true" class="loop_files" compatibility="5.2.008" expanded="true" height="76" name="Loop Files" width="90" x="246" y="30">
        <parameter key="directory" value="/Users/Ermesto/Desktop/files"/>
        <process expanded="true" height="418" width="677">
          <operator activated="true" class="read_csv" compatibility="5.2.008" expanded="true" height="60" name="Read CSV" width="90" x="112" y="30">
            <parameter key="csv_file" value="/Users/Carlos/Desktop/files/file01.csv"/>
            <parameter key="column_separators" value=","/>
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <parameter key="encoding" value="MacRoman"/>
            <list key="data_set_meta_data_information">
              <parameter key="0" value="y.true.integer.attribute"/>
              <parameter key="1" value="x.true.integer.attribute"/>
            </list>
          </operator>
          <connect from_port="file object" to_op="Read CSV" to_port="file"/>
          <connect from_op="Read CSV" from_port="output" to_port="out 1"/>
          <portSpacing port="source_file object" spacing="0"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="append" compatibility="5.2.008" expanded="true" height="76" name="Append" width="90" x="447" y="30"/>
      <connect from_op="Loop Files" from_port="out 1" to_op="Append" to_port="example set 1"/>
      <connect from_op="Append" from_port="merged set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Create the following process:

 

Loop Files -> Append -> Write CSV

 

Click on Loop Files, define the parameter directory to point to the directory where your CSV files (or other files readable by rapidminer) are.

Double click on Loop Files to go into this sub-process.

 

Create the following sub process:

fil -> Read CSV -> Select Attributes -> out

 

Click on Select Attributes and select the parameter attribute filter type to either subset or regular_expression.

For subset, click on the Select Attributes... button, and add the attributes (columns) of your CSVs that you want to have in your merged output. Add them in the right list of the window by typing the name and clicking the plus icon.

For regular_expression you can define a list of attriutes (columns) like this: .*attribute1.*|.*attribute2.*|.*attribute3.*

example:

 

.*mail.*|.*date_submitted.*|.*page_url.*

Then you are done.

 

 

Merge columns with different names into the same column (attribute):

In case you have columns in your CSV with different naming, like: e-mail, eMail, e_mail you can do the following:

In your existing Select Attributes Object choose regular_expressions. Define a regular_expression that contains all columns you want, and also the variants. If I have the following columns:

• E_mail
• e-Mail
• Mail
• date_submitted
• name

I would create the following regular_expression:

 

 

.*mail.*|.*date_submitted.*|.*name.*

This will still create an output with different columns (attribute). To merge the 3 Email columns into one, you have to rename them to be identical. Add a Rename by Replacing Object after the already existing Select Attributes.

 

On the Rename by Replacing Object select regular_expression as the attribute filter type. Then fill out the fields below like:

regular expression: .*mail.*

replace what: .*

replace by: mail

 

I attached a gif of my process, to clarify.

very nice explanation, @underlines !

 

Scott

 

 

Great answer @underlines!

Could someone please suggest what to do:
if the columns being imported don't have column headers? or even,
What should be provided in the sub-process while setting the filter for Select Attribute, in the line above?

Hi @penmen,

When you import a file without headers, it will assign default names of att1, att2, att3 and so on. If the order of attributes is always the same , then it doesn't cause a problem, you can just select the attributes you need (e.g. att1 and att3), and then rename the attributes after the append described above. If there's isn't a consistent order, then you will probably have to implement some more complicated logic to assign the correct attribute names prior to the Select Attributes step.

Hope this helps,

Best,

Roland

Thanks @rjones13 !
I was able to resolve my issue above by setting 'all attributes' in the Select Attribute first.
Then, after Renaming by Generic names, I was able to any required column with 'Select Attributes' and by providing the concatenated string value of my generic name + column number.

Hello there
In my experience first organize all the CSV file in a single folder then open RapidMiner and explore the Repository choose Import Data from the toolbar then select Import Data from Files organize the schema make sure the option is selected then confirm setting