Maximum size of input

fervlrm
fervlrm New Altair Community Member
edited November 5 in Community Q&A
Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?

Thanks
Tagged:

Answers

  • fervlrm
    fervlrm New Altair Community Member
    In fact,

    I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
    any solution?
  • vijaypshah
    vijaypshah New Altair Community Member
    Hi,
    Simple Solution: Use 64 bit machine and increase the RAM memory..

    I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

    Regards,
    Vijay
  • IngoRM
    IngoRM New Altair Community Member
    Hi,

    yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

    Cheers,
    Ingo

Categories