🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Automodel - large (?) CSV dataset memory issues

User: "baeikene"
baeikene
New Altair Community Member
Updated by Jocelyn

I'm doing research on the CIC IDS 2017 dataset which contains 200-300MB of data for one file.

I try to do a automodel and predict the source IP based on other attributes. I get into memory issues running this (I have 16GB RAM) but I assume that I have used a too large dataset or too many attributes for the modeling.

So my question is what number of lines and attributes can I expect to be handled doing this?

 

Find more posts tagged with

AI Studio
CSV
Auto Model
Sort by:
1 - 2 of 21
    User: "rfuentealba"
    rfuentealba
    New Altair Community Member

    Hello,

     

    RapidMiner is a bit resource hungry, but it shouldn't be a problem to load a large file like that one. I have 4 Gb of RAM on my MacBook Air and can load the file.

     

    The thing is that with such a limited amount of memory, I usually do three things to maximize:

     

    1. I write the CSV into an IOObject.
    2. I split my model into many smaller models that still can make sense for the user/developer.
    3. I use the Free Memory operator when I cannot split a model into two or three.

    However, I also do tune my RapidMiner Studio installation to use more memory. In this case:

     

    Preferences > System > Data Management. I configure that number to be at least twice the amount of training data.

     

    Screen Shot 2018-09-27 at 01.58.53.png

     

    Hope this helps,

     

    Screen Shot 2018-09-27 at 01.58.53.png
    User: "SGolbert"
    SGolbert
    New Altair Community Member

    Hi,

     

    can you tell us in which step of Automodel this happens? If you read the file and save it as IOObject in the repository, do you still have the same problems?

     

    You can also try avoiding Deep Learning or Gradient Boosting Models, which are very resource consuming.

     

    Cheers,

    Sebastian

Quick Links