Community & Support
Learn
Marketplace
Discussions
Categories
Discussions
General
Platform
Academic
Partner
Regional
User Groups
Documentation
Events
Altair Exchange
Share or Download Projects
Resources
News & Instructions
Programs
YouTube
Employee Resources
This tab can be seen by employees only. Please do not share these resources externally.
Groups
Join a User Group
Support
Altair RISE
A program to recognize and reward our most engaged community members
Nominate Yourself Now!
Home
Discussions
Altair RapidMiner
large data sample handling issue
bingojosjtu
Hi
I encounter a problem recently when using outlier detection funcition (LOF to be specific).
Condition:
My data sample is about 178000 in total samples and around 10-12 attributes.
My computer has 8 GB RAM and i7 2600 CPU. Hard disk enough space.
Scenario:
I let the program run overnight, but the next morning, the program says that it can not handle the process and the computer memory is too small for this task.
it stopped at outlier detection step, which I know is a very slow process but I did not expect it refuse to complete due to memory size.
Question:
My question is, for a given sample size and attributes number, how am I suppose to know the memory requirement or say upper limit of a particular procedure before hand?
Q2: Is there any way to solve this issue other than shrink my data sample size at current stage?
Q3: What if I increase my RAM to 16 or 32 GB, does it help to solve the issue?
BTW, I have submitted the job on the cloud server (32 GB version), hope with the help of your computation source, this issue can be solved.
Thank you!
RMer
Find more posts tagged with
AI Studio
Sampling
Comments
There are no comments yet
Quick Links
All Categories
Recent Discussions
Activity
My Discussions
Unanswered
日本語 (Japanese)
한국어(Korean)
Groups