Process running out of memory too easily?

brett_800
brett_800 New Altair Community Member
edited November 5 in Community Q&A
I cannot seem to get my process flow to execute, as I get a warning saying that the process cannot fit into memory.  I've tried all sorts of ways to reduce the data set, including just using a subset of 200 examples and attribute reduction using forward selection.  I'm still getting this problem.  I have a fairly good computer with 16GB of DDR3 RAM.

The xml is shown below (I have only kept the first 100 attributes, out of the 800 attributes listed in order to meet the posting requirements of 20,000 characters):

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.012">
 <context>
   <input/>
   <output/>
   <macros/>
 </context>
 <operator activated="true" class="process" compatibility="5.3.012" expanded="true" name="Process">
   <process expanded="true">
     <operator activated="true" class="read_excel" compatibility="5.3.012" expanded="true" height="60" name="Read Excel" width="90" x="45" y="30">
       <parameter key="excel_file" value="C:\Users\Brett\Dropbox\CFB Model\2013 work\DATATRAIN_1_GENERIC\GT-CFB-COMPLETE.xlsx"/>
       <parameter key="imported_cell_range" value="A1:FXSHRXX-2147483648"/>
       <parameter key="first_row_as_names" value="false"/>
       <list key="annotations">
         <parameter key="0" value="Name"/>
       </list>
       <list key="data_set_meta_data_information">
         <parameter key="0" value="GameCode.true.integer.attribute"/>
         <parameter key="1" value="Year.true.integer.attribute"/>
         <parameter key="2" value="Round.true.integer.attribute"/>
         <parameter key="3" value="Date.true.date_time.attribute"/>
         <parameter key="4" value="HT_Code.true.polynominal.attribute"/>
         <parameter key="5" value="AT_Code.true.polynominal.attribute"/>
         <parameter key="6" value="Home.true.integer.attribute"/>
         <parameter key="7" value="Away.true.integer.attribute"/>
         <parameter key="8" value="HLn.true.polynominal.attribute"/>
         <parameter key="9" value="ScoreDiff.true.integer.attribute"/>
         <parameter key="10" value="Hwin_ATS.true.polynominal.attribute"/>
         <parameter key="11" value="Hm_Cover.true.polynominal.attribute"/>
         <parameter key="12" value="TOT_Ln.true.polynominal.attribute"/>
         <parameter key="13" value="TOT_POINTS.true.integer.attribute"/>
         <parameter key="14" value="OverWinATS.true.polynominal.attribute"/>
         <parameter key="15" value="Ov_Cover.true.polynominal.attribute"/>
         <parameter key="16" value="Hm_ATS_LINE_5.true.numeric.attribute"/>
         <parameter key="17" value="Hm_ATS_LINE_4.true.numeric.attribute"/>
         <parameter key="18" value="Hm_ATS_LINE_3.true.numeric.attribute"/>
         <parameter key="19" value="Hm_ATS_LINE_2.true.numeric.attribute"/>
         <parameter key="20" value="Hm_ATS_LINE_1.true.polynominal.attribute"/>
         <parameter key="21" value="Hm_TOTALS_LINE_5.true.numeric.attribute"/>
         <parameter key="22" value="Hm_TOTALS_LINE_4.true.numeric.attribute"/>
         <parameter key="23" value="Hm_TOTALS_LINE_3.true.numeric.attribute"/>
         <parameter key="24" value="Hm_TOTALS_LINE_2.true.numeric.attribute"/>
         <parameter key="25" value="Hm_TOTALS_LINE_1.true.polynominal.attribute"/>
         <parameter key="26" value="Hm_LTR_5.true.real.attribute"/>
         <parameter key="27" value="Hm_LTR_4.true.real.attribute"/>
         <parameter key="28" value="Hm_LTR_3.true.real.attribute"/>
         <parameter key="29" value="Hm_LTR_2.true.real.attribute"/>
         <parameter key="30" value="Hm_LTR_1.true.polynominal.attribute"/>
         <parameter key="31" value="Hm_SU_MOV_5.true.integer.attribute"/>
         <parameter key="32" value="Hm_SU_MOV_4.true.integer.attribute"/>
         <parameter key="33" value="Hm_SU_MOV_3.true.integer.attribute"/>
         <parameter key="34" value="Hm_SU_MOV_2.true.integer.attribute"/>
         <parameter key="35" value="Hm_SU_MOV_1.true.integer.attribute"/>
         <parameter key="36" value="Hm_ATS_MOV_5.true.numeric.attribute"/>
         <parameter key="37" value="Hm_ATS_MOV_4.true.numeric.attribute"/>
         <parameter key="38" value="Hm_ATS_MOV_3.true.numeric.attribute"/>
         <parameter key="39" value="Hm_ATS_MOV_2.true.numeric.attribute"/>
         <parameter key="40" value="Hm_ATS_MOV_1.true.polynominal.attribute"/>
         <parameter key="41" value="Hm_POINTS_FOR_5.true.integer.attribute"/>
         <parameter key="42" value="Hm_POINTS_FOR_4.true.integer.attribute"/>
         <parameter key="43" value="Hm_POINTS_FOR_3.true.integer.attribute"/>
         <parameter key="44" value="Hm_POINTS_FOR_2.true.integer.attribute"/>
         <parameter key="45" value="Hm_POINTS_FOR_1.true.integer.attribute"/>
         <parameter key="46" value="Hm_POINTS_AGAINST_5.true.integer.attribute"/>
         <parameter key="47" value="Hm_POINTS_AGAINST_4.true.integer.attribute"/>
         <parameter key="48" value="Hm_POINTS_AGAINST_3.true.integer.attribute"/>
         <parameter key="49" value="Hm_POINTS_AGAINST_2.true.integer.attribute"/>
         <parameter key="50" value="Hm_POINTS_AGAINST_1.true.integer.attribute"/>
         <parameter key="51" value="Hm_OVER_ATS_5.true.integer.attribute"/>
         <parameter key="52" value="Hm_OVER_ATS_4.true.integer.attribute"/>
         <parameter key="53" value="Hm_OVER_ATS_3.true.integer.attribute"/>
         <parameter key="54" value="Hm_OVER_ATS_2.true.integer.attribute"/>
         <parameter key="55" value="Hm_OVER_ATS_1.true.integer.attribute"/>
         <parameter key="56" value="Hm_OVER_ATS_MOV_5.true.numeric.attribute"/>
         <parameter key="57" value="Hm_OVER_ATS_MOV_4.true.numeric.attribute"/>
         <parameter key="58" value="Hm_OVER_ATS_MOV_3.true.numeric.attribute"/>
         <parameter key="59" value="Hm_OVER_ATS_MOV_2.true.numeric.attribute"/>
         <parameter key="60" value="Hm_OVER_ATS_MOV_1.true.polynominal.attribute"/>
         <parameter key="61" value="Hm_PFTOTOTALSRATIO_5.true.real.attribute"/>
         <parameter key="62" value="Hm_PFTOTOTALSRATIO_4.true.real.attribute"/>
         <parameter key="63" value="Hm_PFTOTOTALSRATIO_3.true.real.attribute"/>
         <parameter key="64" value="Hm_PFTOTOTALSRATIO_2.true.real.attribute"/>
         <parameter key="65" value="Hm_PFTOTOTALSRATIO_1.true.numeric.attribute"/>
         <parameter key="66" value="Hm_PA_TO_TOTALS_RATIO_5.true.real.attribute"/>
         <parameter key="67" value="Hm_PA_TO_TOTALS_RATIO_4.true.real.attribute"/>
         <parameter key="68" value="Hm_PA_TO_TOTALS_RATIO_3.true.real.attribute"/>
         <parameter key="69" value="Hm_PA_TO_TOTALS_RATIO_2.true.real.attribute"/>
         <parameter key="70" value="Hm_PA_TO_TOTALS_RATIO_1.true.numeric.attribute"/>
         <parameter key="71" value="Hm_ATS_MOV_Q1_5.true.polynominal.attribute"/>
         <parameter key="72" value="Hm_ATS_MOV_Q1_4.true.polynominal.attribute"/>
         <parameter key="73" value="Hm_ATS_MOV_Q1_3.true.polynominal.attribute"/>
         <parameter key="74" value="Hm_ATS_MOV_Q1_2.true.polynominal.attribute"/>
         <parameter key="75" value="Hm_ATS_MOV_Q1_1.true.polynominal.attribute"/>
         <parameter key="76" value="Hm_ATS_MOV_Q4_5.true.polynominal.attribute"/>
         <parameter key="77" value="Hm_ATS_MOV_Q4_4.true.polynominal.attribute"/>
         <parameter key="78" value="Hm_ATS_MOV_Q4_3.true.polynominal.attribute"/>
         <parameter key="79" value="Hm_ATS_MOV_Q4_2.true.polynominal.attribute"/>
         <parameter key="80" value="Hm_ATS_MOV_Q4_1.true.polynominal.attribute"/>
         <parameter key="81" value="Hm_ATS_MOV_H1_5.true.polynominal.attribute"/>
         <parameter key="82" value="Hm_ATS_MOV_H1_4.true.polynominal.attribute"/>
         <parameter key="83" value="Hm_ATS_MOV_H1_3.true.polynominal.attribute"/>
         <parameter key="84" value="Hm_ATS_MOV_H1_2.true.polynominal.attribute"/>
         <parameter key="85" value="Hm_ATS_MOV_H1_1.true.polynominal.attribute"/>
         <parameter key="86" value="Hm_ATS_MOV_H2_5.true.polynominal.attribute"/>
         <parameter key="87" value="Hm_ATS_MOV_H2_4.true.polynominal.attribute"/>
         <parameter key="88" value="Hm_ATS_MOV_H2_3.true.polynominal.attribute"/>
         <parameter key="89" value="Hm_ATS_MOV_H2_2.true.polynominal.attribute"/>
         <parameter key="90" value="Hm_ATS_MOV_H2_1.true.polynominal.attribute"/>
         <parameter key="91" value="Hm_TOTALS_MOV_Q1_5.true.polynominal.attribute"/>
         <parameter key="92" value="Hm_TOTALS_MOV_Q1_4.true.polynominal.attribute"/>
         <parameter key="93" value="Hm_TOTALS_MOV_Q1_3.true.polynominal.attribute"/>
         <parameter key="94" value="Hm_TOTALS_MOV_Q1_2.true.polynominal.attribute"/>
         <parameter key="95" value="Hm_TOTALS_MOV_Q1_1.true.polynominal.attribute"/>
         <parameter key="96" value="Hm_TOTALS_MOV_Q4_5.true.polynominal.attribute"/>
         <parameter key="97" value="Hm_TOTALS_MOV_Q4_4.true.polynominal.attribute"/>
         <parameter key="98" value="Hm_TOTALS_MOV_Q4_3.true.polynominal.attribute"/>
         <parameter key="99" value="Hm_TOTALS_MOV_Q4_2.true.polynominal.attribute"/>
         <parameter key="100" value="Hm_TOTALS_MOV_Q4_1.true.polynominal.attribute"/>
       </list>
     </operator>
     <operator activated="true" class="declare_missing_value" compatibility="5.3.012" expanded="true" height="76" name="Declare Missing Value" width="90" x="179" y="120">
       <parameter key="attribute" value="Hm_Cover"/>
       <parameter key="include_special_attributes" value="true"/>
       <parameter key="mode" value="nominal"/>
       <parameter key="nominal_value" value="NULL"/>
       <parameter key="expression_value" value="&quot;NULL&quot;"/>
     </operator>
     <operator activated="true" class="filter_examples" compatibility="5.3.012" expanded="true" height="76" name="Filter Examples" width="90" x="45" y="210">
       <parameter key="condition_class" value="no_missing_labels"/>
     </operator>
     <operator activated="true" class="sample" compatibility="5.3.012" expanded="true" height="76" name="Sample" width="90" x="179" y="210">
       <parameter key="sample_size" value="200"/>
       <list key="sample_size_per_class"/>
       <list key="sample_ratio_per_class"/>
       <list key="sample_probability_per_class"/>
     </operator>
     <operator activated="true" class="normalize" compatibility="5.3.012" expanded="true" height="94" name="Normalize" width="90" x="45" y="300"/>
     <operator activated="true" class="optimize_selection_forward" compatibility="5.3.012" expanded="true" height="94" name="Forward Selection" width="90" x="179" y="300">
       <parameter key="maximal_number_of_attributes" value="15"/>
       <process expanded="true">
         <operator activated="true" class="split_validation" compatibility="5.3.012" expanded="true" height="112" name="Validation" width="90" x="112" y="30">
           <process expanded="true">
             <operator activated="true" class="nominal_to_numerical" compatibility="5.3.012" expanded="true" height="94" name="Nominal to Numerical" width="90" x="45" y="30">
               <list key="comparison_groups"/>
             </operator>
             <operator activated="true" class="support_vector_machine_libsvm" compatibility="5.3.012" expanded="true" height="76" name="SVM" width="90" x="246" y="30">
               <list key="class_weights"/>
             </operator>
             <connect from_port="training" to_op="Nominal to Numerical" to_port="example set input"/>
             <connect from_op="Nominal to Numerical" from_port="example set output" to_op="SVM" to_port="training set"/>
             <connect from_op="SVM" from_port="model" to_port="model"/>
             <portSpacing port="source_training" spacing="0"/>
             <portSpacing port="sink_model" spacing="0"/>
             <portSpacing port="sink_through 1" spacing="0"/>
           </process>
           <process expanded="true">
             <operator activated="true" class="apply_model" compatibility="5.3.012" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
               <list key="application_parameters"/>
             </operator>
             <operator activated="true" class="performance" compatibility="5.3.012" expanded="true" height="76" name="Performance" width="90" x="278" y="30"/>
             <connect from_port="model" to_op="Apply Model" to_port="model"/>
             <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
             <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
             <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
             <portSpacing port="source_model" spacing="0"/>
             <portSpacing port="source_test set" spacing="0"/>
             <portSpacing port="source_through 1" spacing="0"/>
             <portSpacing port="sink_averagable 1" spacing="0"/>
             <portSpacing port="sink_averagable 2" spacing="0"/>
           </process>
         </operator>
         <connect from_port="example set" to_op="Validation" to_port="training"/>
         <connect from_op="Validation" from_port="averagable 1" to_port="performance"/>
         <portSpacing port="source_example set" spacing="0"/>
         <portSpacing port="sink_performance" spacing="0"/>
       </process>
     </operator>
     <connect from_op="Read Excel" from_port="output" to_op="Date to Nominal" to_port="example set input"/>
     <connect from_op="Date to Nominal" from_port="example set output" to_op="Set Role" to_port="example set input"/>
     <connect from_op="Set Role" from_port="example set output" to_op="Declare Missing Value" to_port="example set input"/>
     <connect from_op="Declare Missing Value" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
     <connect from_op="Filter Examples" from_port="example set output" to_op="Sample" to_port="example set input"/>
     <connect from_op="Sample" from_port="example set output" to_op="Normalize" to_port="example set input"/>
     <connect from_op="Normalize" from_port="example set output" to_op="Forward Selection" to_port="example set"/>
     <connect from_op="Forward Selection" from_port="example set" to_port="result 1"/>
     <portSpacing port="source_input 1" spacing="0"/>
     <portSpacing port="sink_result 1" spacing="0"/>
     <portSpacing port="sink_result 2" spacing="0"/>
   </process>
 </operator>
</process>
Tagged:

Answers

  • Nils_Woehler
    Nils_Woehler New Altair Community Member
    Hi,

    how do you start RapidMiner? How much maximum memory is indicated in the system monitor of RapidMiner?

    Best,
    Nils
  • brett_800
    brett_800 New Altair Community Member
    I looked at the system monitor this time around when I executed the process and I noticed there is 11GB of 16GB allocated to RapidMiner.  The usage of this 11GB went from 0% to 100% with 60 seconds of beginning process execution.  In fact this time, I have not received a memory warning, but RapidMiner has just frozen this time instead.

    I start RapidMiner the 'normal' way I suppose, just double clicking the short cut icon to begin the program.

Categories