"Kmeans Cluster"
venkat
New Altair Community Member
Hi,
I am doing kmeans clustering through rapidminer. Earlier I used 2GB for rapidminer. I faced java heap error.
Then I have increased RAM to 8GB now I am using 6GB for rapidminer. Even though the same error is coming. My input dataset contains 65K records.
The size of my input file 25 MB. I am bit confused, if the small input file can not able to process kmeans, how the big data can be deal with rapidminer.
Apr 03, 2014 5:03:22 PM com.rapidminer.gui.ProcessThread run
SEVERE: Process failed: Java heap space
java.lang.OutOfMemoryError: Java heap space
at com.rapidminer.example.table.DoubleArrayDataRow.ensureNumberOfColumns(DoubleArrayDataRow.java:72)
at com.rapidminer.example.table.MemoryExampleTable.addAttributes(MemoryExampleTable.java:209)
at com.rapidminer.operator.preprocessing.filter.NominalToNumericModel.applyOnDataDummyCoding(NominalToNumericModel.java:250)
at com.rapidminer.operator.preprocessing.filter.NominalToNumericModel.applyOnData(NominalToNumericModel.java:196)
at com.rapidminer.operator.preprocessing.PreprocessingModel.apply(PreprocessingModel.java:95)
at com.rapidminer.operator.preprocessing.PreprocessingOperator.apply(PreprocessingOperator.java:130)
at com.rapidminer.operator.AbstractExampleSetProcessing.doWork(AbstractExampleSetProcessing.java:116)
at com.rapidminer.operator.Operator.execute(Operator.java:867)
at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:375)
at com.rapidminer.operator.Operator.execute(Operator.java:867)
at com.rapidminer.Process.run(Process.java:949)
at com.rapidminer.Process.run(Process.java:873)
at com.rapidminer.Process.run(Process.java:832)
at com.rapidminer.Process.run(Process.java:827)
at com.rapidminer.Process.run(Process.java:817)
at com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)
Apr 03, 2014 5:03:22 PM com.rapidminer.gui.ProcessThread run
SEVERE: Here: Root[1] (Process)
subprocess 'Main Process'
+- Retrieve sample[1] (Retrieve)
+- Normalize[1] (Normalize)
+- Set Role[1] (Set Role)
+- Sample[1] (Sample)
==> +- Nominal to Numerical[1] (Nominal to Numerical)
+- KMeans[0] (k-Means)
+- SVDReduction[0] (Singular Value Decomposition)
My XML is like this:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Root">
<description><p>In many cases, no target attribute (label) can be defined and the data should be automatically grouped. This procedure is called &quot;Clustering&quot;. RapidMiner supports a wide range of clustering schemes which can be used in just the same way like any other learning scheme. This includes the combination with all preprocessing operators. <p> <p> In this experimen, the well-known Iris data set is loaded (the label is loaded, too, but it is only used for visualization and comparison and not for building the clusters itself). One of the most simple clustering schemes, namely KMeans, is then applied to this data set. Afterwards, a dimensionality reduction is performed in order to better support the visualization of the data set in two dimensions. </p><p> Just perform the process and compare the clustering result with the original label (e.g. in the plot view of the example set). You can also visualize the cluster model itself. </p></description>
<parameter key="logverbosity" value="warning"/>
<process expanded="true">
<operator activated="true" class="retrieve" compatibility="5.3.013" expanded="true" height="60" name="Retrieve sample" width="90" x="45" y="75">
<parameter key="repository_entry" value="//Local Repository/data/sample"/>
</operator>
<operator activated="true" class="normalize" compatibility="5.3.013" expanded="true" height="94" name="Normalize" width="90" x="246" y="300">
<parameter key="invert_selection" value="true"/>
<parameter key="include_special_attributes" value="true"/>
</operator>
<operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="76" name="Set Role" width="90" x="313" y="30">
<parameter key="attribute_name" value="content"/>
<list key="set_additional_roles"/>
</operator>
<operator activated="true" class="sample" compatibility="5.3.013" expanded="true" height="76" name="Sample" width="90" x="380" y="165">
<parameter key="sample" value="relative"/>
<parameter key="sample_size" value="-1"/>
<list key="sample_size_per_class"/>
<list key="sample_ratio_per_class"/>
<list key="sample_probability_per_class"/>
<parameter key="use_local_random_seed" value="true"/>
</operator>
<operator activated="true" class="nominal_to_numerical" compatibility="5.3.013" expanded="true" height="94" name="Nominal to Numerical" width="90" x="447" y="300">
<list key="comparison_groups"/>
</operator>
<operator activated="true" class="k_means" compatibility="5.3.013" expanded="true" height="76" name="KMeans" width="90" x="581" y="30">
<parameter key="k" value="3"/>
<parameter key="measure_types" value="NumericalMeasures"/>
<parameter key="numerical_measure" value="CosineSimilarity"/>
<parameter key="use_local_random_seed" value="true"/>
</operator>
<operator activated="true" class="singular_value_decomposition" compatibility="5.1.004" expanded="true" height="94" name="SVDReduction" width="90" x="715" y="120">
<parameter key="dimensions" value="2"/>
</operator>
<connect from_op="Retrieve sample" from_port="output" to_op="Normalize" to_port="example set input"/>
<connect from_op="Normalize" from_port="example set output" to_op="Set Role" to_port="example set input"/>
<connect from_op="Set Role" from_port="example set output" to_op="Sample" to_port="example set input"/>
<connect from_op="Sample" from_port="example set output" to_op="Nominal to Numerical" to_port="example set input"/>
<connect from_op="Nominal to Numerical" from_port="example set output" to_op="KMeans" to_port="example set"/>
<connect from_op="KMeans" from_port="cluster model" to_port="result 1"/>
<connect from_op="KMeans" from_port="clustered set" to_op="SVDReduction" to_port="example set input"/>
<connect from_op="SVDReduction" from_port="example set output" to_port="result 2"/>
<connect from_op="SVDReduction" from_port="original" to_port="result 3"/>
<connect from_op="SVDReduction" from_port="preprocessing model" to_port="result 4"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="72"/>
<portSpacing port="sink_result 3" spacing="0"/>
<portSpacing port="sink_result 4" spacing="0"/>
<portSpacing port="sink_result 5" spacing="0"/>
</process>
</operator>
</process>
Please let me know if any suggestions. Now I am in dilemma whether the rapidminer is suitable for large data ?
Thanks in advance,
Venkat
I am doing kmeans clustering through rapidminer. Earlier I used 2GB for rapidminer. I faced java heap error.
Then I have increased RAM to 8GB now I am using 6GB for rapidminer. Even though the same error is coming. My input dataset contains 65K records.
The size of my input file 25 MB. I am bit confused, if the small input file can not able to process kmeans, how the big data can be deal with rapidminer.
Apr 03, 2014 5:03:22 PM com.rapidminer.gui.ProcessThread run
SEVERE: Process failed: Java heap space
java.lang.OutOfMemoryError: Java heap space
at com.rapidminer.example.table.DoubleArrayDataRow.ensureNumberOfColumns(DoubleArrayDataRow.java:72)
at com.rapidminer.example.table.MemoryExampleTable.addAttributes(MemoryExampleTable.java:209)
at com.rapidminer.operator.preprocessing.filter.NominalToNumericModel.applyOnDataDummyCoding(NominalToNumericModel.java:250)
at com.rapidminer.operator.preprocessing.filter.NominalToNumericModel.applyOnData(NominalToNumericModel.java:196)
at com.rapidminer.operator.preprocessing.PreprocessingModel.apply(PreprocessingModel.java:95)
at com.rapidminer.operator.preprocessing.PreprocessingOperator.apply(PreprocessingOperator.java:130)
at com.rapidminer.operator.AbstractExampleSetProcessing.doWork(AbstractExampleSetProcessing.java:116)
at com.rapidminer.operator.Operator.execute(Operator.java:867)
at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:375)
at com.rapidminer.operator.Operator.execute(Operator.java:867)
at com.rapidminer.Process.run(Process.java:949)
at com.rapidminer.Process.run(Process.java:873)
at com.rapidminer.Process.run(Process.java:832)
at com.rapidminer.Process.run(Process.java:827)
at com.rapidminer.Process.run(Process.java:817)
at com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)
Apr 03, 2014 5:03:22 PM com.rapidminer.gui.ProcessThread run
SEVERE: Here: Root[1] (Process)
subprocess 'Main Process'
+- Retrieve sample[1] (Retrieve)
+- Normalize[1] (Normalize)
+- Set Role[1] (Set Role)
+- Sample[1] (Sample)
==> +- Nominal to Numerical[1] (Nominal to Numerical)
+- KMeans[0] (k-Means)
+- SVDReduction[0] (Singular Value Decomposition)
My XML is like this:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Root">
<description><p>In many cases, no target attribute (label) can be defined and the data should be automatically grouped. This procedure is called &quot;Clustering&quot;. RapidMiner supports a wide range of clustering schemes which can be used in just the same way like any other learning scheme. This includes the combination with all preprocessing operators. <p> <p> In this experimen, the well-known Iris data set is loaded (the label is loaded, too, but it is only used for visualization and comparison and not for building the clusters itself). One of the most simple clustering schemes, namely KMeans, is then applied to this data set. Afterwards, a dimensionality reduction is performed in order to better support the visualization of the data set in two dimensions. </p><p> Just perform the process and compare the clustering result with the original label (e.g. in the plot view of the example set). You can also visualize the cluster model itself. </p></description>
<parameter key="logverbosity" value="warning"/>
<process expanded="true">
<operator activated="true" class="retrieve" compatibility="5.3.013" expanded="true" height="60" name="Retrieve sample" width="90" x="45" y="75">
<parameter key="repository_entry" value="//Local Repository/data/sample"/>
</operator>
<operator activated="true" class="normalize" compatibility="5.3.013" expanded="true" height="94" name="Normalize" width="90" x="246" y="300">
<parameter key="invert_selection" value="true"/>
<parameter key="include_special_attributes" value="true"/>
</operator>
<operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="76" name="Set Role" width="90" x="313" y="30">
<parameter key="attribute_name" value="content"/>
<list key="set_additional_roles"/>
</operator>
<operator activated="true" class="sample" compatibility="5.3.013" expanded="true" height="76" name="Sample" width="90" x="380" y="165">
<parameter key="sample" value="relative"/>
<parameter key="sample_size" value="-1"/>
<list key="sample_size_per_class"/>
<list key="sample_ratio_per_class"/>
<list key="sample_probability_per_class"/>
<parameter key="use_local_random_seed" value="true"/>
</operator>
<operator activated="true" class="nominal_to_numerical" compatibility="5.3.013" expanded="true" height="94" name="Nominal to Numerical" width="90" x="447" y="300">
<list key="comparison_groups"/>
</operator>
<operator activated="true" class="k_means" compatibility="5.3.013" expanded="true" height="76" name="KMeans" width="90" x="581" y="30">
<parameter key="k" value="3"/>
<parameter key="measure_types" value="NumericalMeasures"/>
<parameter key="numerical_measure" value="CosineSimilarity"/>
<parameter key="use_local_random_seed" value="true"/>
</operator>
<operator activated="true" class="singular_value_decomposition" compatibility="5.1.004" expanded="true" height="94" name="SVDReduction" width="90" x="715" y="120">
<parameter key="dimensions" value="2"/>
</operator>
<connect from_op="Retrieve sample" from_port="output" to_op="Normalize" to_port="example set input"/>
<connect from_op="Normalize" from_port="example set output" to_op="Set Role" to_port="example set input"/>
<connect from_op="Set Role" from_port="example set output" to_op="Sample" to_port="example set input"/>
<connect from_op="Sample" from_port="example set output" to_op="Nominal to Numerical" to_port="example set input"/>
<connect from_op="Nominal to Numerical" from_port="example set output" to_op="KMeans" to_port="example set"/>
<connect from_op="KMeans" from_port="cluster model" to_port="result 1"/>
<connect from_op="KMeans" from_port="clustered set" to_op="SVDReduction" to_port="example set input"/>
<connect from_op="SVDReduction" from_port="example set output" to_port="result 2"/>
<connect from_op="SVDReduction" from_port="original" to_port="result 3"/>
<connect from_op="SVDReduction" from_port="preprocessing model" to_port="result 4"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="72"/>
<portSpacing port="sink_result 3" spacing="0"/>
<portSpacing port="sink_result 4" spacing="0"/>
<portSpacing port="sink_result 5" spacing="0"/>
</process>
</operator>
</process>
Please let me know if any suggestions. Now I am in dilemma whether the rapidminer is suitable for large data ?
Thanks in advance,
Venkat
Tagged:
0
Answers
-
As you can see from the error log the problematic operator is Nominal to Numerical. It creates one column for each nominal value you have in your dataset. If you have a lot of different values it creates many many rows which need a lot of memory. For example if you had a nominal id then alone from that you would end up with 65k columns.
k-Means can also work directly with nominal values if you select the mixed measures as distance measure.
Best regards,
Marius0