nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Siemens Community Catalyst Program

The Siemens Community Catalyst program was co-created with our community to acknowledge technology leaders who consistently contribute to the Siemens Community. Nominations are accepted on a rolling basis.

Nominate Now

mining high dimensional data...

yogafire

hello!!!!

i have a large dimensional data set.... actually the data set consist of about 2000 record and its dimension is about 2000 indeed....

i admit maybe i am still amateur in mining high dimensional data... ;D

what i'm going to ask is how are strategies to mine high dimensional data using RM5.

thank you for your immediate reply!!!

regs,

dimas yogatama

Find more posts tagged with

AI Studio

Accepted answers

All comments

haddock

Hi there Dimas,

I'm not clear as to what you want to know, but you should understand that RM can handle much larger datasets than you are talking about. For example if I run the following to get a 10k * 10k matrix ...

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="391" width="915">
      <operator activated="true" class="generate_massive_data" expanded="true" height="60" name="Generate Massive Data" width="90" x="135" y="90">
        <parameter key="sparse_representation" value="false"/>
      </operator>
      <connect from_op="Generate Massive Data" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

It doesn't take too long.

Mar 24, 2010 6:16:23 PM INFO: Decoupling process from location //R5 Forum/data. Process is now associated with file //R5 Forum/data.
Mar 24, 2010 6:17:15 PM INFO: No filename given for result file, using stdout for logging results!
Mar 24, 2010 6:17:15 PM INFO: Loading initial data.
Mar 24, 2010 6:17:15 PM INFO: Process starts
Mar 24, 2010 6:17:21 PM INFO: Saving results.
Mar 24, 2010 6:17:21 PM INFO: Process finished successfully after 5 s

Just so you can compare I'm on XP64 double quad with 16G, and for windows boxes it is that 64 that matters, as 32 bit boxes can only address 3??G ( you'll have to Google for the right number ).

So the bottom line is that the main strategy is to have lots of memory, if I remember correctly..

yogafire

haddock wrote:
Hi there Dimas,

I'm not clear as to what you want to know, but you should understand that RM can handle much larger datasets than you are talking about. For example if I run the following to get a 10k * 10k matrix ...
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="391" width="915">
      <operator activated="true" class="generate_massive_data" expanded="true" height="60" name="Generate Massive Data" width="90" x="135" y="90">
        <parameter key="sparse_representation" value="false"/>
      </operator>
      <connect from_op="Generate Massive Data" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>
It doesn't take too long.

Just so you can compare I'm on XP64 double quad with 16G, and for windows boxes it is that 64 that matters, as 32 bit boxes can only address 3??G ( you'll have to Google for the right number ).

So the bottom line is that the main strategy is to have lots of memory, if I remember correctly..

ow, maybe i didn't make it clearer yet, sorry...

what i mean strategy is that, how to optimize accuracy by selecting only "good attribute" among all available ones.... if the specs issue is critical, i only have laptop (lenovo y450-310) with core 2 duo processor @2200 ghz, and 2 gb ddr3 of ram, is it really bothering...?

after all i would like to say sorry for my english, i am still learning.

regs,

Dimas Yogatama

land

Hi Dimas,
of course the amount of memory does make a difference. If the data doesn't fit into the memory, it either fails or you will need it to stream it from a database what might slow down your process a lot.

Coming back to the strategy question: RapidMiner offers several methods for selecting attributes. You might either use the Forward Selection or Backward Elimination operator as a simple start. If that does not suit your needs or they take too long, you might take another operator from the package and it's sub packages Data Transformation / Attribute Set Reduction and Transformation / Selection.

Greetings,
Sebastian

yogafire

Sebastian Land wrote:

Hi Dimas,
of course the amount of memory does make a difference. If the data doesn't fit into the memory, it either fails or you will need it to stream it from a database what might slow down your process a lot.

Coming back to the strategy question: RapidMiner offers several methods for selecting attributes. You might either use the Forward Selection or Backward Elimination operator as a simple start. If that does not suit your needs or they take too long, you might take another operator from the package and it's sub packages Data Transformation / Attribute Set Reduction and Transformation / Selection.

Greetings,
Sebastian

how about attribute weighting? how is the performance between attribute selection/attribute set reduction vs attribute weighting based on your experience in mining high dimensional data?

then what is actually affect the length of model learning by general if we talk about data? is it the total sum of the data (record) or its dimension?

IngoRM

Hello Dimas,

well, there is no general answer for this. In some settings the complete removal of attributes works better, in some others a rescaling based on weights. The same is true if it comes to "weight by wrapper" vs. "weight by filtering". From my experience, I would say that if you have severe problems with data set size and no other option is possible for you, the calculation of weights followed by a weight based selection can help without loosing too much accuracy.

Cheers,
Ingo

yogafire

thank you very much...

Quick Links

Upcoming Events

Advancing power electronics design with Simcenter PSIM and High-Performance Computing
Join us for a community conversation on how High-Performance Computing (HPC) and Simcenter PSIM work together to meet the increasing demands of modern power electronics simulation. During this one-hour technical session, you'll discover how leveraging HPC can speed up simulations, improve scalability, and empower engineers…
Getting Started with Faster Motor Building in Simcenter Simlab
Join us for a community conversation on accelerating your motor simulation workflow with Simcenter Simlab. In this beginner-friendly session, we'll introduce the platform's automated model-building capabilities and show how easy it is to set up your first electric motor analysis. You'll see the basic steps of geometry…
Practical hybrid solver workflows in Simcenter Feko
Join us for a Community Conversation where we'll discuss practical approaches to analyzing complex electromagnetic models using hybrid solver techniques. This session will provide a general overview of how different computational methods can be combined within a single simulation to address models containing varied…

View all community events