"text mining Excel file"

Hello Roger,

use the [tt]StringTextInput[/tt] operator instead of the [tt]TextInput[/tt] operator.

Kind regards,
Tobias

I tried the StringTextInput operator but the ExcelExampleSource operator doesn't allow me to designate a field as string and the StringTextInput operator looks for a field designated as string. I finally just saved the Excel file as a tab delimited file and imported it with the ExampleSource operator followed by the StringTextInput operator and that works fine.

PS, For every document/example in my file, I get this error message: "[Warning] StringTextInput: Warning: Encoding  unknown. Using default." Should I worry about this?
Thanks!

Hi,

rdmckinney wrote:	rdmckinney wrote:
rdmckinney wrote:	PS, For every document/example in my file, I get this error message: "[Warning] StringTextInput: Warning: Encoding  unknown. Using default." Should I worry about this? Thanks!

well if the data shows up correctly, you do not need to worry!

Btw.: the [tt]Nominal2String[/tt] operator converts nominal to string columns. That way, you could load the texts directly from the excel file.

Kind regards,
Tobias

Thanks for the tip!

I'm making progress  I am running the following code and so far it has taken 43 minutes. Is that normal?

<operator name="Root" class="Process" expanded="yes">
    <description text="#ylt#h3#ygt#Specifying texts by an example set#ylt#/h3#ygt##ylt#p#ygt#Using the parameter list or the wizard are simple methods for setting up the directories from which the text documents are read. Sometimes, however, a more flexible solution is needed. If, for instance, your text documents have different types of encoding or are written in different languages, you might wish to provide this information  for each input directory separately.#ylt#/p#ygt# #ylt#p#ygt#You can do this by using an example set that contains one row for each input directory and corresponding attributes for source, encoding, type and class. If such an example set is provided, the texts in the parameter list are ignored.#ylt#/p#ygt#"/>
    <operator name="ExampleSource" class="ExampleSource">
        <parameter key="attributes" value="C:\Documents and Settings\rkenney\My Documents\RapidMiner\TextExamples\MSSComments.aml"/>
    </operator>
    <operator name="StringTextInput" class="StringTextInput" expanded="yes">
        <parameter key="remove_original_attributes" value="true"/>
        <parameter key="default_content_language" value="english"/>
        <list key="namespaces">
        </list>
        <operator name="StringTokenizer" class="StringTokenizer">
        </operator>
        <operator name="EnglishStopwordFilter" class="EnglishStopwordFilter">
        </operator>
        <operator name="TokenLengthFilter" class="TokenLengthFilter">
            <parameter key="min_chars" value="3"/>
        </operator>
        <operator name="PorterStemmer" class="PorterStemmer">
        </operator>
    </operator>
    <operator name="EMClustering" class="EMClustering">
        <parameter key="k" value="5"/>
    </operator>
</operator>

Sorry, I forgot to tell you that the input to the EM clustering operator has 600 examples and about 1,600 attributes. It's now up to 1 hr 9 minutes.

I stopped the EM clustering after about 2 hrs and substituted K-means. It ran in about 20 seconds, which is when I realized that it clusters examples and not attributes. I need to cluster attributes because each attribute is a word from a text mining problem. Is there an operator that will transpose a data set?

Never mind! I realized my mistake. I need to apply dimension reduction, such as principal components to the attributes, then cluster. Sorry!

Is there any way to re-run just one operator in a chain? The reason I ask is I have a model that imports data from Excel, uses the stemmers, tokenizers and stopword filters to create a data set using the stringtextinput, then I apply the GHA operator. Do I always need to have the program run through all operators each time when all I really want to do is re-run the GHA operator with different settings? Thanks!

I have a more serious issue now. I'm getting these messages: May 1, 2009 11:50:55 AM: [Fatal] ArrayIndexOutOfBoundsException occured in 1st application of GHA (GHA)
G May 1, 2009 11:50:55 AM: [Fatal] Process failed: operator cannot be executed (6). Check the log messages...

Here's my code:

<operator name="Root" class="Process" expanded="yes">
    <description text="#ylt#h3#ygt#Specifying texts by an example set#ylt#/h3#ygt##ylt#p#ygt#Using the parameter list or the wizard are simple methods for setting up the directories from which the text documents are read. Sometimes, however, a more flexible solution is needed. If, for instance, your text documents have different types of encoding or are written in different languages, you might wish to provide this information  for each input directory separately.#ylt#/p#ygt# #ylt#p#ygt#You can do this by using an example set that contains one row for each input directory and corresponding attributes for source, encoding, type and class. If such an example set is provided, the texts in the parameter list are ignored.#ylt#/p#ygt#"/>
    <operator name="ExampleSource" class="ExampleSource">
        <parameter key="attributes" value="C:\Documents and Settings\rkenney\My Documents\RapidMiner\TextExamples\MSSComments.aml"/>
    </operator>
    <operator name="StringTextInput" class="StringTextInput" expanded="yes">
        <parameter key="remove_original_attributes" value="true"/>
        <parameter key="default_content_language" value="english"/>
        <list key="namespaces">
        </list>
        <operator name="StringTokenizer" class="StringTokenizer">
        </operator>
        <operator name="EnglishStopwordFilter" class="EnglishStopwordFilter">
        </operator>
        <operator name="TokenLengthFilter" class="TokenLengthFilter">
            <parameter key="min_chars" value="3"/>
        </operator>
        <operator name="PorterStemmer" class="PorterStemmer">
        </operator>
    </operator>
    <operator name="GHA" class="GHA">
        <parameter key="number_of_components" value="6"/>
        <parameter key="number_of_iterations" value="100"/>
    </operator>

I need to add that in the problem above I am trying to reduce 1,600 variables to as few components as possible. If I choose -1 as my number of components, then the program works fine and creates 1,600 components. But If I try to limit the number of components to even 200, I get the error message.