"Market basket analysis using fpgrowth"

ullash

Hi

I tried using fpgrowth for market basket analysis. I used a small data sample.
Correct me if i am wrong but i do not seem to get accurate results in terms of support values. By support i understand the number of orders a set of items are in as compared o the total number of orders.

For item 1="id_200-5745" and item 2="id_202-6176"  the value of Support=0.9
But only 1 order out of 19 orders had these two items together. So should the support not be 0.05 (= 1/19)

The data set is already in the format required for fpgrowth. Each row represents one order and each column represents one item. a column value of 'true' indicates that that item is in the order and 'false' indicates the item is not in the order

data set.

id_200-8216 id_204-6359 id_202-4110 id_204-4431 id_203-6751 id_205-3148 id_204-7961 id_203-8852 id_204-3920 id_203-2779 id_203-7381 id_200-2303 id_100-5163 id_200-3492 id_203-0515 id_202-3716 id_202-8819 id_201-2956 id_203-1769 id_100-6580 id_203-7222 id_204-4623 id_200-0748 id_204-6247 id_204-8905 id_200-5745 id_204-1463 id_202-6176 id_204-7074 id_202-5694 FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE

Output is:

Size Support Item 1 Item 2 1 0.95 id_200-5745 1 0.9 id_202-6176 2 0.9 id_200-5745 id_202-6176

xml code:

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExcelExampleSource (2)" class="ExcelExampleSource" breakpoints="after">
        <parameter key="excel_file" value="C:\market basket\fpgrowth_data2.xls"/>
        <parameter key="first_row_as_names" value="true"/>
    </operator>
    <operator name="FPGrowth" class="FPGrowth">
        <parameter key="find_min_number_of_itemsets" value="false"/>
        <parameter key="min_support" value="0.8"/>
    </operator>
</operator>


Many Thanks

IngoRM

Hi,

and again somebody who got into the "meta-data-was-not-defined-and-rapidminer-guessed-wrong"-trap 

Just search a bit in this forum (e.g. in this recent thread here http://rapid-i.com/rapidforum/index.php/topic,776.0.html ) and you will see what I mean. All those problems go back to a combination of the facts that RapidMiner internally stores nominal values as numbers (mainly for performance and also for memory consumption reasons) and users usually are too lazy to define all their meta data for RM correctly (although this is possible as the thread above shows).

The question (after reading those threads) is: what do those items with this "wrong" support have in common? Correct! They start with a "TRUE" value in the first line. In RapidMiner, if no meta data is defined, the first value becomes "negative" and the second becomes "positive". So from there everything is correctly counted - but the problem is that the wrong thing is counted (in this case the number of "FALSE"s) which leads to the high supports.

So what can be done? Well, this is pretty easy: define the meta data (e.g. by defining an .aml file describing your data) or simply make sure that the first line contains only falses. You can also combine both approaches and add such a line to your data, create the .aml file for example with the Attribute Editor tool and remove the line again so that your counts will not be distorted.

Cheers,
Ingo

P.S.: By the way: I moved this thread into the "Problems" board of this forum.