Market Basket Analysis: First Timer

Um, did I perhaps post this in the wrong forum?

Greetings Windsaloft!

Looks like the lights are on but nobody is at home, so let me confuse you further...

I've used the same template, and it needs some attention, specifically it uses macros ( the RM equivalent of variables which show as %{XXXX} in parameters ), but does not assign values to them, so no wonder it confuses you! I've butchered a template by replacing the data call with a generator, like this...


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="431" width="915">
      <operator activated="false" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Transactions"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Item Count" width="90" x="179" y="30">
        <parameter key="macro" value="%{itemCountAttributeName}"/>
        <parameter key="value" value="itemCount"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Customer" width="90" x="313" y="30">
        <parameter key="macro" value="customerIdAttributeName"/>
        <parameter key="value" value="customerId"/>
      </operator>
      <operator activated="false" breakpoints="after" class="set_macro" expanded="true" height="60" name="Define Item" width="90" x="447" y="30">
        <parameter key="macro" value="itemIdAttributeName"/>
        <parameter key="value" value="itemId"/>
      </operator>
      <operator activated="false" class="aggregate" expanded="true" height="76" name="Aggregate" width="90" x="45" y="255">
        <list key="aggregation_attributes">
          <parameter key="amount" value="sum"/>
        </list>
        <parameter key="group_by_attributes" value="customer_id|product_id"/>
      </operator>
      <operator activated="true" class="generate_transaction_data" expanded="true" height="60" name="Generate Transaction Data" width="90" x="4" y="113">
        <parameter key="number_clusters" value="1"/>
      </operator>
      <operator activated="true" class="pivot" expanded="true" height="76" name="Pivot" width="90" x="179" y="210">
        <parameter key="group_attribute" value="Id"/>
        <parameter key="index_attribute" value="Item"/>
      </operator>
      <operator activated="true" class="replace_missing_values" expanded="true" height="94" name="Replace Missing Values" width="90" x="246" y="75">
        <parameter key="default" value="zero"/>
        <list key="columns"/>
      </operator>
      <operator activated="true" class="numerical_to_binominal" expanded="true" height="76" name="Numerical to Binominal" width="90" x="380" y="75"/>
      <operator activated="false" class="set_role" expanded="true" height="76" name="Set Role" width="90" x="313" y="300">
        <parameter key="name" value="%{customerIdAttributeName}"/>
        <parameter key="target_role" value="id"/>
      </operator>
      <operator activated="true" class="fp_growth" expanded="true" height="76" name="FP-Growth" width="90" x="447" y="210"/>
      <operator activated="true" class="create_association_rules" expanded="true" height="60" name="Create Association Rules" width="90" x="581" y="210">
        <parameter key="min_confidence" value="0.1"/>
      </operator>
      <connect from_op="Generate Transaction Data" from_port="output" to_op="Pivot" to_port="example set input"/>
      <connect from_op="Pivot" from_port="example set output" to_op="Replace Missing Values" to_port="example set input"/>
      <connect from_op="Replace Missing Values" from_port="example set output" to_op="Numerical to Binominal" to_port="example set input"/>
      <connect from_op="Numerical to Binominal" from_port="example set output" to_op="FP-Growth" to_port="example set"/>
      <connect from_op="FP-Growth" from_port="frequent sets" to_op="Create Association Rules" to_port="item sets"/>
      <connect from_op="Create Association Rules" from_port="rules" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Actually there are also relevant samples ( 1-25 and 2-23 ), and this subject has raised its ugly head before, as a quick seach for "Market Basket" shows.

Pip Pip ;D

Thanks for the reply -- I did see a lot of XML when I was doing preliminary searches, but it was so far over my head that I couldn't understand what was going on. I had no idea you could just pate the code and go back to design view and visually see what was going on!

I tried your process but I didn't necessarily get any results that I could see.... but I am going to mess around with this.

Thanks for the reply!

land

Hi,
still any problems?

Greetings,
Sebastian

Yes actually, with the above process I keep getting an error for Regular attributes must be of type binomial. The preview shows the ID field as nominal (I assume thats my problem)

Whats weird is, I actually *get* results with the process above (it generates its own recordset). My OWN recordset, has an ID field which is text, so when I replace the first process with a retrieve, everything transitions fine except I don't get any results. And I'm betting the nominal field is the problem.

I've tried adding a Type Conversion process in between: Nominal to Binomial. But that didn't work either.

Ooops,

Mea maxima culpa :-[ I pasted in completely the wrong code.. this is what should have been there...

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="391" width="915">
      <operator activated="false" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Transactions"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Item Count" width="90" x="179" y="30">
        <parameter key="macro" value="%{itemCountAttributeName}"/>
        <parameter key="value" value="itemCount"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Customer" width="90" x="313" y="30">
        <parameter key="macro" value="customerIdAttributeName"/>
        <parameter key="value" value="customerId"/>
      </operator>
      <operator activated="false" breakpoints="after" class="set_macro" expanded="true" height="60" name="Define Item" width="90" x="447" y="30">
        <parameter key="macro" value="itemIdAttributeName"/>
        <parameter key="value" value="itemId"/>
      </operator>
      <operator activated="false" class="aggregate" expanded="true" height="76" name="Aggregate" width="90" x="45" y="255">
        <list key="aggregation_attributes">
          <parameter key="amount" value="sum"/>
        </list>
        <parameter key="group_by_attributes" value="customer_id|product_id"/>
      </operator>
      <operator activated="true" class="generate_transaction_data" expanded="true" height="60" name="Generate Transaction Data" width="90" x="4" y="113">
        <parameter key="number_clusters" value="1"/>
      </operator>
      <operator activated="true" class="pivot" expanded="true" height="76" name="Pivot" width="90" x="179" y="210">
        <parameter key="group_attribute" value="Id"/>
        <parameter key="index_attribute" value="Item"/>
      </operator>
      <operator activated="true" class="replace_missing_values" expanded="true" height="94" name="Replace Missing Values" width="90" x="246" y="75">
        <parameter key="default" value="zero"/>
        <list key="columns"/>
      </operator>
      <operator activated="true" class="numerical_to_binominal" expanded="true" height="76" name="Numerical to Binominal" width="90" x="380" y="75"/>
      <operator activated="false" class="set_role" expanded="true" height="76" name="Set Role" width="90" x="313" y="300">
        <parameter key="name" value="%{customerIdAttributeName}"/>
        <parameter key="target_role" value="id"/>
      </operator>
      <operator activated="true" class="fp_growth" expanded="true" height="76" name="FP-Growth" width="90" x="447" y="210"/>
      <operator activated="true" class="create_association_rules" expanded="true" height="60" name="Create Association Rules" width="90" x="581" y="210">
        <parameter key="min_confidence" value="0.1"/>
      </operator>
      <connect from_op="Generate Transaction Data" from_port="output" to_op="Pivot" to_port="example set input"/>
      <connect from_op="Pivot" from_port="example set output" to_op="Replace Missing Values" to_port="example set input"/>
      <connect from_op="Replace Missing Values" from_port="example set output" to_op="Numerical to Binominal" to_port="example set input"/>
      <connect from_op="Numerical to Binominal" from_port="example set output" to_op="FP-Growth" to_port="example set"/>
      <connect from_op="Numerical to Binominal" from_port="original" to_port="result 2"/>
      <connect from_op="FP-Growth" from_port="frequent sets" to_op="Create Association Rules" to_port="item sets"/>
      <connect from_op="Create Association Rules" from_port="rules" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Hope that goes a bit better!

The grey boxes can be deleted correct?

And is it okay if I still get the caution for FP-Growth that regular attributes must be binomial? I will experiment with this and see if I can put my own dataset into the input, and see if it works.

I think maybe the problem is that there are descriptive statistics such as Range, etc for my ID field, which happens to be all text. My dataset is OrderID and ProductID, and they are both text. Like, E30098A E230843F E230289D; Product0001, Product0002, Product00003

Perhaps that is my problem. I get the warning with the process you posted, but it actually is successful despite the warning, probably because the ID's are numbers?

Hi,

Yep, you can bin the gray jobs, and you can ignore the warning, especially as it all runs OK. So all you need to do is replace the example generator, and all should be well....

Make sure that your meta-data matches on attribute Name and Content

Role Name Content

id Id nominal
regular Item nominal
regular Amount integer

If my example set is different than the generator in content, i.e. my ID's are text ... should I be applying the to binomial conversion at the beginning?

Hi there,

Don't think so, concentrate first on loading the data and seeing ( from the meta-data ) that RM thinks it has data as I described.

Ok. I had been modifying the processes in the graphical view as I switched to my example set, because the column names were slightly different.

Instead of doing that, I'll simply create a new set of data which has the names and content you describe above. That should eliminate the possibility that I was making mistakes while reconfiguring the different processes.

Hi there,

I've imported this CSV format as a data repository, and substituted that repository for the generator

Id,Item,Amount
E30098AE,Product0001,1
E230843F,Product0001,1
E230289D,Product0002,2
E30098AE,Product0002,1
E230843F,Product0001,1
E230289D,Product0002,2

And it works ( in the sense it doesn't fall over ).

;D

Okay, I'm moving forward. I stopped trying to be smart and I just renamed my headers to match the process so that I didn't have that problem.

Now it doesn't break. But my association rules are blank. However this might mean I'm filtering out rules that might have existed in my data, but didn't meet a criteria.

To get the maximum number of results, I set

FP-Growth
min number of items = 0;
positive value = [blank]
min support = 0
max items = -1
must contain = [blank]

Create Association Rules
Min Conf = 0
Gain theta = 0
laplace k = 0

But still can't see rules.

Some real rows from my data that I have, that I would expect some sort of rule would be:
Id Item Amount
D11131 E1 1
D11131 E5 1
D11124 E5 1
D11125 E5 1

I should see a rule appearing for E1 -- E5 right?

Now we're on the rigth track, I'm thinking my example data isn't very good.

All good, now get sensible data and lower the criterion constraints until rules emerge.

Happy dredging!

I really appreciate your help so much, I hope I can learn how to use this tool!

Could you help me find the criteria that could be the maximum results? or did I have it right with my previous post?

Take it step by step. First thing is to understand about frequent item sets, and the parameters for their generation. If in doubt, as always, check out Wikipedia. Then do the rule building end.

pip pip

Hi

I am just reading this post here, it is very good. I have a question- how would i modify the code to include a zip code, therefore providing associations rules by zip code for each?

Thanks you

Just on my previous question, is a clustering method needed for something like this? All the zip codes (attribute) are there already. I just want to see how the market basket analysis can be done by zip code so the association rules will appear as per zip code?

Thanks

land

Hi,
do you want a rule set per zip code? Then you would have to split your data according to the zip codes and perform the process on each of this subsets. You could do this with an filter Examples and a loop value operator.

Greetings,
Sebastian

Hi Sebastian

Yes it is a per zip code.

I am using the code as shown


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input>
      <location/>
    </input>
    <output>
      <location/>
      <location/>
      <location/>
    </output>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="391" width="915">
      <operator activated="false" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Transactions"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Item Count" width="90" x="179" y="30">
        <parameter key="macro" value="%{itemCountAttributeName}"/>
        <parameter key="value" value="itemCount"/>
      </operator>
      <operator activated="false" class="set_macro" expanded="true" height="60" name="Define Customer" width="90" x="313" y="30">
        <parameter key="macro" value="customerIdAttributeName"/>
        <parameter key="value" value="customerId"/>
      </operator>
      <operator activated="false" breakpoints="after" class="set_macro" expanded="true" height="60" name="Define Item" width="90" x="447" y="30">
        <parameter key="macro" value="itemIdAttributeName"/>
        <parameter key="value" value="itemId"/>
      </operator>
      <operator activated="false" class="aggregate" expanded="true" height="76" name="Aggregate" width="90" x="45" y="255">
        <list key="aggregation_attributes">
          <parameter key="amount" value="sum"/>
        </list>
        <parameter key="group_by_attributes" value="customer_id|product_id"/>
      </operator>
      <operator activated="true" class="generate_transaction_data" expanded="true" height="60" name="Generate Transaction Data" width="90" x="4" y="113">
        <parameter key="number_clusters" value="1"/>
      </operator>
      <operator activated="true" class="pivot" expanded="true" height="76" name="Pivot" width="90" x="179" y="210">
        <parameter key="group_attribute" value="Id"/>
        <parameter key="index_attribute" value="Item"/>
      </operator>
      <operator activated="true" class="replace_missing_values" expanded="true" height="94" name="Replace Missing Values" width="90" x="246" y="75">
        <parameter key="default" value="zero"/>
        <list key="columns"/>
      </operator>
      <operator activated="true" class="numerical_to_binominal" expanded="true" height="76" name="Numerical to Binominal" width="90" x="380" y="75"/>
      <operator activated="false" class="set_role" expanded="true" height="76" name="Set Role" width="90" x="313" y="300">
        <parameter key="name" value="%{customerIdAttributeName}"/>
        <parameter key="target_role" value="id"/>
      </operator>
      <operator activated="true" class="fp_growth" expanded="true" height="76" name="FP-Growth" width="90" x="447" y="210"/>
      <operator activated="true" class="create_association_rules" expanded="true" height="60" name="Create Association Rules" width="90" x="581" y="210">
        <parameter key="min_confidence" value="0.1"/>
      </operator>
      <connect from_op="Generate Transaction Data" from_port="output" to_op="Pivot" to_port="example set input"/>
      <connect from_op="Pivot" from_port="example set output" to_op="Replace Missing Values" to_port="example set input"/>
      <connect from_op="Replace Missing Values" from_port="example set output" to_op="Numerical to Binominal" to_port="example set input"/>
      <connect from_op="Numerical to Binominal" from_port="example set output" to_op="FP-Growth" to_port="example set"/>
      <connect from_op="Numerical to Binominal" from_port="original" to_port="result 2"/>
      <connect from_op="FP-Growth" from_port="frequent sets" to_op="Create Association Rules" to_port="item sets"/>
      <connect from_op="Create Association Rules" from_port="rules" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Where exactly can i put these into this? Rather than zip code i am looking at State.

Thanks you