EM Clustering as supervised method in X-Validation

fschm

Hi folks,
I'm playing around with different clustering/segmentation operators and ran into an issue with the Expectation Maximization one.
I incorporated k-means, x-means and EMC in a cross validation comparing their results with the actual label.

The attached process is basically the same for all three, except for an additional 'Generate ID' step in the preprocessing.

The issue I run into is that the 'Map Clustering on Labels' operator does not receive cluster results i.e. placeholder '?' value from the 'Apply Model' operator in the testing phase.

Any idea where this stems from? Outside of the X-Validation EM works just fine and there aren't many parameters I could tweak.
Thanks already!

x-means before map clustering:

EM before map clustering:

<?xml version="1.0" encoding="UTF-8"?><process version="9.3.001"><br>  <context><br>    <input/><br>    <output/><br>    <macros/><br>  </context><br>  <operator activated="true" class="process" compatibility="9.3.001" expanded="true" name="Process"><br>    <parameter key="logverbosity" value="init"/><br>    <parameter key="random_seed" value="2001"/><br>    <parameter key="send_mail" value="never"/><br>    <parameter key="notification_email" value=""/><br>    <parameter key="process_duration_for_mail" value="30"/><br>    <parameter key="encoding" value="SYSTEM"/><br>    <process expanded="true"><br>      <operator activated="true" class="read_excel" compatibility="9.3.001" expanded="true" height="68" name="Read Excel" width="90" x="112" y="34"><br>        <parameter key="excel_file" value="..."/><br>        <parameter key="sheet_selection" value="sheet number"/><br>        <parameter key="sheet_number" value="5"/><br>        <parameter key="imported_cell_range" value="A1"/><br>        <parameter key="encoding" value="SYSTEM"/><br>        <parameter key="first_row_as_names" value="true"/><br>        <list key="annotations"/><br>        <parameter key="date_format" value=""/><br>        <parameter key="time_zone" value="SYSTEM"/><br>        <parameter key="locale" value="English (United States)"/><br>        <parameter key="read_all_values_as_polynominal" value="false"/><br>        <list key="data_set_meta_data_information"/><br>        <parameter key="read_not_matching_values_as_missings" value="true"/><br>        <parameter key="datamanagement" value="double_array"/><br>        <parameter key="data_management" value="auto"/><br>      </operator><br>      <operator activated="true" class="generate_id" compatibility="9.3.001" expanded="true" height="82" name="Generate ID" width="90" x="313" y="34"><br>        <parameter key="create_nominal_ids" value="false"/><br>        <parameter key="offset" value="0"/><br>      </operator><br>      <operator activated="true" class="set_role" compatibility="9.3.001" expanded="true" height="82" name="Set Role" width="90" x="514" y="34"><br>        <parameter key="attribute_name" value="Label"/><br>        <parameter key="target_role" value="label"/><br>        <list key="set_additional_roles"/><br>      </operator><br>      <operator activated="true" class="concurrency:cross_validation" compatibility="9.3.001" expanded="true" height="145" name="Cross Validation" width="90" x="715" y="34"><br>        <parameter key="split_on_batch_attribute" value="false"/><br>        <parameter key="leave_one_out" value="false"/><br>        <parameter key="number_of_folds" value="10"/><br>        <parameter key="sampling_type" value="automatic"/><br>        <parameter key="use_local_random_seed" value="false"/><br>        <parameter key="local_random_seed" value="1992"/><br>        <parameter key="enable_parallel_execution" value="true"/><br>        <process expanded="true"><br>          <operator activated="true" class="expectation_maximization_clustering" compatibility="9.3.001" expanded="true" height="82" name="Clustering" width="90" x="179" y="34"><br>            <parameter key="k" value="3"/><br>            <parameter key="add_cluster_attribute" value="true"/><br>            <parameter key="add_as_label" value="false"/><br>            <parameter key="remove_unlabeled" value="false"/><br>            <parameter key="max_runs" value="5"/><br>            <parameter key="max_optimization_steps" value="100"/><br>            <parameter key="quality" value="1.0E-10"/><br>            <parameter key="use_local_random_seed" value="false"/><br>            <parameter key="local_random_seed" value="1992"/><br>            <parameter key="show_probabilities" value="true"/><br>            <parameter key="inital_distribution" value="k-means run"/><br>            <parameter key="numerical_measure_types" value="BregmanDivergences"/><br>            <parameter key="numerical_measure" value="EuclideanDistance"/><br>            <parameter key="divergence" value="SquaredEuclideanDistance"/><br>            <parameter key="kernel_type" value="radial"/><br>            <parameter key="kernel_gamma" value="1.0"/><br>            <parameter key="kernel_sigma1" value="1.0"/><br>            <parameter key="kernel_sigma2" value="0.0"/><br>            <parameter key="kernel_sigma3" value="2.0"/><br>            <parameter key="kernel_degree" value="3.0"/><br>            <parameter key="kernel_shift" value="1.0"/><br>            <parameter key="kernel_a" value="1.0"/><br>            <parameter key="kernel_b" value="0.0"/><br>            <parameter key="correlated_attributes" value="true"/><br>          </operator><br>          <connect from_port="training set" to_op="Clustering" to_port="example set"/><br>          <connect from_op="Clustering" from_port="cluster model" to_port="model"/><br>          <portSpacing port="source_training set" spacing="0"/><br>          <portSpacing port="sink_model" spacing="0"/><br>          <portSpacing port="sink_through 1" spacing="0"/><br>        </process><br>        <process expanded="true"><br>          <operator activated="true" class="apply_model" compatibility="9.3.001" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34"><br>            <list key="application_parameters"/><br>            <parameter key="create_view" value="false"/><br>          </operator><br>          <operator activated="true" breakpoints="before" class="map_clustering_on_labels" compatibility="9.3.001" expanded="true" height="82" name="Map Clustering on Labels" width="90" x="179" y="34"/><br>          <operator activated="true" class="performance" compatibility="9.3.001" expanded="true" height="82" name="Performance" width="90" x="313" y="34"><br>            <parameter key="use_example_weights" value="true"/><br>          </operator><br>          <connect from_port="model" to_op="Apply Model" to_port="model"/><br>          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/><br>          <connect from_op="Apply Model" from_port="labelled data" to_op="Map Clustering on Labels" to_port="example set"/><br>          <connect from_op="Apply Model" from_port="model" to_op="Map Clustering on Labels" to_port="cluster model"/><br>          <connect from_op="Map Clustering on Labels" from_port="example set" to_op="Performance" to_port="labelled data"/><br>          <connect from_op="Performance" from_port="performance" to_port="performance 1"/><br>          <portSpacing port="source_model" spacing="0"/><br>          <portSpacing port="source_test set" spacing="0"/><br>          <portSpacing port="source_through 1" spacing="0"/><br>          <portSpacing port="sink_test set results" spacing="0"/><br>          <portSpacing port="sink_performance 1" spacing="0"/><br>          <portSpacing port="sink_performance 2" spacing="0"/><br>        </process><br>      </operator><br>      <connect from_op="Read Excel" from_port="output" to_op="Generate ID" to_port="example set input"/><br>      <connect from_op="Generate ID" from_port="example set output" to_op="Set Role" to_port="example set input"/><br>      <connect from_op="Set Role" from_port="example set output" to_op="Cross Validation" to_port="example set"/><br>      <connect from_op="Cross Validation" from_port="performance 1" to_port="result 1"/><br>      <portSpacing port="source_input 1" spacing="0"/><br>      <portSpacing port="sink_result 1" spacing="0"/><br>      <portSpacing port="sink_result 2" spacing="0"/><br>    </process><br>  </operator><br></process><br><br><br>

IngoRM

Hi,

I may be mistaken but I remember that the adding of the cluster column with Apply Model only worked with centroid-based cluster models.  Maybe somebody else can confirm or knows more?

Cheers,
Ingo

fschm

Update: Same happens when I use a flattened hierarchical model, generated in the training phase of X-Validation.

IngoRM

Hi,

I may be mistaken but I remember that the adding of the cluster column with Apply Model only worked with centroid-based cluster models.  Maybe somebody else can confirm or knows more?

Cheers,
Ingo

fschm

@IngoRM Thanks for your post. No confirmation yet, but I'll take it.

SGolbert

Hi @fschm ,

I wanted to point out that you are validating an unsupervised model, which is kind of pointless. If you have labels, you should use a supervised model instead and you will have much better results.

Regards,

Sebastian

fschm

Hi Sebastian,

thanks for the hint. I'm doing so.

However, I was asking myself if the known labels have some 'natural' justification behind them that results from the patterns in the data.
So the thought was to validate if my labels are actually true or might need some refinement.

Hope this makes sense.
Regards,
Fabian