Automated Model Selection and Optimization for Multi-Label Prediction

User: "wada77"
wada77
New Altair Community Member
Updated by Jocelyn

 

Hi,

 

I've got a dataset with a lot of attributes and several labels (in the example below 3 labels) I want to predict simultaneoulsy. I've therfore built a process with "Loop Attributes" to predict each label. Furthermore I've added a "Select Subprocess" to test different models.


My questions so far are:

 

1. How can I impelement "Optimize Parameters" to automatically select the "best" model concerning e. g. the lowst relative error. (It might be possible, that model 1 is better for label 1 and 2 but not for label 3).
2. Which possible solutions (e. g. feature selection) do I have to optimize the whole process (and to lower the relative error)? How do I implement those into my process?

 

Here is a XML process that looks like mine:

 

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.1.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="8.1.001" expanded="true" height="68" name="Retrieve data" width="90" x="45" y="34">
        <parameter key="repository_entry" value="//Frage Forum/Data/data"/>
      </operator>
      <operator activated="true" class="select_subprocess" compatibility="8.1.001" expanded="true" height="82" name="Select Subprocess" width="90" x="380" y="34">
        <parameter key="select_which" value="1"/>
        <process expanded="true">
          <operator activated="true" class="concurrency:loop_attributes" compatibility="8.1.001" expanded="true" height="103" name="Loop Attributes" width="90" x="179" y="34">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attribute" value=""/>
            <parameter key="attributes" value="DFA|RPDE|PPE"/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="attribute_value"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="time"/>
            <parameter key="block_type" value="attribute_block"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="value_matrix_row_start"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="attribute_name_macro" value="loop_attribute"/>
            <parameter key="reuse_results" value="false"/>
            <parameter key="enable_parallel_execution" value="true"/>
            <process expanded="true">
              <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role" width="90" x="112" y="34">
                <parameter key="attribute_name" value="%{loop_attribute}"/>
                <parameter key="target_role" value="label"/>
                <list key="set_additional_roles"/>
              </operator>
              <operator activated="true" class="concurrency:cross_validation" compatibility="8.1.001" expanded="true" height="145" name="Cross Validation" width="90" x="313" y="34">
                <parameter key="split_on_batch_attribute" value="false"/>
                <parameter key="leave_one_out" value="false"/>
                <parameter key="number_of_folds" value="10"/>
                <parameter key="sampling_type" value="automatic"/>
                <parameter key="use_local_random_seed" value="false"/>
                <parameter key="local_random_seed" value="1992"/>
                <parameter key="enable_parallel_execution" value="true"/>
                <process expanded="true">
                  <operator activated="true" class="k_nn" compatibility="8.1.001" expanded="true" height="82" name="k-NN" width="90" x="179" y="34">
                    <parameter key="k" value="1"/>
                    <parameter key="weighted_vote" value="false"/>
                    <parameter key="measure_types" value="MixedMeasures"/>
                    <parameter key="mixed_measure" value="MixedEuclideanDistance"/>
                    <parameter key="nominal_measure" value="NominalDistance"/>
                    <parameter key="numerical_measure" value="EuclideanDistance"/>
                    <parameter key="divergence" value="GeneralizedIDivergence"/>
                    <parameter key="kernel_type" value="radial"/>
                    <parameter key="kernel_gamma" value="1.0"/>
                    <parameter key="kernel_sigma1" value="1.0"/>
                    <parameter key="kernel_sigma2" value="0.0"/>
                    <parameter key="kernel_sigma3" value="2.0"/>
                    <parameter key="kernel_degree" value="3.0"/>
                    <parameter key="kernel_shift" value="1.0"/>
                    <parameter key="kernel_a" value="1.0"/>
                    <parameter key="kernel_b" value="0.0"/>
                  </operator>
                  <connect from_port="training set" to_op="k-NN" to_port="training set"/>
                  <connect from_op="k-NN" from_port="model" to_port="model"/>
                  <connect from_op="k-NN" from_port="exampleSet" to_port="through 1"/>
                  <portSpacing port="source_training set" spacing="0"/>
                  <portSpacing port="sink_model" spacing="0"/>
                  <portSpacing port="sink_through 1" spacing="0"/>
                  <portSpacing port="sink_through 2" spacing="0"/>
                </process>
                <process expanded="true">
                  <operator activated="true" class="apply_model" compatibility="8.1.001" expanded="true" height="82" name="Apply Model" width="90" x="45" y="34">
                    <list key="application_parameters"/>
                    <parameter key="create_view" value="false"/>
                  </operator>
                  <operator activated="true" class="performance_regression" compatibility="8.1.001" expanded="true" height="82" name="Performance" width="90" x="246" y="34">
                    <parameter key="main_criterion" value="first"/>
                    <parameter key="root_mean_squared_error" value="true"/>
                    <parameter key="absolute_error" value="false"/>
                    <parameter key="relative_error" value="true"/>
                    <parameter key="relative_error_lenient" value="false"/>
                    <parameter key="relative_error_strict" value="false"/>
                    <parameter key="normalized_absolute_error" value="false"/>
                    <parameter key="root_relative_squared_error" value="false"/>
                    <parameter key="squared_error" value="false"/>
                    <parameter key="correlation" value="false"/>
                    <parameter key="squared_correlation" value="false"/>
                    <parameter key="prediction_average" value="false"/>
                    <parameter key="spearman_rho" value="false"/>
                    <parameter key="kendall_tau" value="false"/>
                    <parameter key="skip_undefined_labels" value="true"/>
                    <parameter key="use_example_weights" value="true"/>
                  </operator>
                  <connect from_port="model" to_op="Apply Model" to_port="model"/>
                  <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
                  <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
                  <connect from_op="Performance" from_port="performance" to_port="performance 1"/>
                  <connect from_op="Performance" from_port="example set" to_port="test set results"/>
                  <portSpacing port="source_model" spacing="0"/>
                  <portSpacing port="source_test set" spacing="0"/>
                  <portSpacing port="source_through 1" spacing="0"/>
                  <portSpacing port="source_through 2" spacing="0"/>
                  <portSpacing port="sink_test set results" spacing="0"/>
                  <portSpacing port="sink_performance 1" spacing="0"/>
                  <portSpacing port="sink_performance 2" spacing="0"/>
                </process>
              </operator>
              <connect from_port="input 1" to_op="Set Role" to_port="example set input"/>
              <connect from_op="Set Role" from_port="example set output" to_op="Cross Validation" to_port="example set"/>
              <connect from_op="Cross Validation" from_port="test result set" to_port="output 2"/>
              <connect from_op="Cross Validation" from_port="performance 1" to_port="output 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
              <portSpacing port="sink_output 3" spacing="0"/>
            </process>
          </operator>
          <connect from_port="input 1" to_op="Loop Attributes" to_port="input 1"/>
          <connect from_op="Loop Attributes" from_port="output 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="concurrency:loop_attributes" compatibility="8.1.001" expanded="true" height="103" name="Loop Attributes (2)" width="90" x="179" y="34">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attribute" value=""/>
            <parameter key="attributes" value="DFA|RPDE|PPE"/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="attribute_value"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="time"/>
            <parameter key="block_type" value="attribute_block"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="value_matrix_row_start"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="attribute_name_macro" value="loop_attribute"/>
            <parameter key="reuse_results" value="false"/>
            <parameter key="enable_parallel_execution" value="true"/>
            <process expanded="true">
              <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role (2)" width="90" x="112" y="34">
                <parameter key="attribute_name" value="%{loop_attribute}"/>
                <parameter key="target_role" value="label"/>
                <list key="set_additional_roles"/>
              </operator>
              <operator activated="true" class="concurrency:cross_validation" compatibility="8.1.001" expanded="true" height="145" name="Cross Validation (2)" width="90" x="313" y="34">
                <parameter key="split_on_batch_attribute" value="false"/>
                <parameter key="leave_one_out" value="false"/>
                <parameter key="number_of_folds" value="10"/>
                <parameter key="sampling_type" value="automatic"/>
                <parameter key="use_local_random_seed" value="false"/>
                <parameter key="local_random_seed" value="1992"/>
                <parameter key="enable_parallel_execution" value="true"/>
                <process expanded="true">
                  <operator activated="true" class="linear_regression" compatibility="8.1.001" expanded="true" height="103" name="Linear Regression" width="90" x="179" y="34">
                    <parameter key="feature_selection" value="M5 prime"/>
                    <parameter key="alpha" value="0.05"/>
                    <parameter key="max_iterations" value="10"/>
                    <parameter key="forward_alpha" value="0.05"/>
                    <parameter key="backward_alpha" value="0.05"/>
                    <parameter key="eliminate_colinear_features" value="true"/>
                    <parameter key="min_tolerance" value="0.05"/>
                    <parameter key="use_bias" value="true"/>
                    <parameter key="ridge" value="1.0E-8"/>
                  </operator>
                  <connect from_port="training set" to_op="Linear Regression" to_port="training set"/>
                  <connect from_op="Linear Regression" from_port="model" to_port="model"/>
                  <portSpacing port="source_training set" spacing="0"/>
                  <portSpacing port="sink_model" spacing="0"/>
                  <portSpacing port="sink_through 1" spacing="0"/>
                </process>
                <process expanded="true">
                  <operator activated="true" class="apply_model" compatibility="8.1.001" expanded="true" height="82" name="Apply Model (2)" width="90" x="45" y="34">
                    <list key="application_parameters"/>
                    <parameter key="create_view" value="false"/>
                  </operator>
                  <operator activated="true" class="performance_regression" compatibility="8.1.001" expanded="true" height="82" name="Performance (2)" width="90" x="246" y="34">
                    <parameter key="main_criterion" value="first"/>
                    <parameter key="root_mean_squared_error" value="true"/>
                    <parameter key="absolute_error" value="false"/>
                    <parameter key="relative_error" value="true"/>
                    <parameter key="relative_error_lenient" value="false"/>
                    <parameter key="relative_error_strict" value="false"/>
                    <parameter key="normalized_absolute_error" value="false"/>
                    <parameter key="root_relative_squared_error" value="false"/>
                    <parameter key="squared_error" value="false"/>
                    <parameter key="correlation" value="false"/>
                    <parameter key="squared_correlation" value="false"/>
                    <parameter key="prediction_average" value="false"/>
                    <parameter key="spearman_rho" value="false"/>
                    <parameter key="kendall_tau" value="false"/>
                    <parameter key="skip_undefined_labels" value="true"/>
                    <parameter key="use_example_weights" value="true"/>
                  </operator>
                  <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
                  <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
                  <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
                  <connect from_op="Performance (2)" from_port="performance" to_port="performance 1"/>
                  <connect from_op="Performance (2)" from_port="example set" to_port="test set results"/>
                  <portSpacing port="source_model" spacing="0"/>
                  <portSpacing port="source_test set" spacing="0"/>
                  <portSpacing port="source_through 1" spacing="0"/>
                  <portSpacing port="sink_test set results" spacing="0"/>
                  <portSpacing port="sink_performance 1" spacing="0"/>
                  <portSpacing port="sink_performance 2" spacing="0"/>
                </process>
              </operator>
              <connect from_port="input 1" to_op="Set Role (2)" to_port="example set input"/>
              <connect from_op="Set Role (2)" from_port="example set output" to_op="Cross Validation (2)" to_port="example set"/>
              <connect from_op="Cross Validation (2)" from_port="test result set" to_port="output 2"/>
              <connect from_op="Cross Validation (2)" from_port="performance 1" to_port="output 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
              <portSpacing port="sink_output 3" spacing="0"/>
            </process>
          </operator>
          <connect from_port="input 1" to_op="Loop Attributes (2)" to_port="input 1"/>
          <connect from_op="Loop Attributes (2)" from_port="output 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve data" from_port="output" to_op="Select Subprocess" to_port="input 1"/>
      <connect from_op="Select Subprocess" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

 

Find more posts tagged with

AI Studio
Sort by:
1 - 3 of 31
    User: "wada77"
    wada77
    New Altair Community Member
    OP

    Thank you very much for your answer and your link @sgenzer. I know how to use "Optimize Parameters" or "Optimize Selection" for a dataset with one label. 

     

    In my case, I've got multiple labels (see opening post and XML-file) and therefore have no idea how to use e. g. "Optimize Paramters" because with looping over all label attributes I get several "IOObjectCollections" - "Optimize Parameter" expects a single Peformance Vector.

     

    Attached the error message (In a first step, I simply wanted to add "Optimize Parameters" to the XML-Process above to automatically select the "better" subprocess.)

     

     

    Error.png
    User: "tftemme"
    tftemme
    New Altair Community Member
    Hi @wada77

    Also this thread is a bit older, I just wanted to update a bit on this.

    Since 9.4.1 RM contains now the 'Multi Label Modeling' operator to train a model for multiple label attributes. There are also the operators 'Multi Label Performance' (for evaluating the performance of such a model), and the 'Multi Horizon Forecast' and 'Multi Horizon Performance' operator which handles the time series related multi label operations.

    You should be able to use these operators also in combination with Optimize Parameters.

    Best regards,
    Fabian
Quick Links