How to get index of a word in the list and can we extract the value based on index?

Anusha

Hi All!

My dataset has six columns. All columns are in the list format.
first column(list) has common words which are present in 2nd(ch_1) and 3rd(ch_2) columns. The respective values are present in 4th(va_1) and 5th(va_2) columns. So, I need to loop over the each word present in "list" column and extract the index of that particular in ch_1& ch_2 and get the respective values in va_1&va_2. After getting va_1 and va_2, I have to compare if va_1 not equal to va_2 flag "mis" value becomes "1" else "0".

Source:

list                          ch_1                         ch_2                  va_1                        va_2
typ,shap               typ,shap,fgh            typ,fgi,shap           8,"ghk",90               8,97, "oki" 

----here typ,shap are common words in ch_1&ch_2 and their respective indexes are (0,1)in ch_1 and (0,2) in ch_2. so, I need values from va_1&va_2 based on indexes. As (0,1) in ch_1, need to extract (0,1) values from va_1 and in the same way (0,2) are indexes of ch_2, need to get values of (0,2) from va_2.

Required output:

list                          ch_1                         ch_2                  va_1                        va_2                         mis
typ,shap               typ,shap,fgh            typ,fgi,shap           8,"ghk",90               8,97, "oki"                  1

----"mis" value is "1" because for the "typ"----va_1=8 and va_2=8 matched in the case.
But for the "shap"----va_1="ghk" and va_2="oki", here the values are mismatched. so, If one value is mismatched, then the flag should be "1".

Can anyone help me?

Thanks in Advance!

MartinLiebig

Hi there,

the first step is to convert your data set into a proper data frame. I recommend to do this in the ETL process which pushes the data into your data lake. This is 100% standard thing to do.

Attached is a process to convert it into a format we may use more easily. I think afterwards its just another generate Attributes + maybe pivot it back.

Best,

Martin

<?xml version="1.0" encoding="UTF-8"?><process version="9.9.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.9.000" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="utility:create_exampleset" compatibility="9.9.000" expanded="true" height="68" name="Create ExampleSet" width="90" x="45" y="34">
        <parameter key="generator_type" value="comma separated text"/>
        <parameter key="number_of_examples" value="100"/>
        <parameter key="use_stepsize" value="false"/>
        <list key="function_descriptions"/>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)"/>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="input_csv_text" value="list;ch_1;ch_2;va_1;va_2&#10;typ,shap;typ,shap,fgh;typ,fgi,shap;  8,&quot;ghk&quot;,90;8,97, &quot;oki&quot;  "/>
        <parameter key="column_separator" value=";"/>
        <parameter key="parse_all_as_nominal" value="true"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="generate_id" compatibility="9.9.000" expanded="true" height="82" name="Generate ID" width="90" x="380" y="34">
        <parameter key="create_nominal_ids" value="false"/>
        <parameter key="offset" value="0"/>
      </operator>
      <operator activated="true" class="split" compatibility="9.9.000" expanded="true" height="82" name="Split" width="90" x="514" y="34">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="va_1|va_2"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="nominal"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="file_path"/>
        <parameter key="block_type" value="single_value"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="single_value"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="split_pattern" value=","/>
        <parameter key="split_mode" value="ordered_split"/>
      </operator>
      <operator activated="true" class="de_pivot" compatibility="9.9.000" expanded="true" height="82" name="De-Pivot" width="90" x="648" y="34">
        <list key="attribute_name">
          <parameter key="first_list" value="va_1_(\d+)"/>
          <parameter key="secondlist" value="va_2_(\d+)"/>
        </list>
        <parameter key="index_attribute" value="rowId"/>
        <parameter key="create_nominal_index" value="false"/>
        <parameter key="keep_missings" value="false"/>
      </operator>
      <connect from_op="Create ExampleSet" from_port="output" to_op="Generate ID" to_port="example set input"/>
      <connect from_op="Generate ID" from_port="example set output" to_op="Split" to_port="example set input"/>
      <connect from_op="Split" from_port="example set output" to_op="De-Pivot" to_port="example set input"/>
      <connect from_op="De-Pivot" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>