Keras SP 500 sample to LSTM conversion doesn't work

Hey guys,

I've been fooling around with the new Keras toolkit for Rapidminer. Great stuff! Also very complicated as i'm not yet familiar with Keras and Tensorflow.

That is why i'm fooling around with the sample processes provided in the package: I'm trying to convert the SP 500 sample project from a convolutional model to a LSTM recurrent model. I found a Github repository where some guy did this: he combined 2 LSTM layers with a couple of dropout layers. Source: Link I just tried to replicate this layer structure via the Keras layers in Rapidminer but it won't work.

I think there is something wrong with the 'input shape' setting of the Keras Model operator. Also I can't find some of the functionalities of the layers on github in the operator (like return_sequences).

I'll post the code below. I didn't really change the process. Just used the same dataset and switched the inside operators of the keras model to something similar to the python code I posted above (just some LSTM layers).

Does anyone have an example of a working LSTM rapidminer project or can help me with my project? Hope you guys can help me out! Thanks in advance.

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.6.001" expanded="true" height="68" name="Retrieve s&amp;p-500-data" width="90" x="45" y="85">
        <parameter key="repository_entry" value="//Keras Samples/sp_500_regression/s&amp;p-500-data"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="179" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="Close"/>
        <parameter key="attributes" value="Date|Open|Close"/>
      </operator>
      <operator activated="true" class="normalize" compatibility="7.6.001" expanded="true" height="103" name="Normalize" width="90" x="313" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="Close"/>
      </operator>
      <operator activated="true" class="series:windowing" compatibility="7.4.000" expanded="true" height="82" name="Windowing" width="90" x="447" y="85">
        <parameter key="window_size" value="30"/>
        <parameter key="label_attribute" value="Close"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="7.6.001" expanded="true" height="82" name="Set Role" width="90" x="581" y="85">
        <parameter key="attribute_name" value="Close-29"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="split_data" compatibility="7.6.001" expanded="true" height="103" name="Split Data" width="90" x="581" y="238">
        <enumeration key="partitions">
          <parameter key="ratio" value="0.9"/>
          <parameter key="ratio" value="0.1"/>
        </enumeration>
        <parameter key="sampling_type" value="linear sampling"/>
      </operator>
      <operator activated="true" class="keras:sequential" compatibility="1.0.003" expanded="true" height="166" name="Keras Model" width="90" x="782" y="85">
        <parameter key="input shape" value="(1,0)"/>
        <parameter key="optimizer" value="RMSprop"/>
        <parameter key="learning rate" value="0.001"/>
        <enumeration key="metric"/>
        <parameter key="epochs" value="256"/>
        <enumeration key="callbacks">
          <parameter key="callbacks" value="TensorBoard(log_dir='./logs', histogram_freq=0, write_graph=True, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)"/>
        </enumeration>
        <process expanded="true">
          <operator activated="true" class="keras:recurrent_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Recurrent Layer" width="90" x="715" y="187">
            <parameter key="layer_type" value="LSTM"/>
          </operator>
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="68" name="Add Core Layer" width="90" x="313" y="34">
            <parameter key="dims" value="1.1"/>
          </operator>
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Core Layer (2)" width="90" x="849" y="187">
            <parameter key="layer_type" value="Dropout"/>
            <parameter key="rate" value="0.2"/>
            <parameter key="dims" value="1.1"/>
          </operator>
          <operator activated="true" class="keras:recurrent_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Recurrent Layer (2)" width="90" x="983" y="187">
            <parameter key="layer_type" value="LSTM"/>
          </operator>
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Core Layer (3)" width="90" x="1117" y="187">
            <parameter key="layer_type" value="Dropout"/>
            <parameter key="rate" value="0.2"/>
            <parameter key="dims" value="1.1"/>
          </operator>
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Core Layer (4)" width="90" x="1251" y="187">
            <parameter key="no_units" value="3"/>
            <parameter key="rate" value="0.2"/>
            <parameter key="dims" value="1.1"/>
          </operator>
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Core Layer (5)" width="90" x="1385" y="187">
            <parameter key="layer_type" value="Activation"/>
            <parameter key="activation_function" value="'linear'"/>
            <parameter key="rate" value="0.2"/>
            <parameter key="dims" value="1.1"/>
          </operator>
          <connect from_op="Add Recurrent Layer" from_port="layers 1" to_op="Add Core Layer (2)" to_port="layers"/>
          <connect from_op="Add Core Layer (2)" from_port="layers 1" to_op="Add Recurrent Layer (2)" to_port="layers"/>
          <connect from_op="Add Recurrent Layer (2)" from_port="layers 1" to_op="Add Core Layer (3)" to_port="layers"/>
          <connect from_op="Add Core Layer (3)" from_port="layers 1" to_op="Add Core Layer (4)" to_port="layers"/>
          <connect from_op="Add Core Layer (4)" from_port="layers 1" to_op="Add Core Layer (5)" to_port="layers"/>
          <connect from_op="Add Core Layer (5)" from_port="layers 1" to_port="layers 1"/>
          <portSpacing port="sink_layers 1" spacing="0"/>
          <portSpacing port="sink_layers 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="keras:apply" compatibility="1.0.003" expanded="true" height="82" name="Apply Keras Model" width="90" x="849" y="391">
        <parameter key="batch_size" value="128"/>
      </operator>
      <connect from_op="Retrieve s&amp;p-500-data" from_port="output" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Normalize" from_port="example set output" to_op="Windowing" to_port="example set input"/>
      <connect from_op="Windowing" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Split Data" to_port="example set"/>
      <connect from_op="Split Data" from_port="partition 1" to_op="Keras Model" to_port="training set"/>
      <connect from_op="Split Data" from_port="partition 2" to_op="Apply Keras Model" to_port="unlabelled data"/>
      <connect from_op="Keras Model" from_port="model" to_op="Apply Keras Model" to_port="model"/>
      <connect from_op="Apply Keras Model" from_port="labelled data" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Find more posts tagged with

AI Studio

Accepted answers

All comments

Thomas_Ott

This is a good question as I'm running into some problems trying to help @romanstrijbos out. It might have to do with me not completely understanding where and how to reshape the arrays in this extension.

Taking the S&P500 regression example (see modified XML code below), I first set the input shape to (1,2265). Why? Because the data delivered to the extension consists of 2265 examples with a target variable (aka label) and a regular attribute Close-0. I understand that in LSTM you have to reshape the data to sample, step, and dimensions. However, in RM the Reshape Layer excludes the sample size because it's set at 32 in the Keras operator. I then set the Recurrent Layer to LSTM with the Unit No set to 1, as that is my intended output dimensionality.

So where am I going wrong because it crashes.

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="7.6.001" expanded="true" height="68" name="Retrieve s&amp;p-500-data" width="90" x="45" y="85">
        <parameter key="repository_entry" value="//Keras Samples/sp_500_regression/s&amp;p-500-data"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="179" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="Close"/>
        <parameter key="attributes" value="Date|Open|Close"/>
      </operator>
      <operator activated="true" class="normalize" compatibility="7.6.001" expanded="true" height="103" name="Normalize" width="90" x="313" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="Close"/>
      </operator>
      <operator activated="true" class="series:windowing" compatibility="7.4.000" expanded="true" height="82" name="Windowing" width="90" x="447" y="85">
        <parameter key="window_size" value="1"/>
        <parameter key="create_label" value="true"/>
        <parameter key="label_attribute" value="Close"/>
      </operator>
      <operator activated="true" class="split_data" compatibility="7.6.001" expanded="true" height="103" name="Split Data" width="90" x="581" y="238">
        <enumeration key="partitions">
          <parameter key="ratio" value="0.9"/>
          <parameter key="ratio" value="0.1"/>
        </enumeration>
        <parameter key="sampling_type" value="linear sampling"/>
      </operator>
      <operator activated="true" class="keras:sequential" compatibility="1.0.003" expanded="true" height="166" name="Keras Model (2)" width="90" x="715" y="34">
        <parameter key="input shape" value="(1,2265)"/>
        <parameter key="loss" value="mean_absolute_percentage_error"/>
        <parameter key="optimizer" value="Adam"/>
        <enumeration key="metric"/>
        <parameter key="epochs" value="2"/>
        <enumeration key="callbacks">
          <parameter key="callbacks" value="None"/>
        </enumeration>
        <process expanded="true">
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Adding Reshape" width="90" x="179" y="34">
            <parameter key="layer_type" value="Reshape"/>
            <parameter key="target_shape" value="(1,17)"/>
            <parameter key="dims" value="1.1"/>
          </operator>
          <operator activated="true" class="keras:recurrent_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Recurrent Layer (2)" width="90" x="313" y="34">
            <parameter key="layer_type" value="LSTM"/>
            <parameter key="activation" value="softmax"/>
            <parameter key="recurrent_activation" value="sigmoid"/>
            <parameter key="kernel_regularizer" value="l1(0.01)"/>
            <parameter key="recurrent_regularizer" value="l1(0.01)"/>
            <parameter key="bias_regularizer" value="l1(0.01)"/>
          </operator>
          <connect from_op="Adding Reshape" from_port="layers 1" to_op="Add Recurrent Layer (2)" to_port="layers"/>
          <connect from_op="Add Recurrent Layer (2)" from_port="layers 1" to_port="layers 1"/>
          <portSpacing port="sink_layers 1" spacing="0"/>
          <portSpacing port="sink_layers 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="keras:apply" compatibility="1.0.003" expanded="true" height="82" name="Apply Keras Model" width="90" x="849" y="391">
        <parameter key="batch_size" value="128"/>
      </operator>
      <connect from_op="Retrieve s&amp;p-500-data" from_port="output" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Normalize" from_port="example set output" to_op="Windowing" to_port="example set input"/>
      <connect from_op="Windowing" from_port="example set output" to_op="Split Data" to_port="example set"/>
      <connect from_op="Split Data" from_port="partition 1" to_op="Keras Model (2)" to_port="training set"/>
      <connect from_op="Split Data" from_port="partition 2" to_op="Apply Keras Model" to_port="unlabelled data"/>
      <connect from_op="Keras Model (2)" from_port="model" to_op="Apply Keras Model" to_port="model"/>
      <connect from_op="Apply Keras Model" from_port="labelled data" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Ping: @dgrzech

romanstrijbos

Thanks for thinking with me here @Thomas_Ott. Let's hope there will be someone with more understanding of the Keras operators to help us out! Of course this is all pretty new so I think there's not much people who figured this out yet!

sgenzer

hi @romanstrijbos and @Thomas_Ott - I would highly advise reposting to the Keras thread as there's a long conversation there already. Also the Keras guru (who wrote the extension) watches that thread I believe.

Scott

Thomas_Ott

@sgenzer I thought about posting there but that's more an installation KB and I pinged @dgrzech above.

sgenzer

<grin> fair enough @Thomas_Ott. Maybe we should have a whole discussion section abt Keras considering its popularity?

Scott

romanstrijbos

Hi Thomas. Have you got the LSTM layer working already? Still pretty curious :-)

luc_bartkowski

Hi,

First: this isn't an LSTM issue, it's about the reshape layer.

But I think I have the answer for you.

With respect to the publisher, I send you this link.

Machine Learning Mastery.com: Reshape iinput data LSTM in Keras

If you rebuild the presented models in RapidMiner you would copy the (values) of the input_shape and target_shape.

IMO the answer is:

1) Batch has nothing to do with it.

2) The target_shape must contain the 3 parameters for (sample, time steps, features)

When you multiply the number of attributes of your input data with the number of examples (2D) and do also the multiplication of samples, time steps and features (3D), then both figures should be the same.