Why does Keras LSTM expects 3 dimensions instead of the offered samples, time steps, and features?

luc_bartkowski

I want to classify a time sequence of OHLC data using LSTM.

This exampleset is converted to samples, time steps, and features to feed the LSTM using a reshape layer.

When I run this model I get an error message as shown in the following picture.

What am I doing wrong?

Screen Shot 2018-04-13 at 07.37.01.png

My model is listed below.

My input exampleset consist out of 2000 examples of OHLC data.

I reshape this data to 100 samples of 20 timesteps of 4 (OHLC) features.

My label is PhasesNum. It contains 3 classes indicated with the real values 10, 100, and 1000.

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.1.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="8.1.001" expanded="true" height="68" name="Retrieve Data" width="90" x="45" y="34">
        <parameter key="repository_entry" value="../Data/inputdata"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="8.1.001" expanded="true" height="82" name="Select Attributes" width="90" x="179" y="34">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="Close|High|Low|Open|PhasesNum"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
      </operator>
      <operator activated="true" class="numerical_to_real" compatibility="8.1.001" expanded="true" height="82" name="Numerical to Real" width="90" x="313" y="34">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="PhasesNum"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="numeric"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="real"/>
        <parameter key="block_type" value="value_series"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_series_end"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="8.1.001" expanded="true" height="82" name="Set Role" width="90" x="447" y="34">
        <parameter key="attribute_name" value="PhasesNum"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles">
          <parameter key="Close" value="regular"/>
          <parameter key="High" value="regular"/>
          <parameter key="Low" value="regular"/>
          <parameter key="Open" value="regular"/>
        </list>
      </operator>
      <operator activated="true" class="filter_example_range" compatibility="8.1.001" expanded="true" height="82" name="Filter Example Range" width="90" x="45" y="136">
        <parameter key="first_example" value="1"/>
        <parameter key="last_example" value="2000"/>
        <parameter key="invert_filter" value="false"/>
      </operator>
      <operator activated="true" class="normalize" compatibility="8.1.001" expanded="true" height="103" name="Normalize" width="90" x="179" y="136">
        <parameter key="return_preprocessing_model" value="false"/>
        <parameter key="create_view" value="false"/>
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="Close|High|Low|Open"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="numeric"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="real"/>
        <parameter key="block_type" value="value_series"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_series_end"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="method" value="Z-transformation"/>
        <parameter key="min" value="0.0"/>
        <parameter key="max" value="1.0"/>
        <parameter key="allow_negative_values" value="false"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="8.1.001" expanded="true" height="103" name="Multiply" width="90" x="313" y="136"/>
      <operator activated="true" class="keras:sequential" compatibility="1.0.003" expanded="true" height="166" name="Keras Model" width="90" x="246" y="289">
        <parameter key="input shape" value="(2000,4)"/>
        <parameter key="loss" value="categorical_crossentropy"/>
        <parameter key="optimizer" value="Adam"/>
        <parameter key="learning rate" value="0.001"/>
        <parameter key="momentum" value="0.0"/>
        <parameter key="rho" value="0.9"/>
        <parameter key="beta 1" value="0.999"/>
        <parameter key="beta 2" value="0.999"/>
        <parameter key="epsilon" value="1.0E-8"/>
        <parameter key="decay" value="0.0"/>
        <parameter key="schedule decay" value="0.004"/>
        <parameter key="Nesterov" value="false"/>
        <parameter key="use metric" value="false"/>
        <enumeration key="metric"/>
        <parameter key="epochs" value="128"/>
        <parameter key="batch size" value="100"/>
        <enumeration key="callbacks">
          <parameter key="callbacks" value="TensorBoard(log_dir='./logs', histogram_freq=0, write_graph=True, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)"/>
        </enumeration>
        <parameter key="verbose" value="1"/>
        <parameter key="validation split" value="0.0"/>
        <parameter key="shuffle" value="false"/>
        <parameter key="fix seed" value="false"/>
        <parameter key="random seed" value="0"/>
        <process expanded="true">
          <operator activated="true" class="keras:core_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Core Layer" width="90" x="112" y="34">
            <parameter key="layer_type" value="Reshape"/>
            <parameter key="no_units" value="1"/>
            <parameter key="activation_function" value="None"/>
            <parameter key="use_bias" value="true"/>
            <parameter key="kernel_initializer" value="glorot_uniform(seed=None)"/>
            <parameter key="bias_initializer" value="Zeros()"/>
            <parameter key="kernel_regularizer" value="None"/>
            <parameter key="bias_regularizer" value="None"/>
            <parameter key="activity_regularizer" value="None"/>
            <parameter key="kernel_constraint" value="None"/>
            <parameter key="bias_constraint" value="None"/>
            <parameter key="rate" value="0.1"/>
            <parameter key="noise_shape" value="None"/>
            <parameter key="seed" value="None"/>
            <parameter key="target_shape" value="(100,20,4)"/>
            <parameter key="dims" value="1.1"/>
            <parameter key="repetition_factor" value="1"/>
            <parameter key="function" value="None"/>
            <parameter key="l1" value="0.0"/>
            <parameter key="l2" value="0.0"/>
            <parameter key="mask_value" value="0.0"/>
          </operator>
          <operator activated="true" class="keras:recurrent_layer" compatibility="1.0.003" expanded="true" height="82" name="Add Recurrent Layer" width="90" x="313" y="34">
            <parameter key="layer_type" value="LSTM"/>
            <parameter key="no_units" value="3"/>
            <parameter key="activation" value="softmax"/>
            <parameter key="recurrent_activation" value="sigmoid"/>
            <parameter key="use_bias" value="true"/>
            <parameter key="kernel_initializer" value="glorot_uniform(seed=None)"/>
            <parameter key="recurrent_initializer" value="glorot_uniform(seed=None)"/>
            <parameter key="bias_initializer" value="Zeros()"/>
            <parameter key="unit_forget_bias" value="true"/>
            <parameter key="kernel_regularizer" value="None"/>
            <parameter key="recurrent_regularizer" value="None"/>
            <parameter key="bias_regularizer" value="None"/>
            <parameter key="activity_regularizer" value="None"/>
            <parameter key="kernel_constraint" value="None"/>
            <parameter key="recurrent_constraint" value="None"/>
            <parameter key="bias_constraint" value="None"/>
            <parameter key="dropout" value="0.0"/>
            <parameter key="recurrent_dropout" value="0.0"/>
            <parameter key="stateful" value="true"/>
            <parameter key="unroll" value="false"/>
            <parameter key="implementation" value="0"/>
          </operator>
          <connect from_op="Add Core Layer" from_port="layers 1" to_op="Add Recurrent Layer" to_port="layers"/>
          <connect from_op="Add Recurrent Layer" from_port="layers 1" to_port="layers 1"/>
          <portSpacing port="sink_layers 1" spacing="0"/>
          <portSpacing port="sink_layers 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="keras:apply" compatibility="1.0.003" expanded="true" height="82" name="Apply Keras Model" width="90" x="447" y="289">
        <parameter key="batch_size" value="100"/>
        <parameter key="verbose" value="0"/>
      </operator>
      <connect from_op="Retrieve Data" from_port="output" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Numerical to Real" to_port="example set input"/>
      <connect from_op="Numerical to Real" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Filter Example Range" to_port="example set input"/>
      <connect from_op="Filter Example Range" from_port="example set output" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Normalize" from_port="example set output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Apply Keras Model" to_port="unlabelled data"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Keras Model" to_port="training set"/>
      <connect from_op="Keras Model" from_port="model" to_op="Apply Keras Model" to_port="model"/>
      <connect from_op="Apply Keras Model" from_port="labelled data" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Find more posts tagged with

AI Studio

Sampling

Deep Learning + Neural Nets

Accepted answers

All comments

luc_bartkowski

Additional information:

Can't find a reason why the following object would return the error message in my previous post.

If anyone has a suggestion, please let me know.

Screen Shot 2018-04-13 at 20.05.01.png

I added a Dense layer as suggested in the following post from the Github keras-team/keras to study the model.

Screen Shot 2018-04-13 at 20.07.16.png

Screen Shot 2018-04-13 at 20.05.01.png

Screen Shot 2018-04-13 at 20.07.16.png

israel_jimenez

Did you got any solution to this issue? I don't see any problem with your process or with your layer's architecture.

sgenzer

cc @pschlunder

pschlunder

Hi @luc_bartkowski,

the input shape for a recurrent layer needs to start with the time steps. So (time steps, input dimension) would be the shape you need. In your case this should be (20, 4). You might want to use the windowing operator of the value series extension to create an example set with window size 20 first. Please check the sample process "s&p-500-regression" in the Keras Samples folder inside "sp_500_regression".

Furthermore your last layer should be a dense layer containing one neuron for each class value, so in your case 3.

BTW: Make sure to always check the "Help" view of RapidMiner Studio for information about Operator handling and help setting the parameters.

Hope this helps,

Philipp

luc_bartkowski

Hi, here's the how-to regarding RapidMiner Keras LSTM input-shape:

1st: the problem domain: My input data is financial OHLC so 4 attributes/features.
Additional assumption: Window size in the Window operator = 10, so 10 timesteps.

I want to do classification, not prediction. Therefore I don't generate a label in the Windowing operator. If you do generate a label in the Window operator, or you use another amount of timesteps, horizon, etc. you have to adjust the parameters below accordingly.

So in my application his results into an Keras operator input example set off 'n' records with 4*10 = 40 normal (windowed) attributes and a label. The Keras operator input shape is "(40,)" including parentheses and the comma.
The Add Core Layer Reshape operator, that you add as the first operator within the Keras operator, should get a target shape of "(10,4)".
This is the point where I suspect everybody goes wrong: Yes, we all use RNN but not as the FIRST operator in the Keras operator but it's the SECOND operator. Therefore output shape consist out of 2 parameters, not 3 (which generates an error).

This solved my issue. No problems with training anymore.
Hope this helps.