Difficulties with macro value in scientific notation

lionelderkrikor

New Altair Community Member

Jan 20, 2019

Updated Nov 5, 2024 by Jocelyn

Hi all,

RapidMiner raises an error when a macro value is in scientific notation:

Image: https://us.v-cdn.net/6030995/uploads/editor/hh/zdchf0wlxw8p.png

The macro value is well recorded ...:

Image: https://us.v-cdn.net/6030995/uploads/editor/j5/zr3piyc2e6iu.png

...and is used in an operator :

Is there a solution / workaround to this ?

Thanks for your explanations...

Regards,

Lionel

NB : The process :

<?xml version="1.0" encoding="UTF-8"?><process version="9.2.000-SNAPSHOT">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.2.000-SNAPSHOT" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="operator_toolbox:create_exampleset" compatibility="1.7.000" expanded="true" height="68" name="Create ExampleSet" width="90" x="45" y="34">
        <parameter key="generator_type" value="date_series"/>
        <parameter key="number_of_examples" value="100"/>
        <parameter key="use_stepsize" value="true"/>
        <list key="function_descriptions"/>
        <parameter key="add_id_attribute" value="false"/>
        <list key="numeric_series_configuration"/>
        <list key="date_series_configuration"/>
        <list key="date_series_configuration (interval)">
          <parameter key="date" value="2019-01-01 00:00:00.1.hour"/>
        </list>
        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>
        <parameter key="column_separator" value=","/>
        <parameter key="parse_all_as_nominal" value="false"/>
        <parameter key="decimal_point_character" value="."/>
        <parameter key="trim_attribute_names" value="true"/>
      </operator>
      <operator activated="true" class="generate_id" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Generate ID" width="90" x="179" y="34">
        <parameter key="create_nominal_ids" value="false"/>
        <parameter key="offset" value="0"/>
      </operator>
      <operator activated="true" class="generate_data" compatibility="9.2.000-SNAPSHOT" expanded="true" height="68" name="Generate Data" width="90" x="45" y="136">
        <parameter key="target_function" value="random"/>
        <parameter key="number_examples" value="100"/>
        <parameter key="number_of_attributes" value="2"/>
        <parameter key="attributes_lower_bound" value="-10.0"/>
        <parameter key="attributes_upper_bound" value="10.0"/>
        <parameter key="gaussian_standard_deviation" value="10.0"/>
        <parameter key="largest_radius" value="10.0"/>
        <parameter key="use_local_random_seed" value="false"/>
        <parameter key="local_random_seed" value="1992"/>
        <parameter key="datamanagement" value="double_array"/>
        <parameter key="data_management" value="auto"/>
      </operator>
      <operator activated="true" class="generate_id" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Generate ID (2)" width="90" x="179" y="136">
        <parameter key="create_nominal_ids" value="false"/>
        <parameter key="offset" value="0"/>
      </operator>
      <operator activated="true" breakpoints="after" class="concurrency:join" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Join" width="90" x="313" y="85">
        <parameter key="remove_double_attributes" value="true"/>
        <parameter key="join_type" value="inner"/>
        <parameter key="use_id_attribute_as_key" value="true"/>
        <list key="key_attributes"/>
        <parameter key="keep_both_join_attributes" value="false"/>
      </operator>
      <operator activated="true" class="extract_macro" compatibility="9.2.000-SNAPSHOT" expanded="true" height="68" name="Extract Macro" width="90" x="447" y="85">
        <parameter key="macro" value="dateOffset"/>
        <parameter key="macro_type" value="data_value"/>
        <parameter key="statistics" value="average"/>
        <parameter key="attribute_name" value="date"/>
        <parameter key="example_index" value="1"/>
        <list key="additional_macros"/>
      </operator>
      <operator activated="true" class="generate_macro" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Generate Macro" width="90" x="581" y="85">
        <list key="function_descriptions">
          <parameter key="dateDiffOffset" value="date_diff(date_parse(&quot;01/01/1970&quot;),date_parse(%{dateOffset}))"/>
        </list>
      </operator>
      <operator activated="true" breakpoints="after" class="date_to_numerical" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Date to Numerical" width="90" x="715" y="85">
        <parameter key="attribute_name" value="date"/>
        <parameter key="time_unit" value="day"/>
        <parameter key="millisecond_relative_to" value="second"/>
        <parameter key="second_relative_to" value="minute"/>
        <parameter key="minute_relative_to" value="hour"/>
        <parameter key="hour_relative_to" value="day"/>
        <parameter key="day_relative_to" value="month"/>
        <parameter key="week_relative_to" value="year"/>
        <parameter key="month_relative_to" value="year"/>
        <parameter key="quarter_relative_to" value="year"/>
        <parameter key="half_year_relative_to" value="year"/>
        <parameter key="year_relative_to" value="era"/>
        <parameter key="keep_old_attribute" value="false"/>
      </operator>
      <operator activated="true" class="aggregate" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Aggregate" width="90" x="849" y="34">
        <parameter key="use_default_aggregation" value="false"/>
        <parameter key="attribute_filter_type" value="all"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value=""/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
        <parameter key="default_aggregation_function" value="average"/>
        <list key="aggregation_attributes">
          <parameter key="att1" value="average"/>
        </list>
        <parameter key="group_by_attributes" value="date"/>
        <parameter key="count_all_combinations" value="false"/>
        <parameter key="only_distinct" value="false"/>
        <parameter key="ignore_missings" value="true"/>
      </operator>
      <operator activated="true" class="numerical_to_date" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Numerical to Date" width="90" x="1117" y="34">
        <parameter key="attribute_name" value="date"/>
        <parameter key="keep_old_attribute" value="false"/>
        <parameter key="time_offset" value="%{dateDiffOffset}"/>
      </operator>
      <operator activated="true" class="concurrency:join" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Join (2)" width="90" x="1275" y="85">
        <parameter key="remove_double_attributes" value="true"/>
        <parameter key="join_type" value="inner"/>
        <parameter key="use_id_attribute_as_key" value="false"/>
        <list key="key_attributes">
          <parameter key="date" value="date"/>
        </list>
        <parameter key="keep_both_join_attributes" value="false"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="9.2.000-SNAPSHOT" expanded="true" height="82" name="Set Role" width="90" x="1448" y="85">
        <parameter key="attribute_name" value="date"/>
        <parameter key="target_role" value="id"/>
        <list key="set_additional_roles"/>
      </operator>
      <connect from_op="Create ExampleSet" from_port="output" to_op="Generate ID" to_port="example set input"/>
      <connect from_op="Generate ID" from_port="example set output" to_op="Join" to_port="left"/>
      <connect from_op="Generate Data" from_port="output" to_op="Generate ID (2)" to_port="example set input"/>
      <connect from_op="Generate ID (2)" from_port="example set output" to_op="Join" to_port="right"/>
      <connect from_op="Join" from_port="join" to_op="Extract Macro" to_port="example set"/>
      <connect from_op="Extract Macro" from_port="example set" to_op="Generate Macro" to_port="through 1"/>
      <connect from_op="Generate Macro" from_port="through 1" to_op="Date to Numerical" to_port="example set input"/>
      <connect from_op="Date to Numerical" from_port="example set output" to_op="Aggregate" to_port="example set input"/>
      <connect from_op="Date to Numerical" from_port="original" to_op="Join (2)" to_port="left"/>
      <connect from_op="Aggregate" from_port="example set output" to_op="Numerical to Date" to_port="example set input"/>
      <connect from_op="Numerical to Date" from_port="example set output" to_op="Join (2)" to_port="right"/>
      <connect from_op="Join (2)" from_port="join" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Find more posts tagged with

AI Studio

MacOS

Macros

Sort by:

1 - 16 of 161

MartinLiebig

Altair Employee

Jan 21, 2019

Updated Jan 21, 2019 by MartinLiebig

@lionelderkrikor

kind of known issue, even though this usually appears in another context. The point is, that Extract Macro always uses the default "toString" version for attributes. Usually you end up with a bit of formatting issues if you are working on Dates. But rarely (like in your case) also for numericals. Our currently proposed fix would be, to have a format option in Extract Macro to specify how to do the translation to string. Would this fix your problem?

Best,

Martin

CC: @sgenzer

Telcontar120

New Altair Community Member

Jan 21, 2019

@mschmitz I've run into this issue before as well. One question, would your proposed solution require all values to be formatted the same way? Because it is possible I believe for some values to be in standard notation and others to be in scientific notation since RapidMiner has a threshold built in which determines when it switches. In that case, would your solution work or would it still create errors?

MartinLiebig

Altair Employee

Jan 21, 2019

Hi @Telcontar120 ,

each Extract Macro operator would have one date and one number format. If you want to have different formats you would need to use two or more operators. Is that fine for you?

BR,
Martin

Telcontar120

New Altair Community Member

Jan 21, 2019

@mschmitz It's certainly an improvement over the current setup! I think there might still be cases where this would run into trouble but your proposed solution will probably cover the significant majority of use cases.

MartinLiebig

Altair Employee

Jan 21, 2019

@Telcontar120 I am open for other ideas, if you have some!

Telcontar120

New Altair Community Member

Jan 22, 2019

@mschmitz I think the only better solution would be to somehow allow Extract Macro to operate natively on numeric fields, in which case specifying formats for numerics would not be necessary, but I suspect that is not trivial since the underlying macro architecture is built around treating everything as a string.

MartinLiebig

Altair Employee

Jan 22, 2019

@Telcontar120 exactly. There is a workaround to use the hex representation of the numerical number as a string and add a function to Generate Attributes to translate it back. That could be one of the formating options for numericals.

lionelderkrikor

New Altair Community Member

Jan 22, 2019

Hi,

@Telcontar120 , @mschmitz
Thanks you for sharing your ideas.

@mschmitz ,
"Our currently proposed fix would be, to format option in Extract Macro to specify how to do the translation to string. Would this fix your problem ?"

OK, but the problematic macro value is generated by Generate Macro operator...
To be more precise, I'm extracting a macro value via Extract Macro operator :

Image: https://us.v-cdn.net/6030995/uploads/editor/pi/gtxg5wnhdc97.png

then I use Generate Macro operator to create the final (problematic) macro value... :

Image: https://us.v-cdn.net/6030995/uploads/editor/32/5u5s0igdr71z.png

Does your fix cover this case ?

Regards,

Lionel

MartinLiebig

Altair Employee

Jan 23, 2019

Updated Jan 23, 2019 by MartinLiebig

hi @lionelderkrikor ,

you need to use the right date_parse. namely:

date_parse_custom(%{dateOffset},"dd/MM/yyyy hh:mm:ss a z")

then it works.

Edit: Nevermind, this is not the case. my fix would not cover this issue, but i think what you need is

str(numeric, FORMAT)

in the Parser, right?

BR,

Martin

lionelderkrikor

New Altair Community Member

Jan 23, 2019

Hi @mschmitz,

str(numeric, FORMAT) don't fix the problem...
To be more complete, when I set directly the value (without macro) in scientific notation in the operator parameters :

Image: https://us.v-cdn.net/6030995/uploads/editor/dx/ni3l9hu4i2z6.png

....I have the same error...

Regards,

Lionel

MartinLiebig

Altair Employee

Jan 24, 2019

@lionelderkrikor

with the formating option you could enforce a 12 digit integer instead of scientific notation. Wouldnt that fix it?

BR,

Martin

lionelderkrikor

New Altair Community Member

Jan 24, 2019

Hi @mschmitz,

I can't set a second argument in the function str() :

Image: https://us.v-cdn.net/6030995/uploads/editor/a0/nf9rqjn9xg4v.png

Any ideas ?

Regards,

Lionel

MartinLiebig

Altair Employee

Accepted Answer

Jan 24, 2019

Hi @lionelderkrikor ,

my proposal was to add this function to the expression parser

. I would add this as a thing to Operator Toolbox.

BR,

Martin

lionelderkrikor

New Altair Community Member

Jan 24, 2019

@mschmitz,

Ah ah..

..OK, I misunderstood. OK for this solution.

Thanks Martin,

Regards,

Lionel

MartinLiebig

Altair Employee

Accepted Answer

Mar 29, 2019

Hi @lionelderkrikor,
just a heads up: The Operator Toolbox (2.0) now contains an operator called Extract Macro (Format) which allows you to set date_time and numerical formats. This should help a lot.

Best,
Martin

lionelderkrikor

New Altair Community Member

Apr 1, 2019

Hi @mschmitz,

Thanks you for this heads up. I updated my process with this new operator and indeed this helped a lot.

Have a nice day,

Regards,

Lionel

Sort by:

1 - 2 of 21

MartinLiebig

Altair Employee

Accepted Answer

Jan 24, 2019

Hi @lionelderkrikor ,

my proposal was to add this function to the expression parser

. I would add this as a thing to Operator Toolbox.

BR,

Martin

View in context