Loop data sets and dynamically generated file path

Serek91

Hi,

I have subprocess with Write CSV operator. It is multiplied ~70 times. Output file has path like "{category_id}/{set_id}/filename.csv" So I want to have it dynamically generated. Can I create it somehow? Like putting to the subprocess two custom variables and then using it in filepath?

EDIT:

I'm using Loop Datasets operator. But after each iteration I have to somehow obtain index of current iteration and generate filepath...

Process added as attachment.

Marco_Boeck

Hi,

Just define a macro before the Loop, and then use it inside the Loop and increment it each iteration. Some Loops do it for you, but for your Loop you have to do it yourself. I would also recommend to check out Loop Collection, then you don't have to define 70 connections to the Loop Data Sets operator.. Anyway, It's quite easy, see the little example below:

<?xml version="1.0" encoding="UTF-8"?><process version="9.4.001-SNAPSHOT">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.4.001-SNAPSHOT" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.4.001-SNAPSHOT" expanded="true" height="68" name="Retrieve Iris" width="90" x="45" y="34">
        <parameter key="repository_entry" value="//Samples/data/Iris"/>
      </operator>
      <operator activated="true" class="set_macro" compatibility="9.4.001-SNAPSHOT" expanded="true" height="82" name="Prepare counter" width="90" x="179" y="34">
        <parameter key="macro" value="i"/>
        <parameter key="value" value="1"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="9.4.001-SNAPSHOT" expanded="true" height="124" name="Multiply" width="90" x="313" y="34"/>
      <operator activated="true" class="loop_data_sets" compatibility="9.4.001-SNAPSHOT" expanded="true" height="124" name="Loop Data Sets" width="90" x="447" y="34">
        <parameter key="only_best" value="false"/>
        <process expanded="true">
          <operator activated="true" class="store" compatibility="9.4.001-SNAPSHOT" expanded="true" height="68" name="Store" width="90" x="179" y="34">
            <parameter key="repository_entry" value="%{i} - myData"/>
          </operator>
          <operator activated="true" class="generate_macro" compatibility="9.4.001-SNAPSHOT" expanded="true" height="82" name="Increment counter" width="90" x="380" y="34">
            <list key="function_descriptions">
              <parameter key="i" value="eval(%{i})+1"/>
            </list>
          </operator>
          <connect from_port="example set" to_op="Store" to_port="input"/>
          <connect from_op="Store" from_port="through" to_op="Increment counter" to_port="through 1"/>
          <connect from_op="Increment counter" from_port="through 1" to_port="output 1"/>
          <portSpacing port="source_example set" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve Iris" from_port="output" to_op="Prepare counter" to_port="through 1"/>
      <connect from_op="Prepare counter" from_port="through 1" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Loop Data Sets" to_port="example set 1"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Loop Data Sets" to_port="example set 2"/>
      <connect from_op="Multiply" from_port="output 3" to_op="Loop Data Sets" to_port="example set 3"/>
      <connect from_op="Loop Data Sets" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Regards,
Marco