"Set MongoDB Instance with a macro, is it possible?"

christian_ploch
christian_ploch New Altair Community Member
edited November 2024 in Community Q&A
Hello everyone,

long time lurker, first time poster.
I make it fast:

I've got a process, which reads with help of the NoSQL Connectors Plugin from my MongoDB. 
So far I was able to set a macro, which says what Collection should be read.

Is it also possible to set a macro, which sets the MongoDB "Instance" (e.g. mongodb://localhost:27017/{%db_name})?

I tried several methods, but it's not working here. Anyone an idea? Thanks!

Welcome!

It looks like you're new here. Sign in or register to get started.

Best Answer

  • kayman
    kayman New Altair Community Member
    Answer ✓
    You cannot really set the database name itself with a macro (to my awareness), but the branch operator could be one option. Below example picks the right database based on a macro value.

    <?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="82" name="Set Macro" width="90" x="112" y="34">
            <parameter key="macro" value="database"/>
            <parameter key="value" value="mongo_1"/>
          </operator>
          <operator activated="true" class="branch" compatibility="9.1.000" expanded="true" height="82" name="choose" width="90" x="246" y="34">
            <parameter key="condition_type" value="expression"/>
            <parameter key="expression" value="equals(%{database}, &quot;mongo_1&quot;)"/>
            <parameter key="io_object" value="AHCDescribedCluster"/>
            <parameter key="return_inner_output" value="false"/>
            <process expanded="true">
              <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB" width="90" x="179" y="34">
                <parameter key="mongodb_instance" value="mongo_1"/>
                <parameter key="collection" value="coll1"/>
                <parameter key="sort_documents" value="false"/>
                <parameter key="limit_results" value="false"/>
                <parameter key="skip" value="0"/>
              </operator>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB (2)" width="90" x="179" y="34">
                <parameter key="mongodb_instance" value="mongo_2"/>
                <parameter key="collection" value="coll1"/>
                <parameter key="sort_documents" value="false"/>
                <parameter key="limit_results" value="false"/>
                <parameter key="skip" value="0"/>
              </operator>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Set Macro" from_port="through 1" to_op="choose" to_port="condition"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>
    

    Another, slightly more complex option, is to use the execute mongoDB command operator, where your command contains the 'macronised' collection name.

    There are a few issues, one is that the command doesn't like linebreaks which makes it a bit difficult to generate, but this can be fixed as in below example. The example is using an aggregate, but it would probably work fine with a more standard command also.

    If full flexibility is really needed you could try python instead

    <?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="68" name="Set Macro" width="90" x="45" y="34">
            <parameter key="macro" value="mycoll"/>
            <parameter key="value" value="some_collection"/>
          </operator>
          <operator activated="true" class="text:create_document" compatibility="8.1.000" expanded="true" height="68" name="Create Document (4)" width="90" x="179" y="34">
            <parameter key="text" value="{aggregate: &quot;%{mycoll}&quot;,pipeline: [&#10;{$match: { &quot;Something&quot;: &quot;something&quot; } },&#10;&#10;{$project:{&#10;...&#10;}}&#10;],cursor: {},allowDiskUse: true}"/>
            <parameter key="add label" value="false"/>
            <parameter key="label_type" value="nominal"/>
          </operator>
          <operator activated="true" class="text:replace_tokens" compatibility="8.1.000" expanded="true" height="68" name="Replace Tokens (8)" width="90" x="313" y="34">
            <list key="replace_dictionary">
              <parameter key="\n" value=" "/>
            </list>
          </operator>
          <operator activated="true" class="nosql:mongodb_command" compatibility="9.0.000" expanded="true" height="68" name="Execute MongoDB Command (5)" width="90" x="447" y="34">
            <parameter key="mongodb_instance" value="my_database"/>
          </operator>
          <connect from_op="Create Document (4)" from_port="output" to_op="Replace Tokens (8)" to_port="document"/>
          <connect from_op="Replace Tokens (8)" from_port="document" to_op="Execute MongoDB Command (5)" to_port="command"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>
    
    


Answers

  • kayman
    kayman New Altair Community Member
    Answer ✓
    You cannot really set the database name itself with a macro (to my awareness), but the branch operator could be one option. Below example picks the right database based on a macro value.

    <?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="82" name="Set Macro" width="90" x="112" y="34">
            <parameter key="macro" value="database"/>
            <parameter key="value" value="mongo_1"/>
          </operator>
          <operator activated="true" class="branch" compatibility="9.1.000" expanded="true" height="82" name="choose" width="90" x="246" y="34">
            <parameter key="condition_type" value="expression"/>
            <parameter key="expression" value="equals(%{database}, &quot;mongo_1&quot;)"/>
            <parameter key="io_object" value="AHCDescribedCluster"/>
            <parameter key="return_inner_output" value="false"/>
            <process expanded="true">
              <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB" width="90" x="179" y="34">
                <parameter key="mongodb_instance" value="mongo_1"/>
                <parameter key="collection" value="coll1"/>
                <parameter key="sort_documents" value="false"/>
                <parameter key="limit_results" value="false"/>
                <parameter key="skip" value="0"/>
              </operator>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="nosql:mongodb_document_reader" compatibility="9.0.000" expanded="true" height="103" name="Read MongoDB (2)" width="90" x="179" y="34">
                <parameter key="mongodb_instance" value="mongo_2"/>
                <parameter key="collection" value="coll1"/>
                <parameter key="sort_documents" value="false"/>
                <parameter key="limit_results" value="false"/>
                <parameter key="skip" value="0"/>
              </operator>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Set Macro" from_port="through 1" to_op="choose" to_port="condition"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>
    

    Another, slightly more complex option, is to use the execute mongoDB command operator, where your command contains the 'macronised' collection name.

    There are a few issues, one is that the command doesn't like linebreaks which makes it a bit difficult to generate, but this can be fixed as in below example. The example is using an aggregate, but it would probably work fine with a more standard command also.

    If full flexibility is really needed you could try python instead

    <?xml version="1.0" encoding="UTF-8"?><process version="9.1.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="9.1.000" expanded="true" name="Process">
        <parameter key="logverbosity" value="init"/>
        <parameter key="random_seed" value="2001"/>
        <parameter key="send_mail" value="never"/>
        <parameter key="notification_email" value=""/>
        <parameter key="process_duration_for_mail" value="30"/>
        <parameter key="encoding" value="SYSTEM"/>
        <process expanded="true">
          <operator activated="true" class="set_macro" compatibility="9.1.000" expanded="true" height="68" name="Set Macro" width="90" x="45" y="34">
            <parameter key="macro" value="mycoll"/>
            <parameter key="value" value="some_collection"/>
          </operator>
          <operator activated="true" class="text:create_document" compatibility="8.1.000" expanded="true" height="68" name="Create Document (4)" width="90" x="179" y="34">
            <parameter key="text" value="{aggregate: &quot;%{mycoll}&quot;,pipeline: [&#10;{$match: { &quot;Something&quot;: &quot;something&quot; } },&#10;&#10;{$project:{&#10;...&#10;}}&#10;],cursor: {},allowDiskUse: true}"/>
            <parameter key="add label" value="false"/>
            <parameter key="label_type" value="nominal"/>
          </operator>
          <operator activated="true" class="text:replace_tokens" compatibility="8.1.000" expanded="true" height="68" name="Replace Tokens (8)" width="90" x="313" y="34">
            <list key="replace_dictionary">
              <parameter key="\n" value=" "/>
            </list>
          </operator>
          <operator activated="true" class="nosql:mongodb_command" compatibility="9.0.000" expanded="true" height="68" name="Execute MongoDB Command (5)" width="90" x="447" y="34">
            <parameter key="mongodb_instance" value="my_database"/>
          </operator>
          <connect from_op="Create Document (4)" from_port="output" to_op="Replace Tokens (8)" to_port="document"/>
          <connect from_op="Replace Tokens (8)" from_port="document" to_op="Execute MongoDB Command (5)" to_port="command"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
        </process>
      </operator>
    </process>
    
    


  • christian_ploch
    christian_ploch New Altair Community Member
    Yeah, I suspected that, it is not possible, hoped i overlooked something.
    But I will give a try with your solutions @kayman

    appreciate it! Thanks!
  • MartinLiebig
    MartinLiebig
    Altair Employee
    @christian_ploch
    i think it's technically possible. you can hack the xml and just add %{macroName} at the right setting. Not a nice work around though.

    + We are working on it and the problem will be fixed "soon".

    BR,
    Martin
  • christian_ploch
    christian_ploch New Altair Community Member
    @mschmitz Yeah, already tried it with the process itself and also with the configurable-mongodb_instance.xml.
    If I play with the latter it throws me "Invalid JSON/BSON Object" (when I add the macro in the xml into the connection string). While the process gives me the "collection is not available".

    Thanks anyway :)

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.