Enrich Data by Webservice debugging

sgenzer

hi...is there any way to debug the Enrich Data by Webservice operator when it fails?  I would to see the JSONpath that it is sending out in a POST request, and the actually reply from the server with the error code.  I can do it in curl but not via the RapidMiner interface.

In a similar vein, is there a way to send a curl request directly from RM?

And by the way, when it fails on a GET request, the salmon-colored error box is enormous (due to the large URL) and runs off the left side of the screen.  It is almost impossible to click the "Got it" button.

Scott

BalazsBaranyRM

Hi,

sgenzer wrote:
hi...is there any way to debug the Enrich Data by Webservice operator when it fails?  I would to see the JSONpath that it is sending out in a POST request, and the actually reply from the server with the error code.  I can do it in curl but not via the RapidMiner interface.

If your web service is accessible over HTTP, you can easily dump the entire communication with Wireshark.
If it needs HTTPS, you could try something like Fiddler <http://www.telerik.com/fiddler>.

In a similar vein, is there a way to send a curl request directly from RM?

You can use the Execute Program operator.

sgenzer

hmm thanks.  I was hoping to tap into the logging features in RM rather than something outside of it.

ah Execute Program.    I should have thought of that!  Duh. Thanks.  How do I pass the information into Execute Program so that I can use curl?  macros?

Scott

JEdward

It comes with handy output ports so you can use the result in RapidMiner. 

See this example:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.1.000-BETA">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <parameter key="parallelize_main_process" value="false"/>
    <process expanded="true">
      <operator activated="true" class="productivity:execute_program" compatibility="7.1.000-BETA" expanded="true" height="103" name="Execute Program" width="90" x="112" y="85">
        <parameter key="command" value="cmd /c java -version"/>
        <parameter key="log_stdout" value="true"/>
        <parameter key="log_stderr" value="true"/>
        <list key="env_variables"/>
      </operator>
      <operator activated="true" class="text:read_document" compatibility="7.0.000" expanded="true" height="68" name="Read Document (2)" width="90" x="380" y="136">
        <parameter key="extract_text_only" value="true"/>
        <parameter key="use_file_extension_as_type" value="true"/>
        <parameter key="content_type" value="txt"/>
        <parameter key="encoding" value="SYSTEM"/>
      </operator>
      <operator activated="true" class="text:read_document" compatibility="7.0.000" expanded="true" height="68" name="Read Document" width="90" x="380" y="34">
        <parameter key="extract_text_only" value="true"/>
        <parameter key="use_file_extension_as_type" value="true"/>
        <parameter key="content_type" value="txt"/>
        <parameter key="encoding" value="SYSTEM"/>
      </operator>
      <connect from_op="Execute Program" from_port="out" to_op="Read Document" to_port="file"/>
      <connect from_op="Execute Program" from_port="err" to_op="Read Document (2)" to_port="file"/>
      <connect from_op="Read Document (2)" from_port="output" to_port="result 2"/>
      <connect from_op="Read Document" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

sgenzer

ok yes that works.  Thank you.  It's still not solving my problem.  I am running this command:

curl -X GET --header 'Accept: text/plain' --header 'accept: text/plain' 'https://watson-api-explorer.mybluemix.net/language-translation/api/v2/identify?text=je suis faim'

if I run it in terminal, I get the correct response ("fr"), but in "Execute Program", I get an error message and it will not run.

Scott

BalazsBaranyRM

Hi!

You can use "Read Document" to convert the standard output and standard error of your process into readable text.

Example process:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="productivity:execute_program" compatibility="7.0.001" expanded="true" height="103" name="Execute Program" width="90" x="112" y="34">
        <parameter key="command" value="ls /tmp"/>
        <list key="env_variables"/>
      </operator>
      <operator activated="true" class="text:read_document" compatibility="6.5.000" expanded="true" height="68" name="Read stdout" width="90" x="380" y="34"/>
      <operator activated="true" class="text:read_document" compatibility="6.5.000" expanded="true" height="68" name="Read stderr" width="90" x="514" y="85"/>
      <connect from_op="Execute Program" from_port="out" to_op="Read stdout" to_port="file"/>
      <connect from_op="Execute Program" from_port="err" to_op="Read stderr" to_port="file"/>
      <connect from_op="Read stdout" from_port="output" to_port="result 1"/>
      <connect from_op="Read stderr" from_port="output" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Do you get the error message from RapidMiner or on the error output from curl? Do you need authentication for accessing the web service?

With curl -v you can get verbose output that shows the details of the communication with the server.

sgenzer

thanks.  No I do not get any errors when I use curl in the Unix terminal shell.  Only in "Execute Program".  In fact I cannot even get that to work when using your example, finding the java version.