Get Page with query parameters

JEdward

Hello I'm trying to use the Get Page operator & finding a problem I haven't done before.  After I put in the query parameters RapidMiner adds an additional '&' after the line when it sends it to the web.  This is causing the REST service to reply back saying that the request is invalid. 

It seems a little strict, but why is it that RM is adding the '&'?  Has anyone else faced a similar issue? 

Below is an example of the process I mean, it would generate the request URL "https://webservice.com/detail/number/site-here?withParameter=true&"

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.012">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.012" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="web:get_webpage" compatibility="5.3.001" expanded="true" height="60" name="Get Page" width="90" x="112" y="75">
        <parameter key="url" value="https://webservice.com/detail/number/site-here"/>
        <parameter key="follow_redirects" value="false"/>
        <list key="query_parameters">
          <parameter key="withParameter" value="true"/>
        </list>
        <list key="request_properties">
          <parameter key="Content-Type" value="application/xml"/>
          <parameter key="Authorization" value="Basic xxxx=="/>
        </list>
        <parameter key="override_encoding" value="true"/>
        <parameter key="encoding" value="UTF-8"/>
      </operator>
      <connect from_op="Get Page" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

MariusHelf

Hi,

thanks for reporting this issue. I have created an internal ticket for this. Until it is fixed you can add the query parameters manually directly into the url parameter.

Best regards,
Marius

JEdward

Hi Marius,

I didn't realise it was a bug as it also does the same when I manually entered the parameters as a URL. 
I'll try a few workarounds & post up a working one to this thread. 

Best,
J.

JEdward

Just to update.  I haven't yet found a workable workaround for this. 
Marius suggestion above is still having the same issue on my machine, passing the URL to the operator as a macro (RM is still adding the additional '&' at the end). 

I'll keep searching & let you all know. 

JEdward

Just to update: in the version of RapidMiner I use (5.3) this has not been fixed.  Can anyone confirm if it's fixed in RapidMiner 6+? 

Also I now have a work around which involves writing a new version of the Get Page operator using a bunch of macros alongside the Execute Script operator with Groovy HTTPBuilder. 
http://groovy.codehaus.org/modules/http-builder/download.html

It's not pretty and has to be heavily customized for each use case so I'm afraid the process isn't in a fit state to share, however, I wish this post this might help someone who comes across the same problem. Feel free to contact me if you need any assistance or would like me to share some of the 'ugly script'. 

MariusHelf

Hi J,

I am sorry, this issue is still marked as open in our bug tracker.

Best regards,
Marius