forgetful password

sgenzer

hi...I notice that when I am running a process where it is doing Enrich Data by Webservice many times (i.e. >1000 or so), RapidMiner "forgets" the authentication password that I told it to "remember" when I entered it the first time.  It seems sort of random when it does it, but it is rather annoying - particularly if I leave it running overnight and find out in the morning that after 4000 API calls, it now forgets the password and asks me to enter it again.

By the way, the API calls I am doing are using the new IBM Bluemix service.  It's pretty cool.

Any ideas?

Scott

Marco_Boeck

Hi,

we are talking about HTTP Basic auth, correct? If so, I highly suggest using the method below to pass the credentials instead of having to rely on the internal Studio authentication cache (which is not intended for this sort of thing):

https://username:password@www.example.com/webcallback?foo=bar

The credentials are protected by SSL in case of a https call, so this is actually safe to do.

Regards,
Marco

sgenzer

oh that very well do the trick.  Thanks very much.

Scott

kludikovsky

is there anything special I need to consider if I'd like to do this with MailChimp?

When I try to access MailChimps API I get an error in the brwoser as well as in Rapidminer when using the Base Auth as described here, even as this method is described in the documentation.

The error is:

{"type":"http://developer.mailchimp.com/documentation/mailchimp/guides/error-glossary/","title":"API Key Missing","status":401,"detail":"Your request did not include an API key.","instance":"4646a67f-8cb1-417a-914b-6fabf3539eb3"}

The call I used 

https://kll:686XXXXXXXXXXXXXXXXXXXXXXXXXXXXX-us16@us16.api.mailchimp.com/3.0/lists/

But when I use the RESTED extension in Chrome where I can separately enter the user/password credentials it works as expected.

 

Any hints?

kludikovsky

See BUG-Report: https://community.rapidminer.com/t5/RapidMiner-Studio-Forum/BUG-Enrich-Data-by-Webservice-Web-Mining-Does-not-pass-request/td-p/44049/jump-to/first-unread-message 

sgenzer

hi @kludikovsky - so this actually works fine.  Here's a sample call (API key deleted) with the JSON unparsed:

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data_user_specification" compatibility="7.6.001" expanded="true" height="68" name="Generate Data by User Specification" width="90" x="179" y="238">
        <list key="attribute_values"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice" width="90" x="313" y="238">
        <parameter key="query_type" value="Regular Expression"/>
        <list key="string_machting_queries"/>
        <list key="regular_expression_queries">
          <parameter key="foo" value=".*"/>
        </list>
        <list key="regular_region_queries"/>
        <list key="xpath_queries"/>
        <list key="namespaces"/>
        <list key="index_queries"/>
        <list key="jsonpath_queries"/>
        <parameter key="url" value="https://us7.api.mailchimp.com/schema/3.0/Root.json"/>
        <list key="request_properties">
          <parameter key="content-type" value="application/json"/>
          <parameter key="Authorization" value="apikey xxxxxxxxxxxxxxx-us7"/>
        </list>
      </operator>
      <connect from_op="Generate Data by User Specification" from_port="output" to_op="Enrich Data by Webservice" to_port="Example Set"/>
      <connect from_op="Enrich Data by Webservice" from_port="ExampleSet" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Scott

 

 

 

kludikovsky

Hi @sgenzer, 

I can repeat your process, even as there authentication.

I just c did a copy&paste into a new canvas and got a result. So I did no even need a API-key

But the protocol has no sign of the extra data sent:

Paket #3
GET /schema/3.0/Root.json HTTP/1.1
User-Agent: Java/1.8.0_141
Host: us7.api.mailchimp.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive


Paket #4
HTTP/1.1 200 OK
Server: openresty
Content-Type: application/json
Vary: Accept-Encoding
Date: Wed, 29 Nov 2017 16:32:35 GMT
Content-Length: 8760
Connection: keep-alive

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "title": "API Root",
  "description": "This resource is nothing more than links to other resources available through the API.",
  "properties": {
    "account_id": {

Could you try this please with /lists/. 

sgenzer

hello @kludikovsky - yes I see that.  I was just accessing the schema.  OK please be patient - I will investigate. 

 

Scott

 

sgenzer

OK got it

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="web:get_webpage" compatibility="7.3.000" expanded="true" height="68" name="Get Page" width="90" x="45" y="34">
        <parameter key="url" value="https://us7.api.mailchimp.com/3.0/lists"/>
        <parameter key="user_agent" value="rapidminer"/>
        <parameter key="accept_cookies" value="all"/>
        <list key="query_parameters"/>
        <list key="request_properties">
          <parameter key="Authorization" value="apikey xxxxxxxxx-us7"/>
        </list>
        <parameter key="override_encoding" value="true"/>
        <parameter key="encoding" value="UTF-8"/>
      </operator>
      <connect from_op="Get Page" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Scott

kludikovsky

@sgenzer confirm that it works with the "Get Page".

 

There are the request information included as can be seen in the trace:

GET /3.0/lists HTTP/1.1
User-Agent: rapidminer
Authorization: apikey 68xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us16
Host: us16.api.mailchimp.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Cookie: _AVESTA_ENVIRONMENT=prod


Paket #9
HTTP/1.1 200 OK
Server: openresty
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
X-Request-Id: 3f6c979e-f8f4-412a-bea0-8562753f2d06
Link: <https://us16.api.mailchimp.com/schema/3.0/Lists/Collection.json>; rel="describedBy"
Content-Length: 17816
Date: Wed, 29 Nov 2017 17:27:20 GMT
Connection: keep-alive
Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/

{"lists":[{"id":"5c771cab77","web_id":107653,"name":

While the same with "Enrich Data by Webservice" don't supply the information:

GET /3.0/lists/ HTTP/1.1
User-Agent: Java/1.8.0_141
Host: us16.api.mailchimp.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive


Paket #23
HTTP/1.1 401 Unauthorized
Server: openresty
Content-Type: application/problem+json; charset=utf-8
Content-Length: 229
X-Request-Id: 24037518-be5a-4db7-8c7b-38c33c63e998
Link: <https://us16.api.mailchimp.com/schema/3.0/ProblemDetailDocument.json>; rel="describedBy"
Date: Wed, 29 Nov 2017 17:37:00 GMT
Connection: keep-alive
Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/

{"type":"http://developer.mailchimp.com/documentation/mailchimp/guides/error-glossary/","title":"API Key Missing","status":401,"detail":"Your request did not include an API key.","instance":"24037518-be5a-4db7-8c7b-38c33c63e998"}

So this would conclude that the request properties are not passed on. Which possibly explains some other malfunction I have read on the web.

Thxs Scott.

 

sgenzer

yes exactly. Unfortunately the "Enrich Data via Webservice" has a lot of trouble with authentication - it's rather out-of-date at this point. I am finding that I see more success with "Get Page", and if that does not work, then "Execute Program" with some cURL statements or calling a shell script.

 

FWIW you may see a new extension dedicated completely to consuming the most popular APIs natively in RapidMiner (like we do already for Twitter) sometime in the future... 

 

Scott