Macro/url query parameter malfunctioning in RM web service.

pari1234

Hi there,

 

I have setup a rapidminer webservice with a url query parameter bound to a macro with a default value in the underlying process context. But when testing the web service (in admin console or calling the API url), I enter a different value for the query parameter (or in other words the macro in the underlying process), I still get results for the default value of the macro and not the new value that I just tried to pass. It seems the user entered values are not getting passed to the underlying process. So when testing, even if I do not pass anything, I still get results with the default value for the macro. Any ideas why this might be happening? I remember there was an old video (RM 5 if I recall correctly) about this on youtube but I just can't find it.

 

 

Thank you very much!

pari1234

Hi @sgenzer, sorry about not being able to reply sooner and thank you very much for taking the time to go through this process. You are right about the server logs error messages and access tokens. Anyways, I made some changes to the process such that instead of passing url parameters, I now just send a JSON array in a POST request to the RM webservice API. 

[{
	"platformName": "twitter",
	"username": "rel8edto"
}, {
	"platformName": "facebook",
	"username": "rel8edto"
}]

This then gets converted to an example set using the read document, JSON to Data and De-Pivot operators and using appropriate filters and extracting macro inside the process, I can pass the username to the appropriate branch (facebook or twitter). I still haven't figured out exactly what happened with being able to passing macros. I created an identical server client setup at home and it worked just fine. So I'm guessing something might have led to a bug in our RM Server at some point. But thank you very much for your help. :smileyhappy:

Thomas_Ott

This feature still works in v7+. I would double check the process to make sure the macro is defined in there as well. 

sgenzer

hello @pari1234 - welcome to the community.  Yes passing macros back and forth can be tricky.  Have you set up the context parameters in the process to receive the macro from server?

 

Scott

 

pari1234

Thank you @sgenzer and @Thomas_Ott for your responses. I have defined the macro in process context as well. My process contains auth tokens for Aylien, Twitter and Facebook, would you like me to post the xml here? I've attached a screenshot of my process with the context panel and a screenshot of the service settings page on server.

 

Ironically, the first version with only the twitter component worked perfectly. Then after I added the facebook component and overwrote the previous process, this issue started occurring. I then tried to isolate the issue by creating a fresh process with just the twitter components and the macro and the issue was there as well. Next, I kept just a single twitter operator with a macro and the issue persisted even then. Not sure if it's a server setting I'm missing or what....

 

Does this help? Let me know if you'd need anything else. Thank you. 

Edin_Klapic

Hi @pari1234,

 

Could it be a problem of the execution order of the Operators within your process?

I guess you use the Operator "Generate Data by User Specification" for some kind of if - else statement.

If the Twitter / Facebook branch is executed before this Operator the default macro value is taken.

 

To activate the execution order view click on the blue rectangle symbol in the upper right corner of your process canvas.

Then you can change the order and click on the symbol again to exit this view.

 

By the way, for debugging I propose to use the Operator "Print to Console" as first Operator in your process and enter the macro there.

Whenever the process is executed, the Macro value is printed in the Server log file.

 

Best,

Edin

pari1234

Thank you @Edin_Klapic but this solution did not work. Besides I'm sorry but I found changing the execution order to be very cumbersome. It's difficult to simply drag and drop to change the execition order. I was able to set the top Generate-Data-By-User-Specificaiton operator as the first one in the execution order by using 'bring-to-front', but that was pretty much it. Also if you read my previous response to @sgenzerand @Thomas_Ott, I have said that this problem persists even when I use just a single twitter operator and a macro in the porcess context and nothing else in my entire process while testing as a web service.

 

For print to console, can you tell mewhere will I find the server log? I mean do I have to manually go into my server file system to find a log file? Found it. Not sure which log file should I be looking at under Administration/SystemInformation/ServerLogs in the admin console. Will it also log the events if I make a call to the webservice from an external tool such as 'curl' or 'Postman'?

 

Thank you.

sgenzer

hi @pari1234 - so to change the execution order, you just click on the numbered circles in the order you want. 

 

Here's a quick video: https://rapidminer-my.sharepoint.com/personal/sgenzer_rapidminer_com/_layouts/15/guestaccess.aspx?docid=0a3710a88c8c249d686b86d8fac1780b3&authkey=Afqijaoku5aG2AMyyRBlvaE&expiration=2018-01-02T14%3A43%3A29.000Z&e=c2c861d081f542b5913b7da6653401ca

 

You'll notice that sometimes you're prevented from changing the order because of the way the operators are wired.  This is purposeful of course.  All I'm doing in the video is clicking the numbers.

Scott

 

Edin_Klapic

Hi @pari1234,

 

The actual log file is always server.log.

Unfortunately, there is no auto refresh. So if you executed a process you need to reload the file manually.

 

Best,

Edin

pari1234

I understand @sgenzer and thank you. Looking at your video, it seems I've done exactly what you show. And as you said, some of the orders cant be changed because of the way the operators are wired. But this still isn't solving the issue. Sorry and thanks.

pari1234

Thank you @Edin_Klapic, it may sound weird but I do not even see a server log file for today's date 2017-11-03. The file name would be something like server.log.2017-11-03 right?

 

The macro name for print-to-console operator is 'log'.

Edin_Klapic

The actual server.log has no date. Just "server.log"

The macro to enter in Print to Console should be the Macro name you defined in the context of your process (see screenshot)

 

Best,

Edin

 

 

 

 

pari1234

@Edin_Klapic@sgenzer@Thomas_Ott Here's the process minus the fb auth token. Not sure if process context is included in the xml, if not, then the macro in process context is called - 'twitter', feel free to use whatever default value you want. I'm sure you guys have your own tokens for Twitter and Aylien. I couldn't demo it to our client today on the call so I guess I have more time now. If it runs fine for you guys then I think there might be something going on in our RM Server. Here's how you can  get an access token from fb which you will then enter in the part of the url parameter for Enrich Data by Webservice operator. 

 

• 1) Register at developers.facebook.com
• 2) Create an app, use the wizard to complete.
• 3) Obtain app id and app secret
• 4) (Optional) Add a "facebook login" to your app, so that users can give permission to your app for viewing their content.
• 5) In your browser type the following url - https://graph.facebook.com/oauth/access_token?client_id={YOUR_APP_ID}&client_secret={YOUR_SECRET}&grant_type=client_credentials and hit "enter".

It will return an access token that looks like (not a real token)-

-

{
"access_token": "1745625463738593|w_a8sakjhflaYsCHPTOTDr5H1r-wY", 
"token_type": "bearer"
}

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
 <context>
 <input/>
 <output/>
 <macros>
 <macro>
 <key>twitter</key>
 <value>rel8edto</value>
 </macro>
 </macros>
 </context>
 <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
 <process expanded="true">
 <operator activated="true" class="generate_data_user_specification" compatibility="7.5.001" expanded="true" height="68" name="Generate Data by User Specification (2)" width="90" x="514" y="34">
 <list key="attribute_values">
 <parameter key="overall_social_rating" value="&quot;Coming Soon!&quot;"/>
 </list>
 <list key="set_additional_roles"/>
 </operator>
 <operator activated="true" class="print_to_console" compatibility="7.5.001" expanded="true" height="68" name="Print to Console" width="90" x="246" y="34">
 <parameter key="log_value" value="serverlog"/>
 </operator>
 <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="124" name="Twitter" width="90" x="514" y="136">
 <parameter key="exception_macro" value="exception_twitter"/>
 <process expanded="true">
 <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="238">
 <parameter key="connection" value="Twitter Server"/>
 <parameter key="query" value="@%{twitter}"/&gt;
 <parameter key="limit" value="40"/>
 </operator>
 <operator activated="true" class="com.aylien.textapi.rapidminer:aylien_sentiment" compatibility="0.2.000" expanded="true" height="68" name="Analyze Sentiment" width="90" x="313" y="238">
 <parameter key="connection" value="AylienServer"/>
 <parameter key="input_attribute" value="Text"/>
 </operator>
 <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Aggregate(twitter sentiment)" width="90" x="447" y="238">
 <list key="aggregation_attributes">
 <parameter key="polarity" value="count (percentage)"/>
 </list>
 <parameter key="group_by_attributes" value="polarity"/>
 </operator>
 <operator activated="true" class="social_media:get_twitter_user_statuses" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Statuses" width="90" x="112" y="136">
 <parameter key="connection" value="Twitter Server"/>
 <parameter key="user" value="%{twitter}"/>
 <parameter key="limit" value="40"/>
 </operator>
 <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Avg RT" width="90" x="313" y="136">
 <list key="aggregation_attributes">
 <parameter key="Retweet-Count" value="average"/>
 </list>
 </operator>
 <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="112" y="34">
 <parameter key="connection" value="Twitter Server"/>
 <parameter key="user" value="%{twitter}"/>
 </operator>
 <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="calculate account age" width="90" x="313" y="34">
 <list key="function_descriptions">
 <parameter key="account age (in days)" value="(date_diff([Created-At], date_now())*1.15741e-8)"/>
 </list>
 </operator>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Post Processing (2)" width="90" x="447" y="34">
 <process expanded="true">
 <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename User Details (twit)" width="90" x="246" y="34">
 <parameter key="old_name" value="Created-At"/>
 <parameter key="new_name" value="Start Date"/>
 <list key="rename_additional_attributes">
 <parameter key="Description" value="description"/>
 <parameter key="Favourites" value="favourites"/>
 <parameter key="Followers" value="followers"/>
 <parameter key="Friends" value="friends"/>
 <parameter key="Id" value="id"/>
 <parameter key="Language" value="language"/>
 <parameter key="Location" value="location"/>
 <parameter key="Name" value="name"/>
 <parameter key="Tweets" value="tweets"/>
 <parameter key="URL" value="url"/>
 </list>
 </operator>
 <operator activated="true" class="select_attributes" compatibility="7.5.001" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="34">
 <parameter key="attribute_filter_type" value="subset"/>
 <parameter key="attributes" value="url|tweets|name|location|language|id|friends|followers|favourites|description|account age (in days)"/>
 </operator>
 <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="Generate platformName" width="90" x="581" y="34">
 <list key="function_descriptions">
 <parameter key="platformName" value="&quot;twitter&quot;"/>
 </list>
 </operator>
 <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename (4)" width="90" x="246" y="136">
 <parameter key="old_name" value="average(Retweet-Count)"/>
 <parameter key="new_name" value="average retweet count"/>
 <list key="rename_additional_attributes"/>
 </operator>
 <connect from_port="in 1" to_op="Rename User Details (twit)" to_port="example set input"/>
 <connect from_port="in 2" to_op="Rename (4)" to_port="example set input"/>
 <connect from_op="Rename User Details (twit)" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
 <connect from_op="Select Attributes" from_port="example set output" to_op="Generate platformName" to_port="example set input"/>
 <connect from_op="Generate platformName" from_port="example set output" to_port="out 1"/>
 <connect from_op="Rename (4)" from_port="example set output" to_port="out 2"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="source_in 3" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 </process>
 </operator>
 <connect from_op="Search Twitter" from_port="output" to_op="Analyze Sentiment" to_port="Example Set"/>
 <connect from_op="Analyze Sentiment" from_port="Example Set" to_op="Aggregate(twitter sentiment)" to_port="example set input"/>
 <connect from_op="Aggregate(twitter sentiment)" from_port="example set output" to_port="out 3"/>
 <connect from_op="Get Twitter User Statuses" from_port="output" to_op="Avg RT" to_port="example set input"/>
 <connect from_op="Avg RT" from_port="example set output" to_op="Post Processing (2)" to_port="in 2"/>
 <connect from_op="Get Twitter User Details" from_port="output" to_op="calculate account age" to_port="example set input"/>
 <connect from_op="calculate account age" from_port="example set output" to_op="Post Processing (2)" to_port="in 1"/>
 <connect from_op="Post Processing (2)" from_port="out 1" to_port="out 1"/>
 <connect from_op="Post Processing (2)" from_port="out 2" to_port="out 2"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 <portSpacing port="sink_out 4" spacing="0"/>
 </process>
 <process expanded="true">
 <operator activated="true" class="log" compatibility="7.5.001" expanded="true" height="82" name="Log (2)" width="90" x="179" y="34">
 <list key="log"/>
 </operator>
 <connect from_port="in 1" to_op="Log (2)" to_port="through 1"/>
 <connect from_op="Log (2)" from_port="through 1" to_port="out 1"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 <portSpacing port="sink_out 4" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="generate_data_user_specification" compatibility="7.5.001" expanded="true" height="68" name="Generate Data by User Specification" width="90" x="246" y="340">
 <list key="attribute_values">
 <parameter key="Username" value="%{twitter}"/>
 </list>
 <list key="set_additional_roles"/>
 </operator>
 <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="124" name="Facebook" width="90" x="514" y="340">
 <parameter key="exception_macro" value="exception_facebook"/>
 <process expanded="true">
 <operator activated="true" class="multiply" compatibility="7.5.001" expanded="true" height="103" name="Multiply" width="90" x="45" y="34"/>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="82" name="User Details" width="90" x="313" y="289">
 <process expanded="true">
 <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice (5)" width="90" x="246" y="85">
 <parameter key="query_type" value="Regular Expression"/>
 <list key="string_machting_queries"/>
 <list key="regular_expression_queries">
 <parameter key="jsonResponse" value=".*"/>
 </list>
 <list key="regular_region_queries"/>
 <list key="xpath_queries"/>
 <list key="namespaces"/>
 <list key="index_queries"/>
 <list key="jsonpath_queries">
 <parameter key="message" value="$..message"/>
 <parameter key="post id" value="$..id"/>
 </list>
 <parameter key="url" value="https://graph.facebook.com/v2.10/&amp;lt;%Username%&amp;gt;?access_token=&amp;lt;your-access-token&amp;gt;&amp;amp;fields=username,name,about,fan_count,location,start_info,overall_star_rating,rating_count,talking_about_count"/>
 <list key="request_properties"/>
 <parameter key="encoding" value="UTF-8"/>
 </operator>
 <operator activated="true" class="loop_examples" compatibility="7.5.001" expanded="true" height="103" name="Loop Examples (3)" width="90" x="380" y="85">
 <process expanded="true">
 <operator activated="true" class="filter_example_range" compatibility="7.5.001" expanded="true" height="82" name="Filter Example Range (3)" width="90" x="45" y="34">
 <parameter key="first_example" value="%{example}"/>
 <parameter key="last_example" value="%{example}"/>
 </operator>
 <operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents (3)" width="90" x="179" y="34">
 <parameter key="select_attributes_and_weights" value="true"/>
 <list key="specify_weights">
 <parameter key="jsonResponse" value="1.0"/>
 </list>
 </operator>
 <operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents (3)" width="90" x="313" y="34"/>
 <operator activated="true" class="web:json_to_xml" compatibility="7.3.000" expanded="true" height="68" name="JSON to XML (3)" width="90" x="447" y="34"/>
 <operator activated="true" class="text:write_document" compatibility="7.5.000" expanded="true" height="82" name="Write Document (3)" width="90" x="581" y="34">
 <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\userDetails2.xml"/>
 <parameter key="encoding" value="UTF-16"/>
 </operator>
 <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Read XML (3)" width="90" x="715" y="34">
 <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\userDetails2.xml"/>
 <parameter key="xpath_for_examples" value="//json"/>
 <enumeration key="xpaths_for_attributes">
 <parameter key="xpath_for_attribute" value="username[1]/text()"/>
 <parameter key="xpath_for_attribute" value="name[1]/text()"/>
 <parameter key="xpath_for_attribute" value="about[1]/text()"/>
 <parameter key="xpath_for_attribute" value="fan_count[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/street[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/city[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/state[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/country[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/zip[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/latitude[1]/text()"/>
 <parameter key="xpath_for_attribute" value="location[1]/longitude[1]/text()"/>
 <parameter key="xpath_for_attribute" value="start_info[1]/date[1]/year[1]/text()"/>
 <parameter key="xpath_for_attribute" value="start_info[1]/type[1]/text()"/>
 <parameter key="xpath_for_attribute" value="rating_count[1]/text()"/>
 <parameter key="xpath_for_attribute" value="overall_star_rating[1]/text()"/>
 <parameter key="xpath_for_attribute" value="talking_about_count[1]/text()"/>
 </enumeration>
 <list key="namespaces"/>
 <parameter key="use_default_namespace" value="false"/>
 <list key="annotations"/>
 <list key="data_set_meta_data_information">
 <parameter key="0" value="username[1]/text().true.attribute_value.attribute"/>
 <parameter key="1" value="name[1]/text().true.attribute_value.attribute"/>
 <parameter key="2" value="about[1]/text().true.attribute_value.attribute"/>
 <parameter key="3" value="fan_count[1]/text().true.attribute_value.attribute"/>
 <parameter key="4" value="location[1]/street[1]/text().true.attribute_value.attribute"/>
 <parameter key="5" value="location[1]/city[1]/text().true.attribute_value.attribute"/>
 <parameter key="6" value="location[1]/state[1]/text().true.attribute_value.attribute"/>
 <parameter key="7" value="location[1]/country[1]/text().true.attribute_value.attribute"/>
 <parameter key="8" value="location[1]/zip[1]/text().true.attribute_value.attribute"/>
 <parameter key="9" value="location[1]/latitude[1]/text().true.attribute_value.attribute"/>
 <parameter key="10" value="location[1]/longitude[1]/text().true.attribute_value.attribute"/>
 <parameter key="11" value="start_info[1]/date[1]/year[1]/text().true.attribute_value.attribute"/>
 <parameter key="12" value="start_info[1]/type[1]/text().true.attribute_value.attribute"/>
 <parameter key="13" value="rating_count[1]/text().true.attribute_value.attribute"/>
 <parameter key="14" value="overall_star_rating[1]/text().true.attribute_value.attribute"/>
 <parameter key="15" value="talking_about_count[1]/text().true.attribute_value.attribute"/>
 </list>
 </operator>
 <connect from_port="example set" to_op="Filter Example Range (3)" to_port="example set input"/>
 <connect from_op="Filter Example Range (3)" from_port="example set output" to_op="Data to Documents (3)" to_port="example set"/>
 <connect from_op="Data to Documents (3)" from_port="documents" to_op="Combine Documents (3)" to_port="documents 1"/>
 <connect from_op="Combine Documents (3)" from_port="document" to_op="JSON to XML (3)" to_port="document"/>
 <connect from_op="JSON to XML (3)" from_port="document" to_op="Write Document (3)" to_port="document"/>
 <connect from_op="Write Document (3)" from_port="file" to_op="Read XML (3)" to_port="file"/>
 <connect from_op="Read XML (3)" from_port="output" to_port="output 1"/>
 <portSpacing port="source_example set" spacing="0"/>
 <portSpacing port="sink_example set" spacing="0"/>
 <portSpacing port="sink_output 1" spacing="0"/>
 <portSpacing port="sink_output 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="82" name="Union Append (3)" width="90" x="514" y="85">
 <process expanded="true">
 <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (3)" width="90" x="45" y="34">
 <parameter key="set_iteration_macro" value="true"/>
 <process expanded="true">
 <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (4)" width="90" x="112" y="34">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (4)" width="90" x="313" y="34">
 <parameter key="condition_type" value="expression"/>
 <parameter key="expression" value="%{iteration}==1"/>
 <process expanded="true">
 <connect from_port="condition" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 <process expanded="true">
 <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (3)" width="90" x="45" y="187">
 <parameter key="name" value="LoopData"/>
 </operator>
 <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (User Details)" width="90" x="179" y="34"/>
 <connect from_port="condition" to_op="Union (User Details)" to_port="example set 1"/>
 <connect from_op="Recall (3)" from_port="result" to_op="Union (User Details)" to_port="example set 2"/>
 <connect from_op="Union (User Details)" from_port="union" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (3)" width="90" x="581" y="34">
 <parameter key="name" value="LoopData"/>
 </operator>
 <connect from_port="single" to_op="Branch (4)" to_port="condition"/>
 <connect from_op="Branch (4)" from_port="input 1" to_op="Remember (3)" to_port="store"/>
 <connect from_op="Remember (3)" from_port="stored" to_port="output 1"/>
 <portSpacing port="source_single" spacing="0"/>
 <portSpacing port="sink_output 1" spacing="0"/>
 <portSpacing port="sink_output 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (7)" width="90" x="179" y="34">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <connect from_port="in 1" to_op="Output (3)" to_port="collection"/>
 <connect from_op="Output (3)" from_port="output 1" to_op="Select (7)" to_port="collection"/>
 <connect from_op="Select (7)" from_port="selected" to_port="out 1"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 </process>
 </operator>
 <connect from_port="in 1" to_op="Enrich Data by Webservice (5)" to_port="Example Set"/>
 <connect from_op="Enrich Data by Webservice (5)" from_port="ExampleSet" to_op="Loop Examples (3)" to_port="example set"/>
 <connect from_op="Loop Examples (3)" from_port="output 1" to_op="Union Append (3)" to_port="in 1"/>
 <connect from_op="Union Append (3)" from_port="out 1" to_port="out 1"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Sentiment and Avg. Share" width="90" x="246" y="34">
 <process expanded="true">
 <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice (2)" width="90" x="112" y="34">
 <parameter key="query_type" value="Regular Expression"/>
 <list key="string_machting_queries"/>
 <list key="regular_expression_queries">
 <parameter key="jsonResponse" value=".*"/>
 </list>
 <list key="regular_region_queries"/>
 <list key="xpath_queries"/>
 <list key="namespaces"/>
 <list key="index_queries"/>
 <list key="jsonpath_queries">
 <parameter key="message" value="$..message"/>
 <parameter key="post id" value="$..id"/>
 </list>
 <parameter key="url" value="https://graph.facebook.com/v2.10/&amp;lt;%Username%&amp;gt;/posts?access_token=&amp;lt;your_access_token&amp;gt;&amp;amp;fields=message,link,created_time,type,name,id,comments.limit(10),shares&amp;amp;limit=100"/>
 <list key="request_properties"/>
 <parameter key="encoding" value="UTF-8"/>
 </operator>
 <operator activated="true" class="loop_examples" compatibility="7.5.001" expanded="true" height="124" name="Loop Examples" width="90" x="246" y="34">
 <process expanded="true">
 <operator activated="true" class="filter_example_range" compatibility="7.5.001" expanded="true" height="82" name="Filter Example Range" width="90" x="45" y="34">
 <parameter key="first_example" value="%{example}"/>
 <parameter key="last_example" value="%{example}"/>
 </operator>
 <operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents" width="90" x="179" y="34">
 <parameter key="select_attributes_and_weights" value="true"/>
 <list key="specify_weights">
 <parameter key="jsonResponse" value="1.0"/>
 </list>
 </operator>
 <operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents" width="90" x="313" y="34"/>
 <operator activated="true" class="web:json_to_xml" compatibility="7.3.000" expanded="true" height="68" name="JSON to XML" width="90" x="447" y="34"/>
 <operator activated="true" class="text:write_document" compatibility="7.5.000" expanded="true" height="82" name="Write Document" width="90" x="581" y="34">
 <parameter key="file" value="/Users/GenzerConsulting/OneDrive - RapidMiner/OneDrive Repository/random community stuff/jsonExport.xml"/>
 <parameter key="encoding" value="UTF-16"/>
 </operator>
 <operator activated="true" class="multiply" compatibility="7.5.001" expanded="true" height="103" name="Multiply (2)" width="90" x="715" y="136"/>
 <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Comments (2)" width="90" x="849" y="289">
 <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\jsonDump.xml"/>
 <parameter key="xpath_for_examples" value="//json/data/comments"/>
 <enumeration key="xpaths_for_attributes">
 <parameter key="xpath_for_attribute" value="data[1]/created_time[1]/text()"/>
 <parameter key="xpath_for_attribute" value="data[1]/from[1]/name[1]/text()"/>
 <parameter key="xpath_for_attribute" value="data[1]/from[1]/id[1]/text()"/>
 <parameter key="xpath_for_attribute" value="data[1]/id[1]/text()"/>
 <parameter key="xpath_for_attribute" value="data[1]/message[1]/text()"/>
 </enumeration>
 <list key="namespaces"/>
 <parameter key="use_default_namespace" value="false"/>
 <list key="annotations"/>
 <list key="data_set_meta_data_information">
 <parameter key="0" value="data[1]/created_time[1]/text().true.attribute_value.attribute"/>
 <parameter key="1" value="data[1]/from[1]/name[1]/text().true.attribute_value.attribute"/>
 <parameter key="2" value="data[1]/from[1]/id[1]/text().true.attribute_value.attribute"/>
 <parameter key="3" value="data[1]/id[1]/text().true.attribute_value.attribute"/>
 <parameter key="4" value="data[1]/message[1]/text().true.attribute_value.attribute"/>
 </list>
 </operator>
 <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Posts (2)" width="90" x="849" y="34">
 <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\jsonDump.xml"/>
 <parameter key="xpath_for_examples" value="//json/data"/>
 <enumeration key="xpaths_for_attributes">
 <parameter key="xpath_for_attribute" value="shares[1]/count[1]/text()"/>
 <parameter key="xpath_for_attribute" value="created_time[1]/text()"/>
 <parameter key="xpath_for_attribute" value="link[1]/text()"/>
 <parameter key="xpath_for_attribute" value="id[1]/text()"/>
 <parameter key="xpath_for_attribute" value="message[1]/text()"/>
 <parameter key="xpath_for_attribute" value="type[1]/text()"/>
 <parameter key="xpath_for_attribute" value="name[1]/text()"/>
 </enumeration>
 <list key="namespaces"/>
 <parameter key="use_default_namespace" value="false"/>
 <list key="annotations"/>
 <list key="data_set_meta_data_information">
 <parameter key="0" value="shares[1]/count[1]/text().true.numeric.attribute"/>
 <parameter key="1" value="created_time[1]/text().true.attribute_value.attribute"/>
 <parameter key="2" value="link[1]/text().true.attribute_value.attribute"/>
 <parameter key="3" value="id[1]/text().true.attribute_value.attribute"/>
 <parameter key="4" value="message[1]/text().true.attribute_value.attribute"/>
 <parameter key="5" value="type[1]/text().true.attribute_value.attribute"/>
 <parameter key="6" value="name[1]/text().true.attribute_value.attribute"/>
 </list>
 </operator>
 <connect from_port="example set" to_op="Filter Example Range" to_port="example set input"/>
 <connect from_op="Filter Example Range" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
 <connect from_op="Data to Documents" from_port="documents" to_op="Combine Documents" to_port="documents 1"/>
 <connect from_op="Combine Documents" from_port="document" to_op="JSON to XML" to_port="document"/>
 <connect from_op="JSON to XML" from_port="document" to_op="Write Document" to_port="document"/>
 <connect from_op="Write Document" from_port="file" to_op="Multiply (2)" to_port="input"/>
 <connect from_op="Multiply (2)" from_port="output 1" to_op="Posts (2)" to_port="file"/>
 <connect from_op="Multiply (2)" from_port="output 2" to_op="Comments (2)" to_port="file"/>
 <connect from_op="Comments (2)" from_port="output" to_port="output 2"/>
 <connect from_op="Posts (2)" from_port="output" to_port="output 1"/>
 <portSpacing port="source_example set" spacing="0"/>
 <portSpacing port="sink_example set" spacing="0"/>
 <portSpacing port="sink_output 1" spacing="0"/>
 <portSpacing port="sink_output 2" spacing="0"/>
 <portSpacing port="sink_output 3" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Union Append" width="90" x="380" y="34">
 <process expanded="true">
 <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (4)" width="90" x="179" y="34">
 <parameter key="set_iteration_macro" value="true"/>
 <process expanded="true">
 <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (5)" width="90" x="112" y="136">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (2)" width="90" x="313" y="34">
 <parameter key="condition_type" value="expression"/>
 <parameter key="expression" value="%{iteration}==1"/>
 <process expanded="true">
 <connect from_port="condition" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 <process expanded="true">
 <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (5)" width="90" x="45" y="187">
 <parameter key="name" value="LoopData"/>
 </operator>
 <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (Posts)" width="90" x="179" y="34"/>
 <connect from_port="condition" to_op="Union (Posts)" to_port="example set 1"/>
 <connect from_op="Recall (5)" from_port="result" to_op="Union (Posts)" to_port="example set 2"/>
 <connect from_op="Union (Posts)" from_port="union" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (5)" width="90" x="581" y="34">
 <parameter key="name" value="LoopData"/>
 </operator>
 <connect from_port="single" to_op="Branch (2)" to_port="condition"/>
 <connect from_op="Branch (2)" from_port="input 1" to_op="Remember (5)" to_port="store"/>
 <connect from_op="Remember (5)" from_port="stored" to_port="output 1"/>
 <portSpacing port="source_single" spacing="0"/>
 <portSpacing port="sink_output 1" spacing="0"/>
 <portSpacing port="sink_output 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Posts" width="90" x="447" y="34">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (2)" width="90" x="179" y="187">
 <parameter key="set_iteration_macro" value="true"/>
 <process expanded="true">
 <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (2)" width="90" x="112" y="85">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (3)" width="90" x="313" y="34">
 <parameter key="condition_type" value="expression"/>
 <parameter key="expression" value="%{iteration}==1"/>
 <process expanded="true">
 <connect from_port="condition" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 <process expanded="true">
 <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (2)" width="90" x="45" y="187">
 <parameter key="name" value="LoopData"/>
 </operator>
 <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (Comments)" width="90" x="179" y="34"/>
 <connect from_port="condition" to_op="Union (Comments)" to_port="example set 1"/>
 <connect from_op="Recall (2)" from_port="result" to_op="Union (Comments)" to_port="example set 2"/>
 <connect from_op="Union (Comments)" from_port="union" to_port="input 1"/>
 <portSpacing port="source_condition" spacing="0"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_input 1" spacing="0"/>
 <portSpacing port="sink_input 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (2)" width="90" x="581" y="34">
 <parameter key="name" value="LoopData"/>
 </operator>
 <connect from_port="single" to_op="Branch (3)" to_port="condition"/>
 <connect from_op="Branch (3)" from_port="input 1" to_op="Remember (2)" to_port="store"/>
 <connect from_op="Remember (2)" from_port="stored" to_port="output 1"/>
 <portSpacing port="source_single" spacing="0"/>
 <portSpacing port="sink_output 1" spacing="0"/>
 <portSpacing port="sink_output 2" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Comments" width="90" x="447" y="187">
 <parameter key="index" value="%{iteration}"/>
 </operator>
 <connect from_port="in 1" to_op="Output (4)" to_port="collection"/>
 <connect from_port="in 2" to_op="Output (2)" to_port="collection"/>
 <connect from_op="Output (4)" from_port="output 1" to_op="Posts" to_port="collection"/>
 <connect from_op="Posts" from_port="selected" to_port="out 1"/>
 <connect from_op="Output (2)" from_port="output 1" to_op="Comments" to_port="collection"/>
 <connect from_op="Comments" from_port="selected" to_port="out 2"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="source_in 3" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Avg Share Rate" width="90" x="849" y="34">
 <list key="aggregation_attributes">
 <parameter key="shares[1]/count[1]/text()" value="average"/>
 </list>
 </operator>
 <operator activated="true" class="com.aylien.textapi.rapidminer:aylien_sentiment" compatibility="0.2.000" expanded="true" height="68" name="Analyze Sentiment (2)" width="90" x="581" y="136">
 <parameter key="connection" value="AylienServer"/>
 <parameter key="input_attribute" value="data[1]/message[1]/text()"/>
 </operator>
 <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Aggregate (facebook sentiment)" width="90" x="849" y="187">
 <list key="aggregation_attributes">
 <parameter key="polarity" value="count (percentage)"/>
 </list>
 <parameter key="group_by_attributes" value="polarity"/>
 </operator>
 <connect from_port="in 1" to_op="Enrich Data by Webservice (2)" to_port="Example Set"/>
 <connect from_op="Enrich Data by Webservice (2)" from_port="ExampleSet" to_op="Loop Examples" to_port="example set"/>
 <connect from_op="Loop Examples" from_port="output 1" to_op="Union Append" to_port="in 1"/>
 <connect from_op="Loop Examples" from_port="output 2" to_op="Union Append" to_port="in 2"/>
 <connect from_op="Union Append" from_port="out 1" to_op="Avg Share Rate" to_port="example set input"/>
 <connect from_op="Union Append" from_port="out 2" to_op="Analyze Sentiment (2)" to_port="Example Set"/>
 <connect from_op="Avg Share Rate" from_port="example set output" to_port="out 1"/>
 <connect from_op="Analyze Sentiment (2)" from_port="Example Set" to_op="Aggregate (facebook sentiment)" to_port="example set input"/>
 <connect from_op="Aggregate (facebook sentiment)" from_port="example set output" to_port="out 2"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 </process>
 </operator>
 <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Post Processing" width="90" x="447" y="136">
 <process expanded="true">
 <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename User Details (fb)" width="90" x="179" y="136">
 <parameter key="old_name" value="about[1]/text()"/>
 <parameter key="new_name" value="description"/>
 <list key="rename_additional_attributes">
 <parameter key="fan_count[1]/text()" value="fans"/>
 <parameter key="location[1]/city[1]/text()" value="city"/>
 <parameter key="location[1]/country[1]/text()" value="country"/>
 <parameter key="location[1]/latitude[1]/text()" value="latitude"/>
 <parameter key="location[1]/longitude[1]/text()" value="longitude"/>
 <parameter key="location[1]/state[1]/text()" value="state"/>
 <parameter key="location[1]/street[1]/text()" value="street"/>
 <parameter key="location[1]/zip[1]/text()" value="zip"/>
 <parameter key="name[1]/text()" value="name"/>
 <parameter key="overall_star_rating[1]/text()" value="facebook star rating"/>
 <parameter key="rating_count[1]/text()" value="number of ratings"/>
 <parameter key="start_info[1]/date[1]/year[1]/text()" value="start date"/>
 <parameter key="start_info[1]/type[1]/text()" value="start date type"/>
 <parameter key="talking_about_count[1]/text()" value="number of people talking about this page"/>
 <parameter key="username[1]/text()" value="username"/>
 </list>
 </operator>
 <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="Generate platformName (2)" width="90" x="380" y="136">
 <list key="function_descriptions">
 <parameter key="platformName" value="&quot;facebook&quot;"/>
 </list>
 </operator>
 <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename" width="90" x="179" y="34">
 <parameter key="old_name" value="average(shares[1]/count[1]/text())"/>
 <parameter key="new_name" value="av share rate"/>
 <list key="rename_additional_attributes"/>
 </operator>
 <connect from_port="in 1" to_op="Rename" to_port="example set input"/>
 <connect from_port="in 2" to_op="Rename User Details (fb)" to_port="example set input"/>
 <connect from_op="Rename User Details (fb)" from_port="example set output" to_op="Generate platformName (2)" to_port="example set input"/>
 <connect from_op="Generate platformName (2)" from_port="example set output" to_port="out 2"/>
 <connect from_op="Rename" from_port="example set output" to_port="out 1"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="source_in 3" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 </process>
 </operator>
 <connect from_port="in 1" to_op="Multiply" to_port="input"/>
 <connect from_op="Multiply" from_port="output 1" to_op="Sentiment and Avg. Share" to_port="in 1"/>
 <connect from_op="Multiply" from_port="output 2" to_op="User Details" to_port="in 1"/>
 <connect from_op="User Details" from_port="out 1" to_op="Post Processing" to_port="in 2"/>
 <connect from_op="Sentiment and Avg. Share" from_port="out 1" to_op="Post Processing" to_port="in 1"/>
 <connect from_op="Sentiment and Avg. Share" from_port="out 2" to_port="out 3"/>
 <connect from_op="Post Processing" from_port="out 1" to_port="out 1"/>
 <connect from_op="Post Processing" from_port="out 2" to_port="out 2"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 <portSpacing port="sink_out 4" spacing="0"/>
 </process>
 <process expanded="true">
 <operator activated="true" class="log" compatibility="7.5.001" expanded="true" height="82" name="Log" width="90" x="179" y="34">
 <list key="log"/>
 </operator>
 <connect from_port="in 1" to_op="Log" to_port="through 1"/>
 <connect from_op="Log" from_port="through 1" to_port="out 1"/>
 <portSpacing port="source_in 1" spacing="0"/>
 <portSpacing port="source_in 2" spacing="0"/>
 <portSpacing port="sink_out 1" spacing="0"/>
 <portSpacing port="sink_out 2" spacing="0"/>
 <portSpacing port="sink_out 3" spacing="0"/>
 <portSpacing port="sink_out 4" spacing="0"/>
 </process>
 </operator>
 <operator activated="false" class="free_memory" compatibility="7.5.001" expanded="true" height="68" name="Free Memory" width="90" x="916" y="391"/>
 <connect from_op="Generate Data by User Specification (2)" from_port="output" to_port="result 1"/>
 <connect from_op="Twitter" from_port="out 1" to_port="result 2"/>
 <connect from_op="Twitter" from_port="out 2" to_port="result 3"/>
 <connect from_op="Twitter" from_port="out 3" to_port="result 4"/>
 <connect from_op="Generate Data by User Specification" from_port="output" to_op="Facebook" to_port="in 1"/>
 <connect from_op="Facebook" from_port="out 1" to_port="result 5"/>
 <connect from_op="Facebook" from_port="out 2" to_port="result 6"/>
 <connect from_op="Facebook" from_port="out 3" to_port="result 7"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_result 1" spacing="0"/>
 <portSpacing port="sink_result 2" spacing="0"/>
 <portSpacing port="sink_result 3" spacing="0"/>
 <portSpacing port="sink_result 4" spacing="0"/>
 <portSpacing port="sink_result 5" spacing="0"/>
 <portSpacing port="sink_result 6" spacing="0"/>
 <portSpacing port="sink_result 7" spacing="0"/>
 <portSpacing port="sink_result 8" spacing="0"/>
 </process>
 </operator>
</process>

 

 Thank you.

pari1234

And here's the server log file. I could not figure out what's the issue. In a particular section it says something about twitter API rate limit, but that should not be related to the process macro simply called "twitter" right?

pari1234

@sgenzer, @Edin_Klapic, @Thomas_Ott Okay so this seems to be a common issue regardless of what kind of process I have. Is this a bug? I created a simple webservice with Generate Data by User Specification (attribute: "data", attribute value: %{value}) and a Print to Console operator. A macro named 'value' was created in process context with some default value.

 

So all the process does is return the value of the attribute "data". When I test the service with some other macro value, it still returns the default value. Hope that helps.

Thomas_Ott

Ah, there's something interesting in your server log. It fails right at execution at the Generate Exampleset operator. It says it's a dummy, which means that it's probably not installed on your RM Server installation. Check the operators and extensions tab to see if that operator is there. 

 

OR it means that the data you are pulling from that exampleset is not on the Server but your local machine.

pari1234

Thank you @Thomas_Ott. May I ask if you mean the "Create Exampleset" operator when you say Generate Exampleset? If yes, then I resolved it last night. That is why I have "Generate Data by User Specification" instead now and I double-checked to see whether that operator exists in server or not (it exists). If not, then I'm not sure what more can I try.

 

The process inherently does not contain any data. It is meant to be a part of a larger data delivery framework. It will receive incoming data via the url query parameter from another component within the framework and spit out social analysis results. And that is what I'm trying to test by passing it a twitter/facebook username as a url parameter bound to the macro.

 

Sincerely,

Pari

Thomas_Ott

Yeah I meant Create Exampleset.

 

From inspecting your process, it appears that you are passing a twitter name as %{twitter} macro. Makes complete sense but you use the same twitter handle as a FB name. Will that even work? My Twitter handle is '@neuralmarket' but you can use it as my username for FB. 

 

If you disabled the FB process and run only the Twitter one, does it work locally and then on the Server? Is the Twitter connection on the Server authenticated as well as your local machine?

 

And, where is your %{example} macro defined? I see you use it for your Filter Range but where are the default values stored/defined.

pari1234

@Thomas_Ott, for now I'm only using it to test usernames that have the same name on both twitter and facebook. The production version will incorporate a separate macro for facebook. But I want to make sure that I can pass parameters successfully. 

 

Even if I remove the facebook component, the problem still remains. Executing the process locally does not make sense to me since that means I'm just changing the default value of the macro everytime I run it. My Twitter connection is properly authenticated both on server and locally.

 

I'll mention this again- to see if the problem is specific to the process, I created a completely new process with just one Generate Data by User Specification operator and the process just returned the default value of context macro. When testing it as a webservice with a new value for the macro in url parameter, I still got the default value returned to me. So either I'm doing something dramatically wrong on the server, or something in RM server has gone corrupt since until yesterday afternoon, the twitter component was running perfectly on its own. Problems started popping up when I added the facebook component, but I don't understand what could be wrong there.

 

As for the macro %{example} in filter_example_range, I'm simply using that block from the solution provided by @sgenzer to my previous thread here in the community. The macro seems to be defined in the parameter settings for the Loop Examples operator which is being used after the Enrich Data by Webservice operator in two different places in the Facebook subprocess. That's where default values for %{example} are coming from. 

 

Hope this helps. Thank you.

pari1234

@Thomas_Ott I have a workaround for this that seems to be working when tested partially. It involves passing operator parameters in the url instead of macro values. So I can pass operator parameter to each of the twitter operators. But in the Generate Data by User Specification operator parameter settings (the facebook part), it takes in a list (arrtibutename, attribute value) and not simply a field value. How do I pass a list to that operator via the url on server?

 

Thank you.

sgenzer

Hi @pari1234 - sorry I'm not getting back to this until now.  So I retraced your steps locally on Studio (not Server) and your process works fine.  I'm going to try to retrace your steps on a Server and see how that goes.

Scott

 

sgenzer

hello again @pari1234 - I played around for a while on a server and could not get the Facebook API to work on Server at all.  Could you?  I did not even get to the part of passing macros back and forth, etc... There are some pieces to this thread that may shed some light on this...

 

- As others have mentioned above, whenever you see "Dummy" operators in RapidMiner Server, you have one of two problems: 1) as @Thomas_Ott said, you have not installed the extension needed to run that operator, or 2) the version of the extension installed on Server is not compatible with the version of Server.  When I look at the top of your log, I see this:

 

The dummy operator Create ExampleSet (replacing operator_toolbox:create_exampleset_from_doc) cannot be executed

 So it appears that you're using the Operator Toolbox extension in this process when it generated the error - except that I don't see it in your XML!  Hmmph.  You are using the Web Mining, Text Processing, and Aylien extensions for sure - so you need to ensure that these extensions are installed on your server AND are compatible with the version of Server you're using.

 

- There are some other errors in that log that I will defer to the Server experts here to explain better (@Edin_Klapic or others?).  Working from the bottom, I see the following:

 

15:30:15,611 SEVERE [com.rapidminer.operator.Operator] (http-/0.0.0.0:8080-1) [AYLIEN] Too Many Requests: 1000 out of 1000 hits per day
15:30:15,663 SEVERE [com.rapidminer.operator.Operator] (http-/0.0.0.0:8080-1) [AYLIEN] Too Many Requests: 1000 out of 1000 hits per day

 

So this is for sure a API query limit.

 

15:30:15,124 WARNING [com.rapidminer] (http-/0.0.0.0:8080-1) Failed to retrieve secret key: /root/.RapidMiner/cipher.key (No such file or directory)

 

This is only a warning but it concerns me.  If I remember correctly, cipher.key stores various security features for RapidMiner but my memory escapes me exactly why this would happen.  @Edin_Klapic?

 

Server returned HTTP response code: 400 for URL: https://graph.facebook.com/v2.10/rel8edto?access_token=<your-access-token>&fields=username,name,about,fan_count,location,start_info,overall_star_rating,rating_count,talking_about_count

 

So here I'm assuming (?) that you deleted your token in the log manually and you actually did pass the token.  If so, then I was getting the exact same error and I think it's a funky Facebook thing (see below).  If not, clearly the token is not getting inserted in to the URL and this cannot be a macro issue.

 

- I looked at Facebook API tech docs for a while (I've been interested in using this for quite some time, so thank you for showing how to get the tokens, etc..- big help!).  It seems that Facebook has quite the apparatus going on with their APIs, including a funky "appsecret_proof" security feature that I have never seen before in all my API work.  They also say at the top of this link that, "Almost every Graph API call requires an access token. Malicious developers can steal access tokens and use them to send spam from your app. Facebook has automated systems to detect this, but you can help us secure your app by adding extra parameters to API requests. This document explains how." [emphasis is mine].  I always wondered about the security of sending access tokens directly in the url like it's normally done - it seems that Facebook agrees that more security is needed.  I have NOT played with this at all but I have a sneaky suspicion that their "automated systems" are messing with things.

 

Anyway those are my 10 cents on this very interesting process.  Maybe some of this helps?

 

Scott

 

pari1234

Hi @sgenzer, sorry about not being able to reply sooner and thank you very much for taking the time to go through this process. You are right about the server logs error messages and access tokens. Anyways, I made some changes to the process such that instead of passing url parameters, I now just send a JSON array in a POST request to the RM webservice API. 

[{
	"platformName": "twitter",
	"username": "rel8edto"
}, {
	"platformName": "facebook",
	"username": "rel8edto"
}]

This then gets converted to an example set using the read document, JSON to Data and De-Pivot operators and using appropriate filters and extracting macro inside the process, I can pass the username to the appropriate branch (facebook or twitter). I still haven't figured out exactly what happened with being able to passing macros. I created an identical server client setup at home and it worked just fine. So I'm guessing something might have led to a bug in our RM Server at some point. But thank you very much for your help. :smileyhappy:

sgenzer

oh that's great, @pari1234.  Well done.  Could you post your process so I can see?  I'd be very interested in your solution.

Scott

pari1234

Thank you @sgenzer, and below is the process. You'll need to use your own fb access token and twitter and aylien connections. I'm thinking of making a video about this process (since RM already has operators for twitter). Other than my main job, I also work as a Teaching Assistant for graduate business analytics students at the local university and I think this will be VERY helpful to those students who are scurrying for a social media data collection tool in their text mining semester project. Most of them do not come from a technical background and this would definitely expand the scope of their learning outside the classroom.

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data_user_specification" compatibility="7.5.001" expanded="true" height="68" name="Generate Data (OverallRatingComingSoon)" width="90" x="715" y="34">
        <list key="attribute_values">
          <parameter key="overall_social_rating" value="rand()"/>
        </list>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="format_numbers" compatibility="7.5.001" expanded="true" height="82" name="Format Numbers" width="90" x="916" y="34">
        <parameter key="format_type" value="pattern"/>
        <parameter key="pattern" value="0.00"/>
      </operator>
      <operator activated="true" class="text:read_document" compatibility="7.5.000" expanded="true" height="68" name="Read Document" width="90" x="45" y="34"/>
      <operator activated="true" class="text:json_to_data" compatibility="7.5.000" expanded="true" height="82" name="JSON To Data" width="90" x="179" y="34">
        <parameter key="minimal_examples_(absolute)" value="1"/>
      </operator>
      <operator activated="true" class="de_pivot" compatibility="7.5.001" expanded="true" height="82" name="De-Pivot (2)" width="90" x="313" y="34">
        <list key="attribute_name">
          <parameter key="platformName" value="\[.*]\.platformName"/>
          <parameter key="username" value="\[.*]\.username"/>
        </list>
        <parameter key="index_attribute" value="Index"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="7.5.001" expanded="true" height="103" name="Filter Examples" width="90" x="447" y="34">
        <list key="filters_list">
          <parameter key="filters_entry_key" value="platformName.equals.facebook"/>
        </list>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="7.5.001" expanded="true" height="103" name="Filter Examples (2)" width="90" x="447" y="340">
        <list key="filters_list">
          <parameter key="filters_entry_key" value="platformName.equals.twitter"/>
        </list>
      </operator>
      <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="124" name="Facebook" width="90" x="715" y="136">
        <parameter key="exception_macro" value="exception_facebook"/>
        <process expanded="true">
          <operator activated="true" class="multiply" compatibility="7.5.001" expanded="true" height="103" name="Multiply" width="90" x="45" y="34"/>
          <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="82" name="User Details" width="90" x="313" y="289">
            <process expanded="true">
              <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice (5)" width="90" x="246" y="85">
                <parameter key="query_type" value="Regular Expression"/>
                <list key="string_machting_queries"/>
                <list key="regular_expression_queries">
                  <parameter key="jsonResponse" value=".*"/>
                </list>
                <list key="regular_region_queries"/>
                <list key="xpath_queries"/>
                <list key="namespaces"/>
                <list key="index_queries"/>
                <list key="jsonpath_queries">
                  <parameter key="message" value="$..message"/>
                  <parameter key="post id" value="$..id"/>
                </list>
                <parameter key="url" value="https://graph.facebook.com/v2.10/&amp;lt;%username%&amp;gt;?access_token=&amp;lt;your_access_token&amp;gt;&amp;amp;fields=username,name,about,fan_count,location,start_info,overall_star_rating,rating_count,talking_about_count"/>
                <list key="request_properties"/>
                <parameter key="encoding" value="UTF-8"/>
              </operator>
              <operator activated="true" class="loop_examples" compatibility="7.5.001" expanded="true" height="103" name="Loop Examples (3)" width="90" x="380" y="85">
                <process expanded="true">
                  <operator activated="true" class="filter_example_range" compatibility="7.5.001" expanded="true" height="82" name="Filter Example Range (3)" width="90" x="45" y="34">
                    <parameter key="first_example" value="%{example}"/>
                    <parameter key="last_example" value="%{example}"/>
                  </operator>
                  <operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents (3)" width="90" x="179" y="34">
                    <parameter key="select_attributes_and_weights" value="true"/>
                    <list key="specify_weights">
                      <parameter key="jsonResponse" value="1.0"/>
                    </list>
                  </operator>
                  <operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents (3)" width="90" x="313" y="34"/>
                  <operator activated="true" class="web:json_to_xml" compatibility="7.3.000" expanded="true" height="68" name="JSON to XML (3)" width="90" x="447" y="34"/>
                  <operator activated="true" class="text:write_document" compatibility="7.5.000" expanded="true" height="82" name="Write Document (3)" width="90" x="581" y="34">
                    <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\userDetails2.xml"/>
                    <parameter key="encoding" value="UTF-16"/>
                  </operator>
                  <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Read XML (3)" width="90" x="715" y="34">
                    <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\userDetails2.xml"/>
                    <parameter key="xpath_for_examples" value="//json"/>
                    <enumeration key="xpaths_for_attributes">
                      <parameter key="xpath_for_attribute" value="username[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="name[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="about[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="fan_count[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/street[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/city[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/state[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/country[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/zip[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/latitude[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="location[1]/longitude[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="start_info[1]/date[1]/year[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="start_info[1]/type[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="rating_count[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="overall_star_rating[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="talking_about_count[1]/text()"/>
                    </enumeration>
                    <list key="namespaces"/>
                    <parameter key="use_default_namespace" value="false"/>
                    <list key="annotations"/>
                    <list key="data_set_meta_data_information">
                      <parameter key="0" value="username[1]/text().true.attribute_value.attribute"/>
                      <parameter key="1" value="name[1]/text().true.attribute_value.attribute"/>
                      <parameter key="2" value="about[1]/text().true.attribute_value.attribute"/>
                      <parameter key="3" value="fan_count[1]/text().true.attribute_value.attribute"/>
                      <parameter key="4" value="location[1]/street[1]/text().true.attribute_value.attribute"/>
                      <parameter key="5" value="location[1]/city[1]/text().true.attribute_value.attribute"/>
                      <parameter key="6" value="location[1]/state[1]/text().true.attribute_value.attribute"/>
                      <parameter key="7" value="location[1]/country[1]/text().true.attribute_value.attribute"/>
                      <parameter key="8" value="location[1]/zip[1]/text().true.attribute_value.attribute"/>
                      <parameter key="9" value="location[1]/latitude[1]/text().true.attribute_value.attribute"/>
                      <parameter key="10" value="location[1]/longitude[1]/text().true.attribute_value.attribute"/>
                      <parameter key="11" value="start_info[1]/date[1]/year[1]/text().true.attribute_value.attribute"/>
                      <parameter key="12" value="start_info[1]/type[1]/text().true.attribute_value.attribute"/>
                      <parameter key="13" value="rating_count[1]/text().true.attribute_value.attribute"/>
                      <parameter key="14" value="overall_star_rating[1]/text().true.attribute_value.attribute"/>
                      <parameter key="15" value="talking_about_count[1]/text().true.attribute_value.attribute"/>
                    </list>
                  </operator>
                  <connect from_port="example set" to_op="Filter Example Range (3)" to_port="example set input"/>
                  <connect from_op="Filter Example Range (3)" from_port="example set output" to_op="Data to Documents (3)" to_port="example set"/>
                  <connect from_op="Data to Documents (3)" from_port="documents" to_op="Combine Documents (3)" to_port="documents 1"/>
                  <connect from_op="Combine Documents (3)" from_port="document" to_op="JSON to XML (3)" to_port="document"/>
                  <connect from_op="JSON to XML (3)" from_port="document" to_op="Write Document (3)" to_port="document"/>
                  <connect from_op="Write Document (3)" from_port="file" to_op="Read XML (3)" to_port="file"/>
                  <connect from_op="Read XML (3)" from_port="output" to_port="output 1"/>
                  <portSpacing port="source_example set" spacing="0"/>
                  <portSpacing port="sink_example set" spacing="0"/>
                  <portSpacing port="sink_output 1" spacing="0"/>
                  <portSpacing port="sink_output 2" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="82" name="Union Append (3)" width="90" x="514" y="85">
                <process expanded="true">
                  <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (3)" width="90" x="45" y="34">
                    <parameter key="set_iteration_macro" value="true"/>
                    <process expanded="true">
                      <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (4)" width="90" x="112" y="34">
                        <parameter key="index" value="%{iteration}"/>
                      </operator>
                      <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (4)" width="90" x="313" y="34">
                        <parameter key="condition_type" value="expression"/>
                        <parameter key="expression" value="%{iteration}==1"/>
                        <process expanded="true">
                          <connect from_port="condition" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                        <process expanded="true">
                          <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (3)" width="90" x="45" y="187">
                            <parameter key="name" value="LoopData"/>
                          </operator>
                          <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (User Details)" width="90" x="179" y="34"/>
                          <connect from_port="condition" to_op="Union (User Details)" to_port="example set 1"/>
                          <connect from_op="Recall (3)" from_port="result" to_op="Union (User Details)" to_port="example set 2"/>
                          <connect from_op="Union (User Details)" from_port="union" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                      </operator>
                      <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (3)" width="90" x="581" y="34">
                        <parameter key="name" value="LoopData"/>
                      </operator>
                      <connect from_port="single" to_op="Branch (4)" to_port="condition"/>
                      <connect from_op="Branch (4)" from_port="input 1" to_op="Remember (3)" to_port="store"/>
                      <connect from_op="Remember (3)" from_port="stored" to_port="output 1"/>
                      <portSpacing port="source_single" spacing="0"/>
                      <portSpacing port="sink_output 1" spacing="0"/>
                      <portSpacing port="sink_output 2" spacing="0"/>
                    </process>
                  </operator>
                  <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (7)" width="90" x="179" y="34">
                    <parameter key="index" value="%{iteration}"/>
                  </operator>
                  <connect from_port="in 1" to_op="Output (3)" to_port="collection"/>
                  <connect from_op="Output (3)" from_port="output 1" to_op="Select (7)" to_port="collection"/>
                  <connect from_op="Select (7)" from_port="selected" to_port="out 1"/>
                  <portSpacing port="source_in 1" spacing="0"/>
                  <portSpacing port="source_in 2" spacing="0"/>
                  <portSpacing port="sink_out 1" spacing="0"/>
                  <portSpacing port="sink_out 2" spacing="0"/>
                </process>
              </operator>
              <connect from_port="in 1" to_op="Enrich Data by Webservice (5)" to_port="Example Set"/>
              <connect from_op="Enrich Data by Webservice (5)" from_port="ExampleSet" to_op="Loop Examples (3)" to_port="example set"/>
              <connect from_op="Loop Examples (3)" from_port="output 1" to_op="Union Append (3)" to_port="in 1"/>
              <connect from_op="Union Append (3)" from_port="out 1" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Sentiment and Avg. Share" width="90" x="246" y="34">
            <process expanded="true">
              <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice (2)" width="90" x="112" y="34">
                <parameter key="query_type" value="Regular Expression"/>
                <list key="string_machting_queries"/>
                <list key="regular_expression_queries">
                  <parameter key="jsonResponse" value=".*"/>
                </list>
                <list key="regular_region_queries"/>
                <list key="xpath_queries"/>
                <list key="namespaces"/>
                <list key="index_queries"/>
                <list key="jsonpath_queries">
                  <parameter key="message" value="$..message"/>
                  <parameter key="post id" value="$..id"/>
                </list>
                <parameter key="url" value="https://graph.facebook.com/v2.10/&amp;lt;%username%&amp;gt;/posts?access_token=&amp;lt;your_access_token&amp;gt;&amp;amp;fields=message,link,created_time,type,name,id,comments.limit(10),shares&amp;amp;limit=100"/>
                <list key="request_properties"/>
                <parameter key="encoding" value="UTF-8"/>
              </operator>
              <operator activated="true" class="loop_examples" compatibility="7.5.001" expanded="true" height="124" name="Loop Examples" width="90" x="246" y="34">
                <process expanded="true">
                  <operator activated="true" class="filter_example_range" compatibility="7.5.001" expanded="true" height="82" name="Filter Example Range" width="90" x="45" y="34">
                    <parameter key="first_example" value="%{example}"/>
                    <parameter key="last_example" value="%{example}"/>
                  </operator>
                  <operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents" width="90" x="179" y="34">
                    <parameter key="select_attributes_and_weights" value="true"/>
                    <list key="specify_weights">
                      <parameter key="jsonResponse" value="1.0"/>
                    </list>
                  </operator>
                  <operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents" width="90" x="313" y="34"/>
                  <operator activated="true" class="web:json_to_xml" compatibility="7.3.000" expanded="true" height="68" name="JSON to XML" width="90" x="447" y="34"/>
                  <operator activated="true" class="text:write_document" compatibility="7.5.000" expanded="true" height="82" name="Write Document" width="90" x="581" y="34">
                    <parameter key="file" value="/Users/GenzerConsulting/OneDrive - RapidMiner/OneDrive Repository/random community stuff/jsonExport.xml"/>
                    <parameter key="encoding" value="UTF-16"/>
                  </operator>
                  <operator activated="true" class="multiply" compatibility="7.5.001" expanded="true" height="103" name="Multiply (2)" width="90" x="715" y="136"/>
                  <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Comments (2)" width="90" x="849" y="289">
                    <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\jsonDump.xml"/>
                    <parameter key="xpath_for_examples" value="//json/data/comments"/>
                    <enumeration key="xpaths_for_attributes">
                      <parameter key="xpath_for_attribute" value="data[1]/created_time[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="data[1]/from[1]/name[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="data[1]/from[1]/id[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="data[1]/id[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="data[1]/message[1]/text()"/>
                    </enumeration>
                    <list key="namespaces"/>
                    <parameter key="use_default_namespace" value="false"/>
                    <list key="annotations"/>
                    <list key="data_set_meta_data_information">
                      <parameter key="0" value="data[1]/created_time[1]/text().true.attribute_value.attribute"/>
                      <parameter key="1" value="data[1]/from[1]/name[1]/text().true.attribute_value.attribute"/>
                      <parameter key="2" value="data[1]/from[1]/id[1]/text().true.attribute_value.attribute"/>
                      <parameter key="3" value="data[1]/id[1]/text().true.attribute_value.attribute"/>
                      <parameter key="4" value="data[1]/message[1]/text().true.attribute_value.attribute"/>
                    </list>
                  </operator>
                  <operator activated="true" class="advanced_file_connectors:read_xml" compatibility="7.5.001" expanded="true" height="68" name="Posts (2)" width="90" x="849" y="34">
                    <parameter key="file" value="C:\Users\Pari\Documents\BDC\Socials\Facebook Scrapper\Test\jsonDump.xml"/>
                    <parameter key="xpath_for_examples" value="//json/data"/>
                    <enumeration key="xpaths_for_attributes">
                      <parameter key="xpath_for_attribute" value="shares[1]/count[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="created_time[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="link[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="id[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="message[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="type[1]/text()"/>
                      <parameter key="xpath_for_attribute" value="name[1]/text()"/>
                    </enumeration>
                    <list key="namespaces"/>
                    <parameter key="use_default_namespace" value="false"/>
                    <list key="annotations"/>
                    <list key="data_set_meta_data_information">
                      <parameter key="0" value="shares[1]/count[1]/text().true.numeric.attribute"/>
                      <parameter key="1" value="created_time[1]/text().true.attribute_value.attribute"/>
                      <parameter key="2" value="link[1]/text().true.attribute_value.attribute"/>
                      <parameter key="3" value="id[1]/text().true.attribute_value.attribute"/>
                      <parameter key="4" value="message[1]/text().true.attribute_value.attribute"/>
                      <parameter key="5" value="type[1]/text().true.attribute_value.attribute"/>
                      <parameter key="6" value="name[1]/text().true.attribute_value.attribute"/>
                    </list>
                  </operator>
                  <connect from_port="example set" to_op="Filter Example Range" to_port="example set input"/>
                  <connect from_op="Filter Example Range" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
                  <connect from_op="Data to Documents" from_port="documents" to_op="Combine Documents" to_port="documents 1"/>
                  <connect from_op="Combine Documents" from_port="document" to_op="JSON to XML" to_port="document"/>
                  <connect from_op="JSON to XML" from_port="document" to_op="Write Document" to_port="document"/>
                  <connect from_op="Write Document" from_port="file" to_op="Multiply (2)" to_port="input"/>
                  <connect from_op="Multiply (2)" from_port="output 1" to_op="Posts (2)" to_port="file"/>
                  <connect from_op="Multiply (2)" from_port="output 2" to_op="Comments (2)" to_port="file"/>
                  <connect from_op="Comments (2)" from_port="output" to_port="output 2"/>
                  <connect from_op="Posts (2)" from_port="output" to_port="output 1"/>
                  <portSpacing port="source_example set" spacing="0"/>
                  <portSpacing port="sink_example set" spacing="0"/>
                  <portSpacing port="sink_output 1" spacing="0"/>
                  <portSpacing port="sink_output 2" spacing="0"/>
                  <portSpacing port="sink_output 3" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Union Append" width="90" x="380" y="34">
                <process expanded="true">
                  <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (4)" width="90" x="179" y="34">
                    <parameter key="set_iteration_macro" value="true"/>
                    <process expanded="true">
                      <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (5)" width="90" x="112" y="136">
                        <parameter key="index" value="%{iteration}"/>
                      </operator>
                      <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (2)" width="90" x="313" y="34">
                        <parameter key="condition_type" value="expression"/>
                        <parameter key="expression" value="%{iteration}==1"/>
                        <process expanded="true">
                          <connect from_port="condition" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                        <process expanded="true">
                          <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (5)" width="90" x="45" y="187">
                            <parameter key="name" value="LoopData"/>
                          </operator>
                          <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (Posts)" width="90" x="179" y="34"/>
                          <connect from_port="condition" to_op="Union (Posts)" to_port="example set 1"/>
                          <connect from_op="Recall (5)" from_port="result" to_op="Union (Posts)" to_port="example set 2"/>
                          <connect from_op="Union (Posts)" from_port="union" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                      </operator>
                      <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (5)" width="90" x="581" y="34">
                        <parameter key="name" value="LoopData"/>
                      </operator>
                      <connect from_port="single" to_op="Branch (2)" to_port="condition"/>
                      <connect from_op="Branch (2)" from_port="input 1" to_op="Remember (5)" to_port="store"/>
                      <connect from_op="Remember (5)" from_port="stored" to_port="output 1"/>
                      <portSpacing port="source_single" spacing="0"/>
                      <portSpacing port="sink_output 1" spacing="0"/>
                      <portSpacing port="sink_output 2" spacing="0"/>
                    </process>
                  </operator>
                  <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Posts" width="90" x="447" y="34">
                    <parameter key="index" value="%{iteration}"/>
                  </operator>
                  <operator activated="true" class="loop_collection" compatibility="7.5.001" expanded="true" height="82" name="Output (2)" width="90" x="179" y="187">
                    <parameter key="set_iteration_macro" value="true"/>
                    <process expanded="true">
                      <operator activated="false" breakpoints="after" class="select" compatibility="7.5.001" expanded="true" height="68" name="Select (2)" width="90" x="112" y="85">
                        <parameter key="index" value="%{iteration}"/>
                      </operator>
                      <operator activated="true" class="branch" compatibility="7.5.001" expanded="true" height="82" name="Branch (3)" width="90" x="313" y="34">
                        <parameter key="condition_type" value="expression"/>
                        <parameter key="expression" value="%{iteration}==1"/>
                        <process expanded="true">
                          <connect from_port="condition" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                        <process expanded="true">
                          <operator activated="true" class="recall" compatibility="7.5.001" expanded="true" height="68" name="Recall (2)" width="90" x="45" y="187">
                            <parameter key="name" value="LoopData"/>
                          </operator>
                          <operator activated="true" class="union" compatibility="7.5.001" expanded="true" height="82" name="Union (Comments)" width="90" x="179" y="34"/>
                          <connect from_port="condition" to_op="Union (Comments)" to_port="example set 1"/>
                          <connect from_op="Recall (2)" from_port="result" to_op="Union (Comments)" to_port="example set 2"/>
                          <connect from_op="Union (Comments)" from_port="union" to_port="input 1"/>
                          <portSpacing port="source_condition" spacing="0"/>
                          <portSpacing port="source_input 1" spacing="0"/>
                          <portSpacing port="sink_input 1" spacing="0"/>
                          <portSpacing port="sink_input 2" spacing="0"/>
                        </process>
                      </operator>
                      <operator activated="true" class="remember" compatibility="7.5.001" expanded="true" height="68" name="Remember (2)" width="90" x="581" y="34">
                        <parameter key="name" value="LoopData"/>
                      </operator>
                      <connect from_port="single" to_op="Branch (3)" to_port="condition"/>
                      <connect from_op="Branch (3)" from_port="input 1" to_op="Remember (2)" to_port="store"/>
                      <connect from_op="Remember (2)" from_port="stored" to_port="output 1"/>
                      <portSpacing port="source_single" spacing="0"/>
                      <portSpacing port="sink_output 1" spacing="0"/>
                      <portSpacing port="sink_output 2" spacing="0"/>
                    </process>
                  </operator>
                  <operator activated="true" class="select" compatibility="7.5.001" expanded="true" height="68" name="Comments" width="90" x="447" y="187">
                    <parameter key="index" value="%{iteration}"/>
                  </operator>
                  <connect from_port="in 1" to_op="Output (4)" to_port="collection"/>
                  <connect from_port="in 2" to_op="Output (2)" to_port="collection"/>
                  <connect from_op="Output (4)" from_port="output 1" to_op="Posts" to_port="collection"/>
                  <connect from_op="Posts" from_port="selected" to_port="out 1"/>
                  <connect from_op="Output (2)" from_port="output 1" to_op="Comments" to_port="collection"/>
                  <connect from_op="Comments" from_port="selected" to_port="out 2"/>
                  <portSpacing port="source_in 1" spacing="0"/>
                  <portSpacing port="source_in 2" spacing="0"/>
                  <portSpacing port="source_in 3" spacing="0"/>
                  <portSpacing port="sink_out 1" spacing="0"/>
                  <portSpacing port="sink_out 2" spacing="0"/>
                  <portSpacing port="sink_out 3" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Avg Share Rate" width="90" x="849" y="34">
                <list key="aggregation_attributes">
                  <parameter key="shares[1]/count[1]/text()" value="average"/>
                </list>
              </operator>
              <operator activated="true" class="format_numbers" compatibility="7.5.001" expanded="true" height="82" name="Format Numbers (2)" width="90" x="983" y="34">
                <parameter key="format_type" value="pattern"/>
                <parameter key="pattern" value="0.00"/>
              </operator>
              <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="82" name="Handle Exception (2)" width="90" x="514" y="187">
                <parameter key="exception_macro" value="sampling_exception"/>
                <process expanded="true">
                  <operator activated="true" class="sample" compatibility="7.5.001" expanded="true" height="82" name="Sample" width="90" x="179" y="85">
                    <parameter key="sample_size" value="25"/>
                    <list key="sample_size_per_class"/>
                    <list key="sample_ratio_per_class"/>
                    <list key="sample_probability_per_class"/>
                  </operator>
                  <connect from_port="in 1" to_op="Sample" to_port="example set input"/>
                  <connect from_op="Sample" from_port="example set output" to_port="out 1"/>
                  <portSpacing port="source_in 1" spacing="0"/>
                  <portSpacing port="source_in 2" spacing="0"/>
                  <portSpacing port="sink_out 1" spacing="0"/>
                  <portSpacing port="sink_out 2" spacing="0"/>
                </process>
                <process expanded="true">
                  <connect from_port="in 1" to_port="out 1"/>
                  <portSpacing port="source_in 1" spacing="0"/>
                  <portSpacing port="source_in 2" spacing="0"/>
                  <portSpacing port="sink_out 1" spacing="0"/>
                  <portSpacing port="sink_out 2" spacing="0"/>
                </process>
              </operator>
              <operator activated="true" class="com.aylien.textapi.rapidminer:aylien_sentiment" compatibility="0.2.000" expanded="true" height="68" name="Analyze Sentiment (2)" width="90" x="715" y="187">
                <parameter key="connection" value="AylienRel8ed"/>
                <parameter key="input_attribute" value="data[1]/message[1]/text()"/>
              </operator>
              <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Aggregate (facebook sentiment)" width="90" x="849" y="187">
                <list key="aggregation_attributes">
                  <parameter key="polarity" value="count (percentage)"/>
                </list>
                <parameter key="group_by_attributes" value="polarity"/>
              </operator>
              <operator activated="true" class="format_numbers" compatibility="7.5.001" expanded="true" height="82" name="Format Numbers (4)" width="90" x="983" y="187">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="percentage_count(polarity)"/>
                <parameter key="format_type" value="pattern"/>
                <parameter key="pattern" value="0.00"/>
              </operator>
              <connect from_port="in 1" to_op="Enrich Data by Webservice (2)" to_port="Example Set"/>
              <connect from_op="Enrich Data by Webservice (2)" from_port="ExampleSet" to_op="Loop Examples" to_port="example set"/>
              <connect from_op="Loop Examples" from_port="output 1" to_op="Union Append" to_port="in 1"/>
              <connect from_op="Loop Examples" from_port="output 2" to_op="Union Append" to_port="in 2"/>
              <connect from_op="Union Append" from_port="out 1" to_op="Avg Share Rate" to_port="example set input"/>
              <connect from_op="Union Append" from_port="out 2" to_op="Handle Exception (2)" to_port="in 1"/>
              <connect from_op="Avg Share Rate" from_port="example set output" to_op="Format Numbers (2)" to_port="example set input"/>
              <connect from_op="Format Numbers (2)" from_port="example set output" to_port="out 1"/>
              <connect from_op="Handle Exception (2)" from_port="out 1" to_op="Analyze Sentiment (2)" to_port="Example Set"/>
              <connect from_op="Analyze Sentiment (2)" from_port="Example Set" to_op="Aggregate (facebook sentiment)" to_port="example set input"/>
              <connect from_op="Aggregate (facebook sentiment)" from_port="example set output" to_op="Format Numbers (4)" to_port="example set input"/>
              <connect from_op="Format Numbers (4)" from_port="example set output" to_port="out 2"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
              <portSpacing port="sink_out 3" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Post Processing" width="90" x="447" y="136">
            <process expanded="true">
              <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename User Details (fb)" width="90" x="179" y="136">
                <parameter key="old_name" value="about[1]/text()"/>
                <parameter key="new_name" value="description"/>
                <list key="rename_additional_attributes">
                  <parameter key="fan_count[1]/text()" value="fans"/>
                  <parameter key="location[1]/city[1]/text()" value="city"/>
                  <parameter key="location[1]/country[1]/text()" value="country"/>
                  <parameter key="location[1]/latitude[1]/text()" value="latitude"/>
                  <parameter key="location[1]/longitude[1]/text()" value="longitude"/>
                  <parameter key="location[1]/state[1]/text()" value="state"/>
                  <parameter key="location[1]/street[1]/text()" value="street"/>
                  <parameter key="location[1]/zip[1]/text()" value="zip"/>
                  <parameter key="name[1]/text()" value="name"/>
                  <parameter key="overall_star_rating[1]/text()" value="facebook star rating"/>
                  <parameter key="rating_count[1]/text()" value="number of ratings"/>
                  <parameter key="start_info[1]/date[1]/year[1]/text()" value="start date"/>
                  <parameter key="start_info[1]/type[1]/text()" value="start date type"/>
                  <parameter key="talking_about_count[1]/text()" value="number of people talking about this page"/>
                  <parameter key="username[1]/text()" value="username"/>
                </list>
              </operator>
              <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="Generate platformName (2)" width="90" x="380" y="136">
                <list key="function_descriptions">
                  <parameter key="platformName" value="&quot;facebook&quot;"/>
                </list>
              </operator>
              <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename" width="90" x="179" y="34">
                <parameter key="old_name" value="average(shares[1]/count[1]/text())"/>
                <parameter key="new_name" value="average share rate"/>
                <list key="rename_additional_attributes"/>
              </operator>
              <operator activated="true" class="order_attributes" compatibility="7.5.001" expanded="true" height="82" name="Reorder Facebook" width="90" x="648" y="136">
                <parameter key="attribute_ordering" value="platformName|username|name|description|street|city|state|country|zip|latitude|longitude|facebook star rating|fans|number of people talking about this page|number of ratings|start date|start date type"/>
              </operator>
              <connect from_port="in 1" to_op="Rename" to_port="example set input"/>
              <connect from_port="in 2" to_op="Rename User Details (fb)" to_port="example set input"/>
              <connect from_op="Rename User Details (fb)" from_port="example set output" to_op="Generate platformName (2)" to_port="example set input"/>
              <connect from_op="Generate platformName (2)" from_port="example set output" to_op="Reorder Facebook" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_port="out 2"/>
              <connect from_op="Reorder Facebook" from_port="example set output" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="source_in 3" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
              <portSpacing port="sink_out 3" spacing="0"/>
            </process>
          </operator>
          <connect from_port="in 1" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Sentiment and Avg. Share" to_port="in 1"/>
          <connect from_op="Multiply" from_port="output 2" to_op="User Details" to_port="in 1"/>
          <connect from_op="User Details" from_port="out 1" to_op="Post Processing" to_port="in 2"/>
          <connect from_op="Sentiment and Avg. Share" from_port="out 1" to_op="Post Processing" to_port="in 1"/>
          <connect from_op="Sentiment and Avg. Share" from_port="out 2" to_port="out 3"/>
          <connect from_op="Post Processing" from_port="out 1" to_port="out 1"/>
          <connect from_op="Post Processing" from_port="out 2" to_port="out 2"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="log" compatibility="7.5.001" expanded="true" height="82" name="Log" width="90" x="179" y="34">
            <list key="log"/>
          </operator>
          <connect from_port="in 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="82" name="Handle Exception" width="90" x="581" y="340">
        <parameter key="exception_macro" value="exception_process"/>
        <process expanded="true">
          <operator activated="true" class="extract_macro" compatibility="7.5.001" expanded="true" height="68" name="Extract Macro (2)" width="90" x="179" y="34">
            <parameter key="macro" value="twitter"/>
            <parameter key="macro_type" value="data_value"/>
            <parameter key="attribute_name" value="username"/>
            <parameter key="example_index" value="1"/>
            <list key="additional_macros"/>
          </operator>
          <connect from_port="in 1" to_op="Extract Macro (2)" to_port="example set"/>
          <connect from_op="Extract Macro (2)" from_port="example set" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
        <process expanded="true">
          <connect from_port="in 1" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="handle_exception" compatibility="7.5.001" expanded="true" height="124" name="Twitter" width="90" x="715" y="340">
        <parameter key="exception_macro" value="exception_twitter"/>
        <process expanded="true">
          <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="238">
            <parameter key="connection" value="Twitter Server"/>
            <parameter key="query" value="@%{twitter}"/&gt;
            <parameter key="limit" value="25"/>
          </operator>
          <operator activated="true" class="com.aylien.textapi.rapidminer:aylien_sentiment" compatibility="0.2.000" expanded="true" height="68" name="Analyze Sentiment" width="90" x="313" y="238">
            <parameter key="connection" value="AylienRel8ed"/>
            <parameter key="input_attribute" value="Text"/>
          </operator>
          <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Aggregate(twitter sentiment)" width="90" x="447" y="238">
            <list key="aggregation_attributes">
              <parameter key="polarity" value="count (percentage)"/>
            </list>
            <parameter key="group_by_attributes" value="polarity"/>
          </operator>
          <operator activated="true" class="social_media:get_twitter_user_statuses" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Statuses" width="90" x="112" y="136">
            <parameter key="connection" value="Twitter Server"/>
            <parameter key="user" value="%{twitter}"/>
            <parameter key="limit" value="40"/>
          </operator>
          <operator activated="true" class="aggregate" compatibility="7.5.001" expanded="true" height="82" name="Avg RT" width="90" x="313" y="136">
            <list key="aggregation_attributes">
              <parameter key="Retweet-Count" value="average"/>
            </list>
          </operator>
          <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="112" y="34">
            <parameter key="connection" value="Twitter Server"/>
            <parameter key="user" value="%{twitter}"/>
          </operator>
          <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="calculate account age" width="90" x="313" y="34">
            <list key="function_descriptions">
              <parameter key="account age (in days)" value="(date_diff([Created-At], date_now())*1.15741e-8)"/>
            </list>
          </operator>
          <operator activated="true" class="subprocess" compatibility="7.5.001" expanded="true" height="103" name="Post Processing (2)" width="90" x="447" y="34">
            <process expanded="true">
              <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename User Details (twit)" width="90" x="246" y="34">
                <parameter key="old_name" value="Created-At"/>
                <parameter key="new_name" value="Start Date"/>
                <list key="rename_additional_attributes">
                  <parameter key="Description" value="description"/>
                  <parameter key="Favourites" value="favourites"/>
                  <parameter key="Followers" value="followers"/>
                  <parameter key="Friends" value="friends"/>
                  <parameter key="Id" value="id"/>
                  <parameter key="Language" value="language"/>
                  <parameter key="Location" value="location"/>
                  <parameter key="Name" value="name"/>
                  <parameter key="Tweets" value="tweets"/>
                  <parameter key="URL" value="url"/>
                </list>
              </operator>
              <operator activated="true" class="select_attributes" compatibility="7.5.001" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="34">
                <parameter key="attribute_filter_type" value="subset"/>
                <parameter key="attributes" value="url|tweets|name|location|language|id|friends|followers|favourites|description|account age (in days)"/>
              </operator>
              <operator activated="true" class="generate_attributes" compatibility="7.5.001" expanded="true" height="82" name="Generate platformName" width="90" x="581" y="34">
                <list key="function_descriptions">
                  <parameter key="platformName" value="&quot;twitter&quot;"/>
                </list>
              </operator>
              <operator activated="true" class="rename" compatibility="7.5.001" expanded="true" height="82" name="Rename (4)" width="90" x="447" y="136">
                <parameter key="old_name" value="average(Retweet-Count)"/>
                <parameter key="new_name" value="average retweet count"/>
                <list key="rename_additional_attributes"/>
              </operator>
              <operator activated="true" class="order_attributes" compatibility="7.5.001" expanded="true" height="82" name="Reorder Twitter" width="90" x="715" y="34">
                <parameter key="attribute_ordering" value="platformName|id|name|description|tweets|favourites|followers|friends|language|location|url|account age (in days)"/>
              </operator>
              <operator activated="true" class="format_numbers" compatibility="7.5.001" expanded="true" height="82" name="Format Numbers (3)" width="90" x="648" y="136">
                <parameter key="format_type" value="pattern"/>
                <parameter key="pattern" value="0.00"/>
              </operator>
              <connect from_port="in 1" to_op="Rename User Details (twit)" to_port="example set input"/>
              <connect from_port="in 2" to_op="Rename (4)" to_port="example set input"/>
              <connect from_op="Rename User Details (twit)" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
              <connect from_op="Select Attributes" from_port="example set output" to_op="Generate platformName" to_port="example set input"/>
              <connect from_op="Generate platformName" from_port="example set output" to_op="Reorder Twitter" to_port="example set input"/>
              <connect from_op="Rename (4)" from_port="example set output" to_op="Format Numbers (3)" to_port="example set input"/>
              <connect from_op="Reorder Twitter" from_port="example set output" to_port="out 1"/>
              <connect from_op="Format Numbers (3)" from_port="example set output" to_port="out 2"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="source_in 3" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
              <portSpacing port="sink_out 3" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="format_numbers" compatibility="7.5.001" expanded="true" height="82" name="Format Numbers (5)" width="90" x="581" y="238">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="percentage_count(polarity)"/>
            <parameter key="format_type" value="pattern"/>
            <parameter key="pattern" value="0.00"/>
          </operator>
          <connect from_op="Search Twitter" from_port="output" to_op="Analyze Sentiment" to_port="Example Set"/>
          <connect from_op="Analyze Sentiment" from_port="Example Set" to_op="Aggregate(twitter sentiment)" to_port="example set input"/>
          <connect from_op="Aggregate(twitter sentiment)" from_port="example set output" to_op="Format Numbers (5)" to_port="example set input"/>
          <connect from_op="Get Twitter User Statuses" from_port="output" to_op="Avg RT" to_port="example set input"/>
          <connect from_op="Avg RT" from_port="example set output" to_op="Post Processing (2)" to_port="in 2"/>
          <connect from_op="Get Twitter User Details" from_port="output" to_op="calculate account age" to_port="example set input"/>
          <connect from_op="calculate account age" from_port="example set output" to_op="Post Processing (2)" to_port="in 1"/>
          <connect from_op="Post Processing (2)" from_port="out 1" to_port="out 1"/>
          <connect from_op="Post Processing (2)" from_port="out 2" to_port="out 2"/>
          <connect from_op="Format Numbers (5)" from_port="example set output" to_port="out 3"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="log" compatibility="7.5.001" expanded="true" height="82" name="Log (2)" width="90" x="179" y="34">
            <list key="log"/>
          </operator>
          <connect from_port="in 1" to_op="Log (2)" to_port="through 1"/>
          <connect from_op="Log (2)" from_port="through 1" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
      </operator>
      <connect from_port="input 1" to_op="Read Document" to_port="file"/>
      <connect from_op="Generate Data (OverallRatingComingSoon)" from_port="output" to_op="Format Numbers" to_port="example set input"/>
      <connect from_op="Format Numbers" from_port="example set output" to_port="result 1"/>
      <connect from_op="Read Document" from_port="output" to_op="JSON To Data" to_port="documents 1"/>
      <connect from_op="JSON To Data" from_port="example set" to_op="De-Pivot (2)" to_port="example set input"/>
      <connect from_op="De-Pivot (2)" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_op="Facebook" to_port="in 1"/>
      <connect from_op="Filter Examples" from_port="original" to_op="Filter Examples (2)" to_port="example set input"/>
      <connect from_op="Filter Examples (2)" from_port="example set output" to_op="Handle Exception" to_port="in 1"/>
      <connect from_op="Facebook" from_port="out 1" to_port="result 2"/>
      <connect from_op="Facebook" from_port="out 2" to_port="result 3"/>
      <connect from_op="Facebook" from_port="out 3" to_port="result 4"/>
      <connect from_op="Handle Exception" from_port="out 1" to_op="Twitter" to_port="in 1"/>
      <connect from_op="Twitter" from_port="out 1" to_port="result 5"/>
      <connect from_op="Twitter" from_port="out 2" to_port="result 6"/>
      <connect from_op="Twitter" from_port="out 3" to_port="result 7"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="source_input 2" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
      <portSpacing port="sink_result 6" spacing="0"/>
      <portSpacing port="sink_result 7" spacing="0"/>
      <portSpacing port="sink_result 8" spacing="0"/>
    </process>
  </operator>
</process

 

sgenzer

oh that's great.  Thank you!  Tell you what - if you can write this up for a KB article, we can publish it here and I may be able to get you a cool RapidMiner t-shirt as part of the deal.  

Scott