How to get many user Twitter datas in a short time?

@YYTheFenix There's a sample process on the Community on how to do that with a Loop that I think @Edin_Klapic posted. It uses the Get Twitter User Details operator. <strike>Search for that.</strike>

Here it is: https://community.rapidminer.com/t5/Product-Feedback/HELP-obtaining-Twitter-user-details-I-m-new-to-RapidMiner/idc-p/38750#M90

@Edin_Klapic is the bug fixed?

Hi @YYTheFenix,

There is no built in functionality to get the data for all users in one request.

This has to be achieved by searching for every User detail separately e.g. using the Operator Loop Values.

Here you are looping over every username and search for the user details using Macros (more on Macros can be found here).

The attached example Process also checks for the number of friends of a Twitter user using the Operator Get Twitter Relations. Here you can set the parameter relation to friends. The following Branch Operator uses the condition type min_examples to only search for the User details if the user has at least 199 friends.

Important notes:

1. Be aware that there is a limit on the number of free Twitter calls
2. Usually I recommend to utilize the User ID for searching but currently there is a bug when using this parameter so the Twitter user name is used. This is the one with @ and may differ from the display name which you get from the Operator Search Twitter.

Best regards,

Edin

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
        <parameter key="query" value="rapidminer"/>
        <parameter key="limit" value="1"/>
        <description align="center" color="transparent" colored="false" width="126">get users</description>
      </operator>
      <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="246" y="34">
        <parameter key="attribute" value="From-User"/>
        <parameter key="reuse_results" value="true"/>
        <process expanded="true">
          <operator activated="true" class="social_media:get_twitter_relations" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter Relations" width="90" x="112" y="34">
            <parameter key="user" value="%{loop_value}"/>
            <parameter key="relation" value="friends"/>
          </operator>
          <operator activated="true" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch" width="90" x="246" y="34">
            <parameter key="condition_type" value="min_examples"/>
            <parameter key="condition_value" value="199"/>
            <process expanded="true">
              <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="246" y="34">
                <parameter key="user" value="%{loop_value}"/>
              </operator>
              <connect from_op="Get Twitter User Details" from_port="output" to_port="input 1"/>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Get Twitter Relations" from_port="output" to_op="Branch" to_port="condition"/>
          <connect from_op="Branch" from_port="input 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Search Twitter" from_port="output" to_op="Loop Values" to_port="input 1"/>
      <connect from_op="Loop Values" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

@Edin_Klapic

thank you for nice xml code. when i try it and i found that this code can extract user's details from only one user per time (because i try to change limit more than 1 in search twitter box but it doesn't work , So i change it back to 1). And after that,  i try to change condition in Branch to max_examples 200 . Could you tell me why don't have output result?

PS. my target is to find someone who has keyword rapidminer and has friend less than 200 users.

Thanks that nice topic @Thomas_Ott

I found one xml code from @Edin_Klapic which can extract user's detail more than 1 user in one time.(consider from user-id) 

but i have some problem with that xml code when i change query in search twitter box to other word such as "police" it got error in GET USER TWITTER DETAIL box (twitter api error) Could you tell me why?

and if i want to add some conditions such as user must have friends<200 and followers<200 How should i do? i try to add Branch box connectd with get twitter user details box in Loop box but it doesn't work <<Below is original xml>>

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
        <parameter key="connection" value="Twitter Klapic"/>
        <parameter key="query" value="rapidminer"/>
        <parameter key="limit" value="3"/>
      </operator>
      <operator activated="true" class="numerical_to_polynominal" compatibility="7.6.001" expanded="true" height="82" name="Numerical to Polynominal" width="90" x="246" y="34">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="From-User-Id"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="380" y="34">
        <parameter key="attribute" value="From-User-Id"/>
        <process expanded="true">
          <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="648" y="34">
            <parameter key="connection" value="Twitter Klapic"/>
            <parameter key="query_type" value="id"/>
            <parameter key="id" value="%{loop_value}"/>
            <parameter key="user" value="%{loop_value}"/>
          </operator>
          <connect from_op="Get Twitter User Details (2)" from_port="output" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
          <description align="center" color="yellow" colored="false" height="105" resized="false" width="180" x="364" y="83">Type your comment</description>
        </process>
      </operator>
      <operator activated="true" class="append" compatibility="7.6.001" expanded="true" height="82" name="Append" width="90" x="514" y="34"/>
      <connect from_op="Search Twitter" from_port="output" to_op="Numerical to Polynominal" to_port="example set input"/>
      <connect from_op="Numerical to Polynominal" from_port="example set output" to_op="Loop Values" to_port="input 1"/>
      <connect from_op="Loop Values" from_port="output 1" to_op="Append" to_port="example set 1"/>
      <connect from_op="Append" from_port="merged set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Hi @YYTheFenix,

There are 2 problems when you want to search for Twitter User Details and using Search Twitter as base.

1. Twitter changed the possible length of the From-User-Id which is not handled correct internally and therefore sometimes results in false ids which then probably lead to the error you mentioned.
2. You cannot safely use the From-User you get from Search Twitter because this is only the Display name which does not uniquely identify a user. As I mentioned in my previous post you need to use the Twitter User name which is the one with the @ symbol.

We are investigating how and when we are able to fix the bug with the User-Ids but I cannot make any promises.

Best regards,

Edin

Hi @YYTheFenix,

The attached process contains a solution with which you can use both search conditions in a nested Operator.

An easier way would be to get all User details and use Filter Examples afterwards since the number of Friends and Followers is retrieved from Get User Details.

Best regards,

Edin

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
        <parameter key="query" value="rapidminer"/>
        <parameter key="limit" value="1"/>
        <description align="center" color="transparent" colored="false" width="126">get users</description>
      </operator>
      <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="246" y="34">
        <parameter key="attribute" value="From-User"/>
        <parameter key="reuse_results" value="true"/>
        <process expanded="true">
          <operator activated="true" class="social_media:get_twitter_relations" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter Relations" width="90" x="112" y="34">
            <parameter key="user" value="%{loop_value}"/>
            <parameter key="relation" value="friends"/>
          </operator>
          <operator activated="true" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch" width="90" x="246" y="34">
            <parameter key="condition_type" value="max_examples"/>
            <parameter key="condition_value" value="200"/>
            <process expanded="true">
              <operator activated="true" class="social_media:get_twitter_relations" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter Relations (2)" width="90" x="112" y="34">
                <parameter key="user" value="%{loop_value}"/>
              </operator>
              <operator activated="true" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch (2)" width="90" x="246" y="34">
                <parameter key="condition_type" value="max_examples"/>
                <parameter key="condition_value" value="200"/>
                <process expanded="true">
                  <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="246" y="34">
                    <parameter key="user" value="%{loop_value}"/>
                  </operator>
                  <connect from_op="Get Twitter User Details (2)" from_port="output" to_port="input 1"/>
                  <portSpacing port="source_condition" spacing="0"/>
                  <portSpacing port="source_input 1" spacing="0"/>
                  <portSpacing port="sink_input 1" spacing="0"/>
                  <portSpacing port="sink_input 2" spacing="0"/>
                </process>
                <process expanded="true">
                  <portSpacing port="source_condition" spacing="0"/>
                  <portSpacing port="source_input 1" spacing="0"/>
                  <portSpacing port="sink_input 1" spacing="0"/>
                  <portSpacing port="sink_input 2" spacing="0"/>
                </process>
              </operator>
              <connect from_op="Get Twitter Relations (2)" from_port="output" to_op="Branch (2)" to_port="condition"/>
              <connect from_op="Branch (2)" from_port="input 1" to_port="input 1"/>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Get Twitter Relations" from_port="output" to_op="Branch" to_port="condition"/>
          <connect from_op="Branch" from_port="input 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Search Twitter" from_port="output" to_op="Loop Values" to_port="input 1"/>
      <connect from_op="Loop Values" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Thank you @Edin_Klapic for nice code.

I try to combine all xml which i've got from you.

and from your previous answer . So, i choose user-id to use. (Although, it has problem in get twitter user detail box.)

finally i got below xml .(both condition friends and followers)

I found that it still has problem in get twitter user detail box when the input is a large number.

if below xml can be rewrite more easier you can tell me anytime Thanks.

Ps. i'm trying your new xml for better result

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
        <parameter key="connection" value="NewConnection"/>
        <parameter key="query" value="facebook"/>
        <parameter key="limit" value="4"/>
        <description align="center" color="transparent" colored="false" width="126">get users</description>
      </operator>
      <operator activated="true" class="numerical_to_polynominal" compatibility="7.6.001" expanded="true" height="82" name="Numerical to Polynominal" width="90" x="313" y="34">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="From-User-Id"/>
      </operator>
      <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="581" y="34">
        <parameter key="attribute" value="From-User-Id"/>
        <process expanded="true">
          <operator activated="false" class="social_media:get_twitter_relations" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter Relations" width="90" x="179" y="391">
            <parameter key="connection" value="NewConnection"/>
            <parameter key="query_type" value="id"/>
            <parameter key="id" value="%{loop_value}"/>
            <parameter key="user" value="%{loop_value}"/>
          </operator>
          <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="179" y="187">
            <parameter key="connection" value="NewConnection"/>
            <parameter key="query_type" value="id"/>
            <parameter key="id" value="%{loop_value}"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="380" y="187">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attribute" value="Friends"/>
            <parameter key="attributes" value="Followers|Friends"/>
            <parameter key="include_special_attributes" value="true"/>
          </operator>
          <operator activated="true" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch (3)" width="90" x="581" y="187">
            <parameter key="condition_type" value="attribute_value_filter"/>
            <parameter key="condition_value" value="Friends&lt;=200&amp;&amp;Followers&lt;=200"/>
            <process expanded="true">
              <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (3)" width="90" x="179" y="85">
                <parameter key="connection" value="NewConnection"/>
                <parameter key="query_type" value="id"/>
                <parameter key="id" value="%{loop_value}"/>
                <parameter key="user" value="%{loop_value}"/>
              </operator>
              <connect from_op="Get Twitter User Details (3)" from_port="output" to_port="input 1"/>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="false" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (2)" width="90" x="179" y="85">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="Friends"/>
              </operator>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="false" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch" width="90" x="581" y="34">
            <parameter key="condition_type" value="max_attributes"/>
            <parameter key="condition_value" value="200"/>
            <process expanded="true">
              <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details" width="90" x="246" y="34">
                <parameter key="connection" value="NewConnection"/>
                <parameter key="query_type" value="id"/>
                <parameter key="id" value="%{loop_value}"/>
                <parameter key="user" value="%{loop_value}"/>
              </operator>
              <connect from_op="Get Twitter User Details" from_port="output" to_port="input 1"/>
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
            <process expanded="true">
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
              <portSpacing port="sink_input 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="false" class="branch" compatibility="7.6.001" expanded="true" height="82" name="Branch (2)" width="90" x="648" y="442">
            <parameter key="condition_type" value="max_examples"/>
            <parameter key="condition_value" value="200"/>
            <process expanded="true">
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
            </process>
            <process expanded="true">
              <portSpacing port="source_condition" spacing="0"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_input 1" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Get Twitter User Details (2)" from_port="output" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_op="Branch (3)" to_port="condition"/>
          <connect from_op="Branch (3)" from_port="input 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Search Twitter" from_port="output" to_op="Numerical to Polynominal" to_port="example set input"/>
      <connect from_op="Numerical to Polynominal" from_port="example set output" to_op="Loop Values" to_port="input 1"/>
      <connect from_op="Loop Values" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Hi @YYTheFenix,

the following process uses Filter Example to avoid a duplicate execution of Get User Details.

When you Search for Tweets some of the From-Ids have 16 or more digits. The current implementation produces a rounding error which results in a wrong id macro.
Thus for testing purposes I recommend to remove all Examples from your list where the id is longer than 15 digits. 

Best regards,

Edin

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="social_media:search_twitter" compatibility="7.3.000" expanded="true" height="68" name="Search Twitter" width="90" x="112" y="34">
        <parameter key="connection" value="NewConnection"/>
        <parameter key="query" value="facebook"/>
        <parameter key="limit" value="4"/>
        <description align="center" color="transparent" colored="false" width="126">get users</description>
      </operator>
      <operator activated="true" class="numerical_to_polynominal" compatibility="7.6.001" expanded="true" height="82" name="Numerical to Polynominal" width="90" x="313" y="34">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="From-User-Id"/>
      </operator>
      <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="581" y="34">
        <parameter key="attribute" value="From-User-Id"/>
        <parameter key="reuse_results" value="true"/>
        <process expanded="true">
          <operator activated="true" class="social_media:get_twitter_user_details" compatibility="7.3.000" expanded="true" height="68" name="Get Twitter User Details (2)" width="90" x="112" y="34">
            <parameter key="connection" value="NewConnection"/>
            <parameter key="query_type" value="id"/>
            <parameter key="id" value="%{loop_value}"/>
          </operator>
          <operator activated="true" class="filter_examples" compatibility="7.6.001" expanded="true" height="103" name="Filter Examples" width="90" x="246" y="34">
            <list key="filters_list">
              <parameter key="filters_entry_key" value="Friends.le.200"/>
              <parameter key="filters_entry_key" value="Followers.le.200"/>
            </list>
          </operator>
          <connect from_op="Get Twitter User Details (2)" from_port="output" to_op="Filter Examples" to_port="example set input"/>
          <connect from_op="Filter Examples" from_port="example set output" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Search Twitter" from_port="output" to_op="Numerical to Polynominal" to_port="example set input"/>
      <connect from_op="Numerical to Polynominal" from_port="example set output" to_op="Loop Values" to_port="input 1"/>
      <connect from_op="Loop Values" from_port="output 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Hi @YYTheFenix, @Thomas_Ott,

Just as a follow up on the Twitter user id issues.

The latest RapidMiner version 8.1 has an updated Twitter extension with fixed bugs and also new features.

More information can be found here: https://docs.rapidminer.com/latest/studio/releases/changes-8.1.0.html

Best,

Edin