Twitter Data mining - Dates? Missing Tweets?

ckm17d

Hi Everyone,

I have used Rapid Miner in 2018 for an exploratory project for class and was able to capture tweets about 6 months in time. I'm trying to use it to mine tweets for something less than a month ago, however, I keep only getting 8 days ago. For an example I would only get 4-12-20 or later.

I also see that the parameters options have changed a bit for searching twitter through rapidminer.

Is there any way I can gather tweets from specific dates? I have already tried the "until" option on the parameters and only got a blank excel spreadsheet (I also do the "write excel" operation).

I know for a fact there have been tweets regarding my search prior to 4-12-20 as I tried this last week.

Any help on solving this is appreciated!

jacobcybulski

In 2019 Twitter changed its API access rules and I mposed new limits in response to the new privacy laws. To overcome the limit on accessing historical data you need to have a paid subscription Historical Power Track. However even that imposes limits on how much data you can get in what time range at a specific frequency. 

ckm17d

Jacobcybulski,

Thank you for your input - that makes sense. Do you happen to know if 1 week is the limit or if there are other limitations I should know about?

jacobcybulski

I think that one week limit on reading historical twits has been around for a while. One way to solve this problem is to set up the twitter harvester and start collecting twits every day into the future. We have done this in Python and kept the process alive as a service in the background. This way you collect your limit worth of free daily twits (on the specific topic) and after some weeks you will have the suitable number of "historical" twits. Do not go over the limit as you can be banned for rules abuse. Of course you can get a subscription and access more historical twits. You can also turn to media analytics companies who can do so for a fee.