"Web mining Rapidminer robot_filter"
Hello all,
I don't if it is the right place to post my request. I need to know how you ( a Rapid Miner user who uses it as a web miningusage tool)- when you're importing your web log file- do to set your robot_filter file.
It works when I type in my robot_filter file just [g|G]oogle for example. However I don't really want to do so for a thousand different bots...
So I've tried to find a list which I can paste in my file. On this website http://www.robotstxt.org/db/all.txt they offer the possibility to download the robots list in a .txt format .
But apparently RapidMiner doesn't like it, i got many errors due to bad characters and wrong enclosure...
So what do I have to do in order to have a proper robots list which can be read by rapidminer ?
Thank you in advance,
Antoine
I don't if it is the right place to post my request. I need to know how you ( a Rapid Miner user who uses it as a web miningusage tool)- when you're importing your web log file- do to set your robot_filter file.
It works when I type in my robot_filter file just [g|G]oogle for example. However I don't really want to do so for a thousand different bots...
So I've tried to find a list which I can paste in my file. On this website http://www.robotstxt.org/db/all.txt they offer the possibility to download the robots list in a .txt format .
But apparently RapidMiner doesn't like it, i got many errors due to bad characters and wrong enclosure...
So what do I have to do in order to have a proper robots list which can be read by rapidminer ?
Thank you in advance,
Antoine