Downloading historical financial data since change of Yahoo API
Dear community,
Since Yahoo has changed its API, it is not that easy to access historical financial data anymore by pre-installed operators. Thanks to Thomas who posted a workaround based on the google finance service it is possible to retrieve stock data again.
But unfortunately google finance does not provide CSV download for other object types than stocks. Indices like the Dow Jones, commodities, currencies and bonds are not available.
Is there anyone who knows about free sources of historical data for
- indices
- commodities
- currencies
- bonds
which can be loaded into Rapidminer?
Kind regards
Sachs
Answers
-
While not an elegant solution (i.e. plug right into RapidMiner), have you looked at YQL? https://developer.yahoo.com/yql/console/
They allow you get an SQL dump of data from different parts of their company.
0 -
Quandl has a lot of financial data---some is free, some is paid, so you'll need to search specifically for what you are looking for, but they have many download formats available, and it can be used reliably with RapidMiner (I have done it).
https://www.quandl.com/search?query=
0 -
and I use the API from openexchangerates.com. Take your pick!
Scott
0 -
@sgenzer that's only for Currencies, right?
0 -
yes correct - only currencies. Cost is a freemium model where you get 500(?) calls/month free and pay for more.
0 -
Hi all,
I found that Yahoo still works when a parameter CRUMB is added. This parameter is individual to your session and stored in cookies. So you could still use Yahoo with but with the drawback that you have to start your browser first and call Yahoo page.
Not yet tested with Rapidminer...
Best
Sachs
0 -
Ok, this is obviously not as easy as expected...
1) I opened the Yahoo page
2) There is a little download data button at the right. This is where I copied that link (which includes the session crumb).
3) This link can be opened from the browser several times. But once opened with Rapidminer it got en error
<?xml version="1.0" encoding="UTF-8"?><process version="7.5.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.5.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="open_file" compatibility="7.5.001" expanded="true" height="68" name="Get Online Data as CSV" width="90" x="45" y="34">
<parameter key="resource_type" value="URL"/>
<parameter key="filename" value="http://www.google.com/finance/historical?q=NASDAQ:%{symbol}&amp;ei=LRD-WJmhHIOrmgH_4KToAg&amp;output=csv"/>
<parameter key="url" value="https://query1.finance.yahoo.com/v7/finance/download/SAN?period1=1483250400&period2=1495256400&interval=1d&events=history&crumb=TAInnNixgdJ"/>
</operator>
<operator activated="true" class="read_csv" compatibility="7.5.001" expanded="true" height="68" name="Read CSV" width="90" x="179" y="34">
<parameter key="csv_file" value="C:\Users\Abc\Desktop\bas.csv"/>
<parameter key="column_separators" value=","/>
<parameter key="use_quotes" value="false"/>
<parameter key="date_format" value="dd-MMM-yy"/>
<parameter key="first_row_as_names" value="false"/>
<list key="annotations">
<parameter key="0" value="Name"/>
</list>
<parameter key="encoding" value="UTF-8"/>
<list key="data_set_meta_data_information">
<parameter key="0" value="Date.true.date.id"/>
<parameter key="1" value="Open.false.real.attribute"/>
<parameter key="2" value="High.false.real.attribute"/>
<parameter key="3" value="Low.false.real.attribute"/>
<parameter key="4" value="%{symbol_name}.true.real.attribute"/>
<parameter key="5" value="Volume.false.real.attribute"/>
</list>
</operator>
<connect from_op="Get Online Data as CSV" from_port="file" to_op="Read CSV" to_port="file"/>
<connect from_op="Read CSV" from_port="output" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>This is going to be a looooong night...
Cheers
Sachs
0 -
So far I could not get the historical data download running in Rapidminer. But as an intermediate result I would like to share three interesting links where other people managed to get it running in different programming languages like Java, Python or VBA:
https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working
https://www.mrexcel.com/forum/excel-questions/1005785-yahoo-finance-api-stock-quotes-changed-3.html
One step closer to the goal for sure but no idea which distance is left on the way to go...
Cheers
Sachs
0 -
Well if they can get it to run in Java, you could build a new extension for it.
0 -
Hmm, I tried to get it working in an execute script operator as something to start with. However, sadly I was not able to get it running and I guess I am not the coder with the appropriate knowledge to find the root cause. This is where I have to give in. But hopefully this inspires other members with Java background to continue where I had to stop. Sometimes it's just a colon in the code that messes up everything...
Cheers
Sachs
0 -
Did you ever check to see whether quandl had the data series you were interested in? As I mentioned I am able to use their data with no additional coding required.
0 -
Yes, I tried to but unfortunately the indices data I am looking for is only available in the premium package.
Thanks for your help anyway
Kind regards
Sachs
0 -
The spreadsheet here works with Yahoo Finance to download bulk historical data. The VBA can be viewed and modified. The code is more complicated than the now defunct Yahoo API calls, but it still works
1 -
just to add to this thread...I wrote a KB article two weeks ago about using Alpha Venture API as a good alternative to Yahoo Finance. You can see the article here: http://community.rapidminer.com/t5/RapidMiner-Studio-Knowledge-Base/Real-Time-Financial-Data-via-Alpha-Venture-API-alternative-to/ta-p/41119
Scott
0
Altair Employee