Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Arima & R

Hallo,

I try to replicate an Arima example found at www.rapidminer.com. Here is the XML file:

<?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
<context>
 <input/>
 <output/>
 <macros/>
</context>
<operator activated="true" class="process" compatibility="7.4.000" expanded="true" name="Process">
 <parameter key="logverbosity" value="init"/>
 <parameter key="random_seed" value="2001"/>
 <parameter key="send_mail" value="never"/>
 <parameter key="notification_email" value=""/>
 <parameter key="process_duration_for_mail" value="30"/>
 <parameter key="encoding" value="SYSTEM"/>
 <process expanded="true">
 <operator activated="true" class="quantx1:yahoo_historical_data_extractor" compatibility="1.0.006" expanded="true" height="82" name="Yahoo Historical Stock Data" width="90" x="45" y="120">
 <parameter key="I agree to abide by Yahoo's Terms & Conditions on financial data usage" value="true"/>
 <parameter key="Quick Stock Ticker Data" value="true"/>
 <parameter key="Stock Ticker" value="S&P"/>
 <parameter key="select_fields" value="VOLUME|OPEN|DAY_LOW|DAY_HIGH|CLOSE|ADJUSTED_CLOSE"/>
 <parameter key="date_format" value="yyyy-MM-dd"/>
 <parameter key="date_start" value="2013-01-01"/>
 <parameter key="date_end" value="2015-06-03"/>
 <parameter key="data_frequency" value="DAILY"/>
 <parameter key="Cache Data in Memory" value="false"/>
 </operator>
 <operator activated="true" class="rename" compatibility="7.4.000" expanded="true" height="82" name="Rename" width="90" x="179" y="120">
 <parameter key="old_name" value="S&P_ADJUSTED_CLOSE"/>
 <parameter key="new_name" value="AClose"/>
 <list key="rename_additional_attributes">
 <parameter key="S&P_CLOSE" value="Close"/>
 <parameter key="S&P_DAY_HIGH" value="High"/>
 <parameter key="S&P_DAY_LOW" value="Low"/>
 <parameter key="S&P_OPEN" value="Open"/>
 <parameter key="S&P_VOLUME" value="Volume"/>
 </list>
 </operator>
 <operator activated="true" class="multiply" compatibility="7.4.000" expanded="true" height="124" name="Multiply" width="90" x="313" y="120"/>
 <operator activated="true" class="r_scripting:execute_r" compatibility="7.2.000" expanded="true" height="82" name="Forecasting" width="90" x="715" y="435">
 <parameter key="script" value="### Call this R scripts to get AIC from ARIMA models
rm_main = function(data)
{
 	library(forecast)
 	sp <- data
	sp$Date <- as.Date(sp$Date)
	arima <- arima(ts(sp$Close), order=c(3,1,3))
	print(arima)
	arimaforecast <- forecast.Arima(arima, h=5)
	print(arimaforecast)
 	return(as.data.frame(arimaforecast))
}
"/>
 </operator>
 <operator activated="true" class="optimize_parameters_grid" compatibility="7.4.000" expanded="true" height="103" name="Optimize Parameters (Grid)" width="90" x="514" y="300">
 <list key="parameters">
 <parameter key="Set p.value" value="[0;3;3;linear]"/>
 <parameter key="Set d.value" value="[0.0;2;2;linear]"/>
 <parameter key="Set q.value" value="[0.0;4;4;linear]"/>
 </list>
 <parameter key="error_handling" value="fail on error"/>
 <process expanded="true">
 <operator activated="true" class="set_macro" compatibility="7.4.000" expanded="true" height="76" name="Set p" width="90" x="112" y="30">
 <parameter key="macro" value="p"/>
 <parameter key="value" value="3.0"/>
 </operator>
 <operator activated="true" class="set_macro" compatibility="7.4.000" expanded="true" height="76" name="Set d" width="90" x="112" y="120">
 <parameter key="macro" value="d"/>
 <parameter key="value" value="2.0"/>
 </operator>
 <operator activated="true" class="set_macro" compatibility="7.4.000" expanded="true" height="76" name="Set q" width="90" x="112" y="210">
 <parameter key="macro" value="q"/>
 <parameter key="value" value="4.0"/>
 </operator>
 <operator activated="true" class="r_scripting:execute_r" compatibility="7.2.000" expanded="true" height="112" name="ARIMA" width="90" x="447" y="75">
 <parameter key="script" value="### Call this R scripts to get AIC from ARIMA models
rm_main = function(data)
{
 	sp <- data
	sp$Date <- as.Date(sp$Date)
	arima <- arima(sp$Close, order=c(%{p},%{d},%{q}))
	#print(arima$aic)
 	return(as.data.table(arima$aic))
}
"/>
 <description align="center" color="transparent" colored="false" width="126">Fit ARIMA model in R with diffeferent(p,d,q)</description>
 </operator>
 <operator activated="true" class="extract_performance" compatibility="7.4.000" expanded="true" height="76" name="Performance" width="90" x="581" y="75">
 <parameter key="performance_type" value="data_value"/>
 <parameter key="statistics" value="average"/>
 <parameter key="attribute_name" value="V1"/>
 <parameter key="example_index" value="1"/>
 <parameter key="optimization_direction" value="minimize"/>
 </operator>
 <operator activated="true" class="log" compatibility="7.4.000" expanded="true" height="76" name="Log" width="90" x="715" y="75">
 <list key="log">
 <parameter key="aic" value="operator.Performance.value.performance"/>
 <parameter key="p" value="operator.Set p.parameter.value"/>
 <parameter key="d" value="operator.Set d.parameter.value"/>
 <parameter key="q" value="operator.Set q.parameter.value"/>
 </list>
 <parameter key="sorting_type" value="none"/>
 <parameter key="sorting_k" value="100"/>
 <parameter key="persistent" value="false"/>
 </operator>
 <connect from_port="input 1" to_op="Set p" to_port="through 1"/>
 <connect from_op="Set p" from_port="through 1" to_op="ARIMA" to_port="input 1"/>
 <connect from_op="Set d" from_port="through 1" to_op="ARIMA" to_port="input 2"/>
 <connect from_op="Set q" from_port="through 1" to_op="ARIMA" to_port="input 3"/>
 <connect from_op="ARIMA" from_port="output 1" to_op="Performance" to_port="example set"/>
 <connect from_op="Performance" from_port="performance" to_op="Log" to_port="through 1"/>
 <connect from_op="Log" from_port="through 1" to_port="performance"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="source_input 2" spacing="0"/>
 <portSpacing port="sink_performance" spacing="36"/>
 <portSpacing port="sink_result 1" spacing="0"/>
 </process>
 </operator>
 <connect from_op="Yahoo Historical Stock Data" from_port="example set" to_op="Rename" to_port="example set input"/>
 <connect from_op="Rename" from_port="example set output" to_op="Multiply" to_port="input"/>
 <connect from_op="Multiply" from_port="output 1" to_port="result 1"/>
 <connect from_op="Multiply" from_port="output 2" to_op="Optimize Parameters (Grid)" to_port="input 1"/>
 <connect from_op="Multiply" from_port="output 3" to_op="Forecasting" to_port="input 1"/>
 <connect from_op="Forecasting" from_port="output 1" to_port="result 3"/>
 <connect from_op="Optimize Parameters (Grid)" from_port="performance" to_port="result 2"/>
 <portSpacing port="source_input 1" spacing="0"/>
 <portSpacing port="sink_result 1" spacing="90"/>
 <portSpacing port="sink_result 2" spacing="162"/>
 <portSpacing port="sink_result 3" spacing="126"/>
 <portSpacing port="sink_result 4" spacing="36"/>
 <description align="center" color="yellow" colored="false" height="62" resized="true" width="816" x="305" y="18">Look at Economic Time Series Data (automatically pulled) from public sites and integrate with ARIMA in R extension</description>
 <description align="center" color="yellow" colored="false" height="133" resized="true" width="635" x="490" y="83">Charts for data. Identify any unusual observations for all attributes: day low, high, open, close, adjusted close, volumn</description>
 <description align="center" color="yellow" colored="false" height="177" resized="true" width="626" x="500" y="228">Find the optimized parameter for ARIMA (iterative, and TAKE TIME!! about 1 min) Use R extension for ARIMA models for this demo data, we have ARIMA(3,1,3) as the best fit To chose the best fit model: check Log result, rank by AIC and find the values of p, d, q corresponding to min AIC</description>
 <description align="center" color="yellow" colored="false" height="116" resized="true" width="415" x="713" y="414">Apply ARIMA(3,1,3) for forcasting predict the next 5 days close price </description>
 </process>
</operator>
</process>

Question 1: At Rename, the message "Attribute not found" pops up. I try to modify the parameters but issue remains. Finally, I removed this operator.

Question 2: With operator Rename removed, the message "Rscript not found" shows up at Forescasting (Execute R) operator.

R version 3.5.0 (2018-04-23) is installed.

Can anyone help?

Regards,

Maerkli

Find more posts tagged with

AI Studio

ARIMA

Accepted answers

lionelderkrikor

Hallo @Maerkli,

To answer to Question 1 :

It seems that the Yahoo Historical Stock Data operator is down : When I set a "breapoint after" on this operator

the resulting example set is empty, so the Rename operator can't be executed and raise the error you described.

Regards,

Lionel

lionelderkrikor

Effectively, this extension is dead. My memory is failing...:smileysad: : We talked about that in this thread.

@Maerkli, inside this thread, you can find a link towards the presentation of the Alpha Vantage API (an alternative to Yahoo API).

This API could be useful if you are looking for financial data.

Regards,

Lionel

Thomas_Ott

@lionelderkrikor In search engines we trust. My memory is long gone.

Telcontar120

You can also check out Quandl if you are willing to pay for a premium subscription, they have very complete financial data available in friendly APIs that work well with RapidMiner.

Maerkli

Hallo RapidMiner Community,

Un grand merci for your coopetation.

Maerkli

All comments

lionelderkrikor

Hallo @Maerkli,

To answer to Question 1 :

It seems that the Yahoo Historical Stock Data operator is down : When I set a "breapoint after" on this operator

the resulting example set is empty, so the Rename operator can't be executed and raise the error you described.

Regards,

Lionel

Maerkli

It is true, Lionel. My data file is empty. Thanks for the support.

Maerkli

Thomas_Ott

@Maerkli and @lionelderkrikor there's been a lot of discussion on the community (go search for it) about the old Fin/Econ extension. 1) It's dead, no one is updating it anymore and 2) Yahoo changed it's internals to make it harder to extract stock data.

So you're left with using something else or manually downloading the stock prices via a CSV and loading it in via a Read CSV operator.

lionelderkrikor

Effectively, this extension is dead. My memory is failing...:smileysad: : We talked about that in this thread.

@Maerkli, inside this thread, you can find a link towards the presentation of the Alpha Vantage API (an alternative to Yahoo API).

This API could be useful if you are looking for financial data.

Regards,

Lionel

Thomas_Ott

@lionelderkrikor In search engines we trust. My memory is long gone.

Telcontar120

You can also check out Quandl if you are willing to pay for a premium subscription, they have very complete financial data available in friendly APIs that work well with RapidMiner.

websiteguy

http://investexcel.net/multiple-stock-quote-downloader-for-excel/

this still works its great you can download stock data for say 100 AIM Tickers, and each gets printed to its own csv.

you can merge and find say, all stocks under X with avarage volume Y over N days

to quickly wittle down Stocks with growing interest that are Low cost.

I picked ELan oil and gas with this method @ 0.27 touched 1.37 couple of days ago

Have not incoorporated in rapidminer, would be great to load 100s of ticker data and perform some filtering analysis on them ...

Maerkli

Hallo RapidMiner Community,

Un grand merci for your coopetation.

Maerkli

websiteguy

Hi just a quick update, you have to click the "get stock quote" button twice sometimes to get the cookie it seems..

London stock exchange AIM List Tickers

you can get cryptos like this using the spreadsheet

XRP-USD
BTC-USD
ETH-USD

LSE.csv