"Size of the given column does not match previously defined number of rows" Error

nagai

Dear Community,

I'm trying to use Facebook Prophet in Python Learner operator. However, I am getting the error "Process failed: operator cannot be executed (Size of the given column does not match previously defined number of rows)" during executing the rm_apply function of the Python Learner operator. 

Please tell me the meaning of this error and how to avoid it. 

Best regards, 

MichaelKnopf

Hello,

it is difficult to tell the exact cause of the error without the script and input data. But I suspect the rm_apply function returns a DataFrame not matching the size of the input data.

The Python Learner is designed to be used with classification and regression models which add one ore more columns to the data set, e.g., the predicted class and its probability. Thus, the requirement that the returned columns are of the same size as the input data.

Forecasting models usually output additional rows and do not add columns. As a consequence, it is not trivial to implement them using the Python Learner. In particular, the resulting model will not be compatible with time series specific operators such as the sliding window validation.

However, it is in principle possible: you could return the forecast as a new column representing the prediction of the value x days from now, where now refers to the timestamp of the current row.

MichaelKnopf

Hello,

it is difficult to tell the exact cause of the error without the script and input data. But I suspect the rm_apply function returns a DataFrame not matching the size of the input data.

The Python Learner is designed to be used with classification and regression models which add one ore more columns to the data set, e.g., the predicted class and its probability. Thus, the requirement that the returned columns are of the same size as the input data.

Forecasting models usually output additional rows and do not add columns. As a consequence, it is not trivial to implement them using the Python Learner. In particular, the resulting model will not be compatible with time series specific operators such as the sliding window validation.

However, it is in principle possible: you could return the forecast as a new column representing the prediction of the value x days from now, where now refers to the timestamp of the current row.

nagai

Hello,

It seems that the number of rows in a DataFrame returned by the rm_apply function and the number of rows in the input data must be the same.

I was able to add a column of Prophet's predicted value "yhat" to a DataFrame returned by the rm_apply function.

However, I couldn't add other "trend", "yearly", "weekly", etc. as columns.

Your answer was very helpful.

Thank you very much.