How to improve our accuracy?

szwanen

Hi guys,

For an assignment we are trying to predict churn for a mobile phone company.
Attached you are able to find the dataset and current processes. We have tried a lot of things, but accuracy seems to be stuck around 70%. Is there anything we are missing, or can we improve accuracy in any way?

Next to that any more tips for predictive models? For example, would clustering make sense in order to better explain the model?

Cheers!

varunm1

Hello @szwanen

 can we improve accuracy in any way?

I see you didn't use "optimize hyperparameters" to search for the best combination of hyperparameters good for your data. As you are using the GBT model, try optimizing on "number of trees, learning rate".

Next to that any more tips for predictive models?

I am not sure if you tried other models, you also try simpler models like Decision trees and complex neural nets using optimize parameter option.

For example, would clustering make sense in order to better explain the model?

GBT is complex, most of the explanations are based on features, you can see global importance of feature provided by GBT, you can also use "Explain prediction" to explain local (each prediction) and global features importances. Clustering is also a good way to understand data.

lionelderkrikor

Hello @szwanen ,

First, I encourage to follow Varun's advices.
Secondly, I played with your data with AutoModel. Here are the results with Feature Selection AND Feature Generation enabled :

  
In deed it will be difficult to do better than 70% accuracy.
But by doing feature selection ,for an equivalent accuracy, you can significantly  reduce the complexity of your model. For example here I'm obtaining an accuracy of 68,6 % with only 7 attributes  (on a total of 11 initial attributes).

In my case, I have set the maximum duration for processing duration to 60 min.
To do better than me, you could relaunch AutoModel with your data by setting the max duration of processing to a significantly higher
value (for example 10hours) and launch RapidMiner during a whole night....

Hope this helps,

Regards,

Lionel   

varunm1

Hello @szwanen

 can we improve accuracy in any way?

I see you didn't use "optimize hyperparameters" to search for the best combination of hyperparameters good for your data. As you are using the GBT model, try optimizing on "number of trees, learning rate".

Next to that any more tips for predictive models?

I am not sure if you tried other models, you also try simpler models like Decision trees and complex neural nets using optimize parameter option.

For example, would clustering make sense in order to better explain the model?

GBT is complex, most of the explanations are based on features, you can see global importance of feature provided by GBT, you can also use "Explain prediction" to explain local (each prediction) and global features importances. Clustering is also a good way to understand data.

lionelderkrikor

Hello @szwanen ,

First, I encourage to follow Varun's advices.
Secondly, I played with your data with AutoModel. Here are the results with Feature Selection AND Feature Generation enabled :

  
In deed it will be difficult to do better than 70% accuracy.
But by doing feature selection ,for an equivalent accuracy, you can significantly  reduce the complexity of your model. For example here I'm obtaining an accuracy of 68,6 % with only 7 attributes  (on a total of 11 initial attributes).

In my case, I have set the maximum duration for processing duration to 60 min.
To do better than me, you could relaunch AutoModel with your data by setting the max duration of processing to a significantly higher
value (for example 10hours) and launch RapidMiner during a whole night....

Hope this helps,

Regards,

Lionel