Ellbow Plot

Muhammed_Fatih_

Dear community, 

I want to determine or rather apporximate the optimal k for k-means and tried to generate an Ellbow Plot. The generated plot looks like the following which confused me a bit: 


It is not so easy to detect an optimal k due to the reason that more than one ellbows were generated and the graph doesn't look like a conventional Ellbow Plot. Where to set the k? At k=6, k=7 or k=14 where a brakdown is respectively marked with regard to the avg.centroid distance measure. Or would it be better to generate an Ellbow Plot by considering Davies Bouldin insetead of Centroid Distance? 

The data is represented by TF-IDF values. 

Thanks in advance for your hints! 

Best regards! 

lionelderkrikor

Hi @Muhammed_Fatih_,

Intuitively I would say : 
 k = 3 (the first slope failure)
OR
like you k = 7

Regards,

Lionel

Muhammed_Fatih_

Hi @lionelderkrikor, 

thank you for recommendation!

Would you assess the given course of the graph as "normal"? Is it generally possible to have more than one ellows? What are your experiences with regard to this? 

Best regards!

lionelderkrikor

Hi @Muhammed_Fatih_,

In deed, your curve is not usual.
In most of the cases, the  Elbow curve looks like the curves presented in this thread : 

https://community.rapidminer.com/discussion/comment/61654#Comment_61654

Regards,

Lionel

Muhammed_Fatih_

Hi @lionelderkrikor, 

Thank you for the link. I know this thread. My Ellbow curve obviously differs from the presented one in the thread. But what might be potential reasons for getting such a curve?  

Best regards!