What is the best number of topics on lda?

elena2020chao

Hello
I want to identify the topic in my data.
I used Lda. I plotted the likelihood for different values for the number of topics. Now, how do I know which topic is better and more optimal?
As I increase the number of topics, the liklihood becomes less. But my analysis is getting harder by increasing the number of threads. Is there a way to know how much liklihood should be?
Thanks

Find more posts tagged with

AI Studio

Accepted answers

All comments

elena2020chao

Does not someone help me?
What is the best value for likelihood in each of these charts?
help me
Thanks a lot

444.JPG

lionelderkrikor

Hi
From my point of view, I would use LDA inside an Optimize operator.

Regards

Lionel

Telcontar120

In my view there is no simple answer to this question. In general, the more topics you allow, the better your performance metrics look. But as you noted, having more topics increases the complexity of your analysis. So you have to make a tradeoff decision. I don't think there is any single way to find the "best" number. @mschmitz is the architext of the LDA extension so I would be interested in his thoughts on this.

MartinLiebig

Hi,

for me finding the optimal number of topics is very similar to k in k-means. there is no easy thing to optimize. The next toolbox version will have "Perplexity" in it, which is the common measure.

Here is a ncie read on the topic: LDA Best Practices

BR,

Martin

elena2020chao

Hello

Thank you both dear professors
Just what did you mention, when does Perplexity come from?
What is its purpose?
Is it better for you to review the data for Alpha? Or set heuristics better?
Is there a criterion for assessing the goodness of Lda with different alpha and beta parameters? How?

Thank you
With respect

MartinLiebig

Hi @elena2020chao,

Perplexity is defined as

exp(-LLH/#tokens)

and is thus a direct dereritive of LLH. It will be present in the next release. It's just common to use this measure over LLH.

For alpha/beta: I would go for Heuristics + Optimize Hyperparameters. It supports an automatic change over the fitting process.

BR,

Martin

elena2020chao

Hello
Thank you very much for your reply
Only this operator is Optimize Hyperparameters
I did not find ...
And what is the basis of liklihood?
Thanks if you answer
With regards

MartinLiebig

Hi,

Optimize Hyperparameters is a setting for the LDA operator. Not an operator.

The LLH is the LLH of the underlying model. See: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation

It can be interpreted like a "goodness of fit" in other models.

BR,

Martin

elena2020chao

Hi, thank you very much:smileyhappy:
I realized
Impatiently waiting for the new version of the program ...
How can you find out what each topic is about? Do I need to understand myself by repeating this topic?
Is it possible to determine the content of each cluster in kmeans by the LDA? I could not do anything ...
thanks again