[SOLVED] Deleting text noise from large corpus

Legacy User

Hi

I have a pdf file which contains several thousand pages of emails.  The problem is that each email contains a unique set of noise (unique because it does not repeat).  For example:

x-Mail: hbcFNvIWLDtFlpP.yxyP9bkreUY5ZzdUGPpkOhYIoR

This noise sometimes fills entire pages.  

Can anyone point me in the right direction on how to minimize this noise, or somehow go around it?

Thanks.

MariusHelf

Hi Sal,

if you use the TF-IDF measure, the noise will be ignored (gets value 0), because it appears in only one document and thus does not bring in any advantage for text classification.
Furthermore, the Process Documents operator has parameters to filter out words that appear too seldom (or too often).

Best,
  Marius