preprocessing: remove email signature

Joos

Hi
I am trying to apply LDA to emails. I have the mails in an excel file. My model works, but I have to find a way to remove the email signature. Does anyone have experience?

Thanks

rfuentealba

Hi @Joos,

The problem is that messages have two parts: a header with a number of directions and a body containing text. You want to parse the "body".

The first answer can be demonstrated with tje following two "e-mails":

<b>From: Rodrigo <rodrigo@example.com>
To: Joos <joos@example.org></b>

Hello Joos,

This is an example message.

--
Rodrigo Fuentealba
Chile

<b>From: Joos <joos@example.org>
To: Rodrigo <rodrigo@example.com></b>

Hi Rodrigo,

I see. Even though the footers are different, there is something many users do, which is putting a -- before their signature. Not everyone follows this but a big part do.

--
Joos
Netherlands

Now, finding the last -- wouldn't work on all e-mails, because it is a convention but not a fact that people use -- on their e-mails to separate the signature from the rest. Let's think of another solution Let's say you have 1000 e-mails from me. If 300 of these e-mails end up with the classic "Sent from my iPhone" as the last line, you can identify that parameter and delete the e-mail. But what about all the e-mails that I sent with my own signature? You may be able to identify that 600 e-mails from rodrigo@example.com always have the "Rodrigo Fuentealba / Chile" signature, thus it can be removed.

Answering your other questions:

• Yes, you can use Python code inside RapidMiner with the Python Scripting extension. However, the mail parser extension probably won't help you, this is a natural language processing (or pattern recognition) issue.
• Yes, you can do the pattern recognition in Dutch. I don't speak it, but have done similar stuff in German.

All the best,

Rodrigo.

rfuentealba

Hello @Joos,

I can only recommend two ways. The first one is to remove everything from the last -- signs together to the end. Or, if you have the recipient of the e-mail, trim the message and check the last line on each e-mail until no last lines are the same.

Both aren't battle tested, and involve some processing that I wouldn't have done with RapidMiner but much earlier, while retrieving the e-mails, so you are better of trying your luck with loading your data with Python to remove the e-mail signatures, I'm afraid.

All the best,

Rodrigo.

Joos

Thank you for your answer. Not sure I understand your first option, because all the footers are different, so I do not know how to recognize them. I did find python code on github (mailparser). Is it possible to include this as python script in the code? I can include it in my loop going over the different mails and pass it on the python parser as a document? Probably the python code would need adjustment to get this working? Moreover, it would have to do the parsing in Dutch? Do you have experience in this? Your 

rfuentealba

Hi @Joos,

The problem is that messages have two parts: a header with a number of directions and a body containing text. You want to parse the "body".

The first answer can be demonstrated with tje following two "e-mails":

<b>From: Rodrigo <rodrigo@example.com>
To: Joos <joos@example.org></b>

Hello Joos,

This is an example message.

--
Rodrigo Fuentealba
Chile

<b>From: Joos <joos@example.org>
To: Rodrigo <rodrigo@example.com></b>

Hi Rodrigo,

I see. Even though the footers are different, there is something many users do, which is putting a -- before their signature. Not everyone follows this but a big part do.

--
Joos
Netherlands

Now, finding the last -- wouldn't work on all e-mails, because it is a convention but not a fact that people use -- on their e-mails to separate the signature from the rest. Let's think of another solution Let's say you have 1000 e-mails from me. If 300 of these e-mails end up with the classic "Sent from my iPhone" as the last line, you can identify that parameter and delete the e-mail. But what about all the e-mails that I sent with my own signature? You may be able to identify that 600 e-mails from rodrigo@example.com always have the "Rodrigo Fuentealba / Chile" signature, thus it can be removed.

Answering your other questions:

• Yes, you can use Python code inside RapidMiner with the Python Scripting extension. However, the mail parser extension probably won't help you, this is a natural language processing (or pattern recognition) issue.
• Yes, you can do the pattern recognition in Dutch. I don't speak it, but have done similar stuff in German.

All the best,

Rodrigo.

Joos

Thanks Rodrigo...I kind of fixed the issue in excel with formulas