Parsing email conversations with regular expressions_问答_开发者

I am writing a BI application that needs to extract meta data from email conversation. Namely given an email thread I wish to ext开发者_开发知识库ract all the participants and the structure of the conversation. For example, given a conversation like this:

From: me
To: You
CC: someone

<Body>

From: You
To: Someone
CC: Someone else

<Body>

I want to extract the 'You' sent and email to 'Someone' with a CC to 'SomeOneelse' and all this was forwarded from 'Me' to 'You' with a cc to 'someone'...

My problem is that I need to handle different providers.

Is anybody familiar with a set of regexes somewhere on the net that cover such demands?

Thank you!

I do not understand what you mean with "different providers". Email headers follow a standard. So all you need as regex, is to parse:

To: <contacts>
From: <contact>
Cc: <contacts>

The more "complicated" part is the <contact> bit. For a detailed spec, look at section 3.4 of the mentioned standard.

There are also useful regex examples for email-addresses available. So, taking the most basic example, it could look like this (assuming all is uppercase):

\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b

Now, prepend the To: string:

\bTo: [A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b

Technically, the headers (like To: ) always start at the beginning of the line, so you could do:

^To: [A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b

If you need to extract the email addresses, you need to add a capture group:

^To: ([A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})\b

This will only capture one address though, so you may want to extend from here:

^To: ([A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})(,\s*[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})*\b

This is a very naïve way of matching multiple addresses. You will end up having the first address in one capture group, and all the rest in the second group. A better approach would be to match against this. If it matches, remove the leading To: , split and trim the rest by commas.

As you can see, you are opening a little can of worms here. Parsing emails is not as simple as it may look. Parsing the headers is simple enough (building on the above examples). The message body however is a different kind of beast. Pretty much every email client (Thunderbird, Outlook (express), mutt, ...) handle this slightly differently. Sometimes a new version behaves different to an old one. This largely depends on the client settings, system locale and so on. Does the user send in UTF8, quoted printable, CP1252, ... ? To quote from the standard:

Note: This specification is not intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages. In addition, this document does not specify an encoding of the characters for either transport or storage; that is, it does not specify the number of bits used or how those bits are specifically transferred over the wire or stored on disk.

You may be lucky, in that the sending email-client added a header, specifying the encoding, but they are not enforced to do so (AFAICS).

Next big thing are multipart messages. Which are also somewhat flaky.

My suggestion, is to use a ready-made library to do the parsing. I am certain, that most popular languages have a library available that makes this task much easier.

If you're looking to implement it yourself, here's where to start: http://www.ietf.org/rfc/rfc2822.txt

You are asking the impossible. This is the kind of thing where you need to use a real parser, not just build something with regular expressions.

You can look at the answer here Parse email content from quoted reply where someone is attempting to do something like this with regexes and comments on how tricky it is. Do read the paper on assembling email threads.

And if you want some test data to try this on, you can download the Enron dataset which contains half a million emails (400meg compressed) that were sent/received by Enron people prior to their collapse.