[quote user="Supur"]You can now create an empty file called MSGIDS.MER in any mailbox

directory (i.e, a directory where a .CNM file gets created), and this

signals to Mercury that it should suppress duplicate messages in that

mailbox. Duplicate detection is based on a combination of sender and

message-ID, and only the last 200 messages delivered to the mailbox are

actually remembered.

It is a bit unclear what does it mean "only the last 200 messages delivered to the mailbox are

actually remembered"? I have thousands of messages in these folders - for which I would like to stay there (and not to be.. deleted). Probably it means that a new incomming message will be compared for duplicates only with last 200 messages in folder. Which would be good.[/quote]

We have msgids.mer in place for each of our local user mailbox accounts. Works fine as long as you don't use additional "public mailboxes". In that case Mercury has some problems in identifying of duplicates.

As soon as you put an empty msgids.mer into the user's mailbox directory, Mercury starts holding the incoming mails (for that user) additionally in a kind of cache. But this "cache" is limited to 200 mails to prevent an extensive memory using and/or processing duration. This "cache", after built up, will permanently renewed where new mails will be added and old mails will be removed. Whenever a new mail will be retrieved by MercuryD for that user, Mercury is checking its "last-200-mail-cache" whether this new mail was already delivered in past. And in case it's a duplicate, Mercury will drop this new mail.

Several months ago (probably a year) my email server (Mercury/32 v4.74) started delivering duplicate messages to local users, but that applies only to messages from some particular domains. Actually it started duplicating emails sent form some companies and with other companies everything works fine. So, if a message is sent from such company to our users "a" and "b", containg field "To: a@pom.com; b@pom.com", then both users "a" and "b" would receive two identical emails (two for each).

After checking email headers (source) I found the following scenario:

- It is actually the same email (ID and originating timestamp) with field "To:" containing both "a@pom.com; b@pom.com".

- At some point while transported betwen different servers email is duplicated (from now on two instances flow in paralel) and different field "Envelope To:" is added to each of them. So, one email with "Envelope To: a@pom.com" and the other with "Envelope To: b@pom.com".

- Both emails are routed correctly to our external domain mailbox and then downloaded by our local Mercury server as two different messages.

- Then, MercuryD delivers BOTH messages to local users "a" and "b". So we have 2 messages for each user, 4 emails in total. And we should have one message for each user.

Somewhere I red that Mercury assumes that "Envelope-To:" fields can be deleted during the transport between servers and that therefore Mercury does not check these fields. Supposingly it checks "To:" and "Bcc:" fields instead, to be sure about the final recepients. That would explain why an email message with a clear "Envelope-To: a@pom.com" is still delivered to both "a@pom.com" and to "b@pom.com".

This wasn't happening in the past so I believe that recently somebody updated some widely used email-server softvare so therefore we got this new behaviour - email clones being sent separately for each recipient. (Two examples would be "smtp.outgoing.loopia.se [194.9.95.112]" and "vimdzpsp-hrel04.bluewin.ch [195.186.101.109]".)

This policy doesn't sound bad as soon as our email servers do read "Envelope-To:" fields. However, it seems that Mercury doesn't do that - and that's why we have a problem. That's what I think. If my conclusions are wrong, please somebody correct me.

So, does anybody have a fix for that? Can we make Mercury read (an use) "Envelope-To:" fields - when these exists?

A good example: no problem happens when an email is sent from Gmail with

"To: a@pom.com; b@pom.com" because Gmail servers put "Envelope To:

a@pom.com; b@pom.com" in one email and keeps transporting only one email

message till the end.

Thanks,

Nenad

Here is some additional info from Mercury v4.51 manual "The MercuryD POP3 Client Module / Using MercuryD with Domain Mailboxes" from page 80:

Local user: If you enter the name of a local user on your system (one to which Mercury can
delivery directly) then all the mail downloaded from the remote account will be sent to that
local user, irrespective of the address fields in the message. If you leave this field blank, MercuryD
will examine the To, CC and BCC fields of each message looking for addresses it recognizes
as local. When it finds a local address, it will send a copy of the message to that local
user.

Because of the nature of the POP3 protocol used by the MercuryD module to retrieve mail,
there will be occasions when it cannot properly identify the local recipient of a mail message
retrieved from a domain mailbox. This is because the envelope information, used to ferry the
message around the Internet, is removed once the message is placed into the destination mailbox.
Without the envelope information, MercuryD has to rely on the various From, CC, BCC
and Received headers in the message to determine the intended recipient.

So, I think this confirms to my theory / explanation and this should be fixed (new version, a deamon, an AddOn..).

Regards,

Nenad

Do your messages have an "Envelope-to:" header? It's not Mercury getting rid of it if it doesn't.

Read further down the POP configuration page.  There's a section called "Checking for special headers in messages" and is specifically designed for catering for Envelope-to or Delivered-to headers.

Paul, thans for a quick reply. Yes, these messages do have "Envelope-To:" fields, as explained above.

In the meantime I explored more and few minutes ago I just found the same section you mentioned and I learned that I actually do use "Optional special header processing" and that corresponding field is filled with exactly "Envelope-to". (However, I am not sure if it should say "Envelope-To:" instead.)

So, for sure it should be helpfull when "To", "CC:" and "BCC" fields are missing. But, if I would like to ignore existing "To", "CC:" and "BCC" I should tick also "Check only in thease (special) headers".

(I knew I saw this settings few years ago but I wasn't able to find it until now! It is practically hidden in "POP3 Account Information -> Change".)

But, if I tick "Check only in thease (special) headers" and if  "Envelope-To:" is missing in an email - what would happen? Probably delivered to the Default user... and in that case at least I would be able to fix the "misses" and count them. I don't know how many % of emails exchanged today don't have "Envelope-To:" field. I will try and see if it is worthwhile.

Another method to fix this problem with duplicates is to add an empty MSGIDS.MER file in each of local mailboxes (folders). The Mercury/32 v4.73 Release Notes say:

You can now create an empty file called MSGIDS.MER in any mailbox

directory (i.e, a directory where a .CNM file gets created), and this

signals to Mercury that it should suppress duplicate messages in that

mailbox. Duplicate detection is based on a combination of sender and

message-ID, and only the last 200 messages delivered to the mailbox are

actually remembered.

It is a bit unclear what does it mean "only the last 200 messages delivered to the mailbox are

actually remembered"? I have thousands of messages in these folders - for which I would like to stay there (and not to be.. deleted). Probably it means that a new incomming message will be compared for duplicates only with last 200 messages in folder. Which would be good.

I will try each method for some time and publish here what I got.