Pegasus Mail & Mercury

Using the Win32-Version without Netware we did an Upgrade from 4.52 to 4.61 - every Mail sent via MercuryS and received via MercuryP will be duplicated now.

One Version without the "X-PMFLAGS"-Headerline is delivered and shortly after that another copy of the same Mail appears including the "X-PMFLAGS:"... any suggestions what to do? I dont want to configure a header-filter deleting self-generated Mails.

More Details: Even Mails from Outside (not using MercuryS) arrive twice... First a version without PMail-Flag is received and next time the client checks, the copy from the "invisible" Pegasus-Client is coming in... We don't use PegasusMail anywhere. Trying to stop this behaviour by filtering for the X-PMAIL-Header does not work! This is getting annoying.

 

Could you post the full headers of a pair of messages & a bit more info about your setup and how your mail is sent & received.?

Here the complete header from an example (SPAM):

1st Mail:

From - Mon Jun 23 11:38:00 2008
X-Account-Key: account3
X-UIDL: 7Z3FKKD.CNM38D7234D
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: from spooler by mercury.mydomain.com (Mercury/32 v4.61); 23 Jun 2008 11:33:06 +0200
X-Envelope-To: <me@mydomain.com>
Return-path: <ydpujhv@braindog.com>
Received: from mail.mydomain.com (xxx.xxx.xxx.xxx.) by mercury.mydomain.com (Mercury/32 v4.61) ID MG000491;
   23 Jun 2008 11:33:02 +0200
X-AuditID: 0a636374-abd38bb000000a52-af-485f6d62cb6f
Received: from [87.48.192.172] (unknown [87.48.192.172])
	by mail.mydomain.com with SMTP id EAC06498003
	for <me@mydomain.com>; Mon, 23 Jun 2008 11:31:14 +0200 (CEST)
Received: from [87.48.192.172] by BRAINDOG.COM.MAIL5.PSMTP.com; Mon, 23 Jun 2008 10:31:14 +0100
Message-ID: <01c8d51c$432778c0$acc03057@ydpujhv>
From: "Warren Samuels" <ydpujhv@braindog.com>
To: <me@mydomain.com>
Subject: [Spam]  My number 63
Date: Mon, 23 Jun 2008 10:31:14 +0100
MIME-Version: 1.0
Content-Type: text/plain;
  charset="Windows-1252"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 4.72.3338.1
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3338.1
X-Brightmail-Tracker: AAAAAwOwgp4FclStBiNaHQ==

2nd one:

From - Mon Jun 23 11:45:28 2008
X-Account-Key: account3
X-UIDL: 7Z3FKKD.CNM38D7241D
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: from spooler by mercury.mydomain.com (Mercury/32 v4.61); 23 Jun 2008 11:33:06 +0200
X-Envelope-To: <me@mydomain.com>
Return-path: <ydpujhv@braindog.com>
Received: from mail.mydomain.com (xxx.xxx.xxx.xxx.) by mercury.mydomain.com (Mercury/32 v4.61) ID MG000491;
   23 Jun 2008 11:33:02 +0200
X-AuditID: 0a636374-abd38bb000000a52-af-485f6d62cb6f
Received: from [87.48.192.172] (unknown [87.48.192.172])
	by mail.mydomain.com with SMTP id EAC06498003
	for <me@mydomain.com>; Mon, 23 Jun 2008 11:31:14 +0200 (CEST)
Received: from [87.48.192.172] by BRAINDOG.COM.MAIL5.PSMTP.com; Mon, 23 Jun 2008 10:31:14 +0100
Message-ID: <01c8d51c$432778c0$acc03057@ydpujhv>
From: "Warren Samuels" <ydpujhv@braindog.com>
To: <me@mydomain.com>
Subject: [Spam]  My number 63
Date: Mon, 23 Jun 2008 10:31:14 +0100
MIME-Version: 1.0
Content-Type: text/plain;
  charset="Windows-1252"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 4.72.3338.1
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3338.1
X-Brightmail-Tracker: AAAAAwOwgp4FclStBiNaHQ==
X-PMFLAGS: 34078848 0 0 Y7Z3FKKD.CNM  

As you can see: even the timestamps are identical (but the Copy with the added Header is never fetched in the same POP3-Session as the first Mail).

Setup: external Mailgateway (mail.mydomain.com) for SPAM-Tagging and Connection-Checking delivering to Mercury (mercury.mydomain.com) - Clients are MS-Outlook-Express, Outlook, Thunderbirds, etc. 

 

Yes, that shows identical mails.

It has be received only once and processed by core only once and delivered to the user

X-Envelope-To: <me@mydomain.com>

by the core process, not a filter rule.

It would seem that the server is doing it's job correctly and I would be looking at how the client (Thunderbird presumably) is getting the mail and what it is doing after that.

POP3 from mercP or IMAP (or both? might explain your problem).

Turn on session logging for mercP (and/or mercI) to see what the client is doing. 

I wouldn't blame the clients - it doesn't matter if its a Thunderbird, OE or Outlook: they all receive each mail twice: one without X-PMFLAGS-Header and during the second POP-session the second - almost identical one - with an additional X-PMFLAGS-Header. Never heard of any non-Pegasus-Mailclient providing this Header-entry...

Here a log-excerpt of two POP-sessions (MercuryP):

 13:41:57.734: Connection from xxx.xxx.xxx.xxx, Mon Jun 23 13:41:57 2008
13:41:57.734: << +OK <15188734.14738@mydomain.com>, POP3 server ready.<cr><lf>
13:41:58.890: >> CAPA<cr><lf>
13:41:58.890: << +OK Capability list follows<cr><lf>
13:41:58.890: << USER<cr><lf>
13:41:58.890: << TOP<cr><lf>
13:41:58.890: << UIDL<cr><lf>
13:41:58.890: << EXPIRE NEVER<cr><lf>
13:41:58.890: << .<cr><lf>
13:41:58.890: >> USER me<cr><lf>
13:41:58.890: << +OK me is known here.<cr><lf>
13:41:58.890: >> PASS xxx<cr><lf>
13:42:02.437: << +OK Welcome! 14861 messages (375866889 bytes)<cr><lf>
13:42:02.437: >> STAT<cr><lf>
13:42:02.437: << +OK 14861 375866889<cr><lf>
13:42:02.437: >> LIST<cr><lf>
13:42:02.437: << +OK 14861 messages, 375866889 bytes<cr><lf>

and then immediately after that:

13:42:08.359: Connection from xxx.xxx.xxx.xxx, Mon Jun 23 13:42:08 2008
13:42:08.359: << +OK <15199359.10988@mydomain.com>, POP3 server ready.<cr><lf>
13:42:08.515: >> CAPA<cr><lf>
13:42:08.515: << +OK Capability list follows<cr><lf>
13:42:08.515: << USER<cr><lf>
13:42:08.515: << TOP<cr><lf>
13:42:08.515: << UIDL<cr><lf>
13:42:08.515: << EXPIRE NEVER<cr><lf>
13:42:08.515: << .<cr><lf>
13:42:08.515: >> USER me<cr><lf>
13:42:08.515: << +OK me is known here.<cr><lf>
13:42:08.515: >> PASS xxx<cr><lf>
13:42:13.046: << +OK Welcome! 14861 messages (375867075 bytes)<cr><lf>
13:42:13.046: >> STAT<cr><lf>
13:42:13.046: << +OK 14861 375867075<cr><lf>
13:42:13.046: >> LIST<cr><lf>
13:42:13.046: << +OK 14861 messages, 375867075 bytes<cr><lf>

Both times 14861 Mails, but first it counts 375866889 bytes and next time 37586707... looks like the POP-Server isn't to blame, either. I will downgrade to the previous version tomorrow if we don't find a solution in time... Who or what is generating these X-PMFLAGS-Headers?!?

I consider it confirmed that the file holding the message in a users Mailbox
is changed by mercury after the first fetch of the mailclient.
A new header is added to the message and the changed file will be
delivered to the client on the second request.

Seems it has to do with how mercury marks files as "read"?

acrowley:

One Version without the "X-PMFLAGS"-Headerline is delivered and shortly after that another copy of the same Mail appears including the "X-PMFLAGS:"... any suggestions what to do? I dont want to configure a header-filter deleting self-generated Mails.

MercuryP generates the X-PMFLAGS when you tell the client not to delete the message, but to leave it on the server.

Two things I want you to check right off -
Have you ticked the DST Proof UID, setting in MercuryP?
Have you checked, mark succefully delivered emails as read?

When you change the setting to DST Proof UIDs all clients will download all previously downloaded messages again as a new algorithm is used for this purpose.

---

Kind regards / Peter

Peter Strömblad:

When you change the setting to DST Proof UIDs all clients will download all previously downloaded messages again as a new algorithm is used for this purpose.

acrowley:

+OK 14861 messages, 375867075 bytes

 

 

No, we don't use the 'Daylight Savings-proof' message IDs (I guess that's what Peter Strömblad means, isn't it?)

Our "Global POP3 Profile Settings":

1. Mark successfully-retrieved mail as 'Read'
2. POP3 deletions survive broken connections

And yes: our Clients are configured to leave messages on the server until deletion.

My investigations showed following:

• The problem occurs since CNM-Files are not marked with a leading exclamation-mark anymore
• The X-PMFLAGS-Header is used now instead of changing the filename
• A new file "MERCURYP.CAC" - representing a "maildrop cache" has been generated in each users mailbox
• The new mechanism lets MercuryP deliver one "unmarked" message and after that the "marked" one
  

I'm going to reactivate version 4.52 now - my users are starting to riot... But that's not that easy: because of the new mechanisms every user will get the messages from the last day (since upgrade) now a third time (cnm-filenames without leading exclamation-mark)! Is there an easier way to prevent that from happening than deleting all those mailbox-contents of the last 24 hours?

acrowley:

Using the Win32-Version without Netware we did an Upgrade from 4.52 to 4.61 - every Mail sent via MercuryS and received via MercuryP will be duplicated now.

I had one beta tester report problems like this late in the beta test process, but only one, and I was unable to duplicate them myself, or to develop any theory for why they might be happening.

It definitely looks as though there is a problem, but I'm currently unable to give you a diagnosis, or any sensible time estimate for a fix. I *am* working on it though.

Cheers!

-- David --

acrowley:

I consider it confirmed that the file holding the message in a users Mailbox
is changed by mercury after the first fetch of the mailclient.
A new header is added to the message and the changed file will be
delivered to the client on the second request.

Seems it has to do with how mercury marks files as "read"?

Yes, this is the general way it works, but it shouldn't be resulting in duplicated downloads, and does not do so here.

My best guess is that the message's UID is somehow changing between sessions, but I can't see any way that can happen. What would be *really* useful for me would be to see the section of a session transcript that shows the UIDs being returned to the client in response to a UIDL command. I'd definitely prefer NOT to see this from an inbox with 14000 messages: if you can produce the problem on demand, it would really help me if you could have an inbox with one message and send me both session transcripts showing the original and the duplicate being downloaded.

Like I say, trying to trace and fix this type of problem when I can't reproduce it is on the same general level as finding a needle in a haystack - any assistance you can give me (in the form of the requested session transcripts) would be potentially enormously helpful.

Cheers!

-- David --

First of all: I appreciate your honest and competent answers very much - that's one of the reasons we decided to productively deploy Mercury in our company (Healthcare - and Yes, we purchased an unlimited license;).

I could reproduce the behaviour in our testenvironment: Win2003 Server, XP SP2 Clients - Mercury 4.52 upgraded to 4.61 - User-Mail-Directories and Mailqueue residing on separate partition (RAID-Mirror) - Clients configured to leave mails on the server until deletion. MercuryP default config (General: Port 110, Timeout 60, Refuse access without PWD - Global: First and last option checked).

Here session 1:

07:28:26.046: Connection from 192.168.xxx.xxx, Wed Jun 25 07:28:26 2008
07:28:26.046: << +OK <165577046.11839@xxx.co.at>, POP3 server ready.<cr><lf>
07:28:26.046: >> CAPA<cr><lf>
07:28:26.046: << +OK Capability list follows<cr><lf>
07:28:26.046: << USER<cr><lf>
07:28:26.046: << TOP<cr><lf>
07:28:26.046: << UIDL<cr><lf>
07:28:26.046: << EXPIRE NEVER<cr><lf>
07:28:26.046: << .<cr><lf>
07:28:26.062: >> USER xxx<cr><lf>
07:28:26.062: << +OK xxx is known here.<cr><lf>
07:28:26.062: >> PASS xxx<cr><lf>
07:28:26.062: << +OK Welcome! 1 messages (650 bytes)<cr><lf>
07:28:26.062: >> STAT<cr><lf>
07:28:26.062: << +OK 1 650<cr><lf>
07:28:26.062: >> LIST<cr><lf>
07:28:26.062: << +OK 1 messages, 650 bytes<cr><lf>
07:28:26.062: << 1 650<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.062: >> UIDL<cr><lf>
07:28:26.062: << +OK unique IDs follow...<cr><lf>
07:28:26.062: << 1 5RCTZ3R.CNM38D902B2<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.062: >> RETR 1<cr><lf>
07:28:26.062: << +OK Here it comes...<cr><lf>
07:28:26.062: << Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200<cr><lf>
07:28:26.062: << X-Envelope-To: <xxx@xxx.co.at><cr><lf>
07:28:26.062: << Return-path: <xxx@xxx.co.at> <cr><lf>
07:28:26.062: << Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;<cr><lf>
07:28:26.062: <<    25 Jun 2008 07:28:18 +0200<cr><lf>
07:28:26.062: << Message-ID: <4861D6CD.1000503@xxx.co.at><cr><lf>
07:28:26.062: << Date: Wed, 25 Jun 2008 07:25:33 +0200<cr><lf>
07:28:26.062: << From: XXX <xxx@xxx.co.at><cr><lf>
07:28:26.062: << User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)<cr><lf>
07:28:26.062: << MIME-Version: 1.0<cr><lf>
07:28:26.062: << To: XXX <xxx@xxx.co.at><cr><lf>
07:28:26.062: << Subject: test<cr><lf>
07:28:26.062: << Content-Type: text/plain; charset=ISO-8859-15; format=flowed<cr><lf>
07:28:26.062: << Content-Transfer-Encoding: 7bit<cr><lf>
07:28:26.062: << <cr><lf>
07:28:26.062: << this<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.125: >> QUIT<cr><lf>
07:28:26.125: << +OK xxx.co.at Server closing down.<cr><lf>
07:28:26.125: --- Connection closed normally at Wed Jun 25 07:28:26 2008. ---
07:28:26.125:

and session 2:

07:28:29.546: Connection from 192.168.xxx.xxx, Wed Jun 25 07:28:29 2008
07:28:29.546: << +OK <165580546.6839@xxx.co.at>, POP3 server ready.<cr><lf>
07:28:29.546: >> CAPA<cr><lf>
07:28:29.546: << +OK Capability list follows<cr><lf>
07:28:29.546: << USER<cr><lf>
07:28:29.546: << TOP<cr><lf>
07:28:29.546: << UIDL<cr><lf>
07:28:29.546: << EXPIRE NEVER<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> USER xxx<cr><lf>
07:28:29.546: << +OK xxx is known here.<cr><lf>
07:28:29.546: >> PASS xxx<cr><lf>
07:28:29.546: << +OK Welcome! 1 messages (712 bytes)<cr><lf>
07:28:29.546: >> STAT<cr><lf>
07:28:29.546: << +OK 1 712<cr><lf>
07:28:29.546: >> LIST<cr><lf>
07:28:29.546: << +OK 1 messages, 712 bytes<cr><lf>
07:28:29.546: << 1 712<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> UIDL<cr><lf>
07:28:29.546: << +OK unique IDs follow...<cr><lf>
07:28:29.546: << 1 5RCTZ3R.CNM38D902B5<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> RETR 1<cr><lf>
07:28:29.546: << +OK Here it comes...<cr><lf>
07:28:29.546: << Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200<cr><lf>
07:28:29.546: << X-Envelope-To: <xxx@xxx.co.at><cr><lf>
07:28:29.546: << Return-path: <xxx@xxx.co.at> <cr><lf>
07:28:29.546: << Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;<cr><lf>
07:28:29.546: <<    25 Jun 2008 07:28:18 +0200<cr><lf>
07:28:29.546: << Message-ID: <4861D6CD.1000503@xxx.co.at><cr><lf>
07:28:29.546: << Date: Wed, 25 Jun 2008 07:25:33 +0200<cr><lf>
07:28:29.546: << From: XXX <xxx@xxx.co.at><cr><lf>
07:28:29.546: << User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)<cr><lf>
07:28:29.546: << MIME-Version: 1.0<cr><lf>
07:28:29.546: << To: XXX <xxx@xxx.co.at><cr><lf>
07:28:29.546: << Subject: test<cr><lf>
07:28:29.546: << Content-Type: text/plain; charset=ISO-8859-15; format=flowed<cr><lf>
07:28:29.546: << Content-Transfer-Encoding: 7bit<cr><lf>
07:28:29.546: << X-PMFLAGS: 34078848 0 0 Y5RCTZ3R.CNM                        <cr><lf>
07:28:29.546: << <cr><lf>
07:28:29.546: << this<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.609: >> QUIT<cr><lf>
07:28:29.609: << +OK xxx.co.at Server closing down.<cr><lf>
07:28:29.609: --- Connection closed normally at Wed Jun 25 07:28:29 2008. ---
07:28:29.609:

This is the content of the MERCURYP.CAC (more than this one mail received...):

 # MercuryP maildrop cache, generated Wed, 25 Jun 2008 08:19:59 +0200

0,3,0
1,"Y51MXA2M.CNM","",1880,953747694,2,0,9946344
1,"Y57IR7R0.CNM","",1095,953746787,2,34078848,9946416
1,"Y5RCTZ3R.CNM","",712,953746101,2,34078848,9946405
1,"Y8NDMN3S.CNM","",3029,953747396,2,570950016,9946428
1,"YAUJKGO4.CNM","",1095,953746787,2,34078848,9946413
1,"YLV0KYIP.CNM","",1033,953747706,2,4194304,9946452
1,"YQXROT94.CNM","",1095,953747396,2,34078848,9946436
1,"YUBTC5JR.CNM","",1849478,953746787,2,570949760,9946412
1,"YXCZU9O0.CNM","",1095,953746787,2,34078848,9946420

This the content of the Y5RCTZ3R.CNM:

Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200
X-Envelope-To: <xxx@xxx.co.at>
Return-path: <xxx@xxx.co.at>
Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;
   25 Jun 2008 07:28:18 +0200
Message-ID: <4861D6CD.1000503@xxx.co.at>
Date: Wed, 25 Jun 2008 07:25:33 +0200
From: XXX <xxx@xxx.co.at>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: XXX <xxx@xxx.co.at>
Subject: test
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
X-PMFLAGS: 34078848 0 0 Y5RCTZ3R.CNM                        

this

And I don't show you my PASSWD.PM-file;) - anything else I can provide to help? Would be great if we could fix this...

acrowley:

I could reproduce the behaviour in our testenvironment: Win2003 Server, XP SP2 Clients - Mercury 4.52 upgraded to 4.61 - User-Mail-Directories and Mailqueue residing on separate partition (RAID-Mirror) - Clients configured to leave mails on the server until deletion. MercuryP default config (General: Port 110, Timeout 60, Refuse access without PWD - Global: First and last option checked).

We have this setup too, and many users leaving the mail and no duplicates. I'm too very interested in seeing this solved. Now that you can duplicate the problem, and David can not - I'll contact you "off-forum", so we can solve this together.

---

Kind regards / Peter

OK, there clearly *is* a problem there: the UID for the message changes between sessions, which should not happen. The change is quite subtle (just 3 in what is effectively a date calculation), but significant.

Now that I have a clearer idea of what I'm looking for, I'll spend time today trying to narrow it down. If I can work out what's going on, I'll get a patch up as soon as possible.

Cheers!

-- David --

One way to get around the problem that all mail gets downloaded again would be to make a temporary folder and move the inbox-mail to it and then move them back after changing the DST Proof UIDs.

Works in small environments ;-)