Peter, David,

thanks to your quick responses and davids magick (coding) it's solved: the provided pre-release candidate (only changing MERCURYP a bit) now works nearly flawless. The POP3-Server is a lot faster than it was in 4.52...

Thanks a lot to everyone - keep on that great work! (I think the "big" competitors never can keep up with that flexibility and reaction-time...)

Kindest Regards!

P.S.: Nils: Thanks to David's bugfix there's no more need to work around anymore - thank you anyways...

Using the Win32-Version without Netware we did an Upgrade from 4.52 to 4.61 - every Mail sent via MercuryS and received via MercuryP will be duplicated now.

One Version without the "X-PMFLAGS"-Headerline is delivered and shortly after that another copy of the same Mail appears including the "X-PMFLAGS:"... any suggestions what to do? I dont want to configure a header-filter deleting self-generated Mails.

More Details: Even Mails from Outside (not using MercuryS) arrive twice... First a version without PMail-Flag is received and next time the client checks, the copy from the "invisible" Pegasus-Client is coming in... We don't use PegasusMail anywhere. Trying to stop this behaviour by filtering for the X-PMAIL-Header does not work! This is getting annoying.

Could you post the full headers of a pair of messages & a bit more info about your setup and how your mail is sent & received.?

Here the complete header from an example (SPAM):

1st Mail:

From - Mon Jun 23 11:38:00 2008
X-Account-Key: account3
X-UIDL: 7Z3FKKD.CNM38D7234D
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: from spooler by mercury.mydomain.com (Mercury/32 v4.61); 23 Jun 2008 11:33:06 +0200
X-Envelope-To: <me@mydomain.com>
Return-path: <ydpujhv@braindog.com>
Received: from mail.mydomain.com (xxx.xxx.xxx.xxx.) by mercury.mydomain.com (Mercury/32 v4.61) ID MG000491;
   23 Jun 2008 11:33:02 +0200
X-AuditID: 0a636374-abd38bb000000a52-af-485f6d62cb6f
Received: from [87.48.192.172] (unknown [87.48.192.172])
	by mail.mydomain.com with SMTP id EAC06498003
	for <me@mydomain.com>; Mon, 23 Jun 2008 11:31:14 +0200 (CEST)
Received: from [87.48.192.172] by BRAINDOG.COM.MAIL5.PSMTP.com; Mon, 23 Jun 2008 10:31:14 +0100
Message-ID: <01c8d51c$432778c0$acc03057@ydpujhv>
From: "Warren Samuels" <ydpujhv@braindog.com>
To: <me@mydomain.com>
Subject: [Spam]  My number 63
Date: Mon, 23 Jun 2008 10:31:14 +0100
MIME-Version: 1.0
Content-Type: text/plain;
  charset="Windows-1252"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 4.72.3338.1
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3338.1
X-Brightmail-Tracker: AAAAAwOwgp4FclStBiNaHQ==

2nd one:

From - Mon Jun 23 11:45:28 2008
X-Account-Key: account3
X-UIDL: 7Z3FKKD.CNM38D7241D
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: from spooler by mercury.mydomain.com (Mercury/32 v4.61); 23 Jun 2008 11:33:06 +0200
X-Envelope-To: <me@mydomain.com>
Return-path: <ydpujhv@braindog.com>
Received: from mail.mydomain.com (xxx.xxx.xxx.xxx.) by mercury.mydomain.com (Mercury/32 v4.61) ID MG000491;
   23 Jun 2008 11:33:02 +0200
X-AuditID: 0a636374-abd38bb000000a52-af-485f6d62cb6f
Received: from [87.48.192.172] (unknown [87.48.192.172])
	by mail.mydomain.com with SMTP id EAC06498003
	for <me@mydomain.com>; Mon, 23 Jun 2008 11:31:14 +0200 (CEST)
Received: from [87.48.192.172] by BRAINDOG.COM.MAIL5.PSMTP.com; Mon, 23 Jun 2008 10:31:14 +0100
Message-ID: <01c8d51c$432778c0$acc03057@ydpujhv>
From: "Warren Samuels" <ydpujhv@braindog.com>
To: <me@mydomain.com>
Subject: [Spam]  My number 63
Date: Mon, 23 Jun 2008 10:31:14 +0100
MIME-Version: 1.0
Content-Type: text/plain;
  charset="Windows-1252"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 4.72.3338.1
X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3338.1
X-Brightmail-Tracker: AAAAAwOwgp4FclStBiNaHQ==
X-PMFLAGS: 34078848 0 0 Y7Z3FKKD.CNM  

As you can see: even the timestamps are identical (but the Copy with the added Header is never fetched in the same POP3-Session as the first Mail).

Setup: external Mailgateway (mail.mydomain.com) for SPAM-Tagging and Connection-Checking delivering to Mercury (mercury.mydomain.com) - Clients are MS-Outlook-Express, Outlook, Thunderbirds, etc. 

Yes, that shows identical mails.

It has be received only once and processed by core only once and delivered to the user

[quote]

X-Envelope-To: <me@mydomain.com>

[/quote]

by the core process, not a filter rule.

It would seem that the server is doing it's job correctly and I would be looking at how the client (Thunderbird presumably) is getting the mail and what it is doing after that.

POP3 from mercP or IMAP (or both? might explain your problem).

Turn on session logging for mercP (and/or mercI) to see what the client is doing. 

I wouldn't blame the clients - it doesn't matter if its a Thunderbird, OE or Outlook: they all receive each mail twice: one without X-PMFLAGS-Header and during the second POP-session the second - almost identical one - with an additional X-PMFLAGS-Header. Never heard of any non-Pegasus-Mailclient providing this Header-entry...

Here a log-excerpt of two POP-sessions (MercuryP):

 13:41:57.734: Connection from xxx.xxx.xxx.xxx, Mon Jun 23 13:41:57 2008
13:41:57.734: << +OK <15188734.14738@mydomain.com>, POP3 server ready.<cr><lf>
13:41:58.890: >> CAPA<cr><lf>
13:41:58.890: << +OK Capability list follows<cr><lf>
13:41:58.890: << USER<cr><lf>
13:41:58.890: << TOP<cr><lf>
13:41:58.890: << UIDL<cr><lf>
13:41:58.890: << EXPIRE NEVER<cr><lf>
13:41:58.890: << .<cr><lf>
13:41:58.890: >> USER me<cr><lf>
13:41:58.890: << +OK me is known here.<cr><lf>
13:41:58.890: >> PASS xxx<cr><lf>
13:42:02.437: << +OK Welcome! 14861 messages (375866889 bytes)<cr><lf>
13:42:02.437: >> STAT<cr><lf>
13:42:02.437: << +OK 14861 375866889<cr><lf>
13:42:02.437: >> LIST<cr><lf>
13:42:02.437: << +OK 14861 messages, 375866889 bytes<cr><lf>

and then immediately after that:

13:42:08.359: Connection from xxx.xxx.xxx.xxx, Mon Jun 23 13:42:08 2008
13:42:08.359: << +OK <15199359.10988@mydomain.com>, POP3 server ready.<cr><lf>
13:42:08.515: >> CAPA<cr><lf>
13:42:08.515: << +OK Capability list follows<cr><lf>
13:42:08.515: << USER<cr><lf>
13:42:08.515: << TOP<cr><lf>
13:42:08.515: << UIDL<cr><lf>
13:42:08.515: << EXPIRE NEVER<cr><lf>
13:42:08.515: << .<cr><lf>
13:42:08.515: >> USER me<cr><lf>
13:42:08.515: << +OK me is known here.<cr><lf>
13:42:08.515: >> PASS xxx<cr><lf>
13:42:13.046: << +OK Welcome! 14861 messages (375867075 bytes)<cr><lf>
13:42:13.046: >> STAT<cr><lf>
13:42:13.046: << +OK 14861 375867075<cr><lf>
13:42:13.046: >> LIST<cr><lf>
13:42:13.046: << +OK 14861 messages, 375867075 bytes<cr><lf>

I consider it confirmed that the file holding the message in a users Mailbox
is changed by mercury after the first fetch of the mailclient.
A new header is added to the message and the changed file will be
delivered to the client on the second request.

Seems it has to do with how mercury marks files as "read"?

[quote user="acrowley"] One Version without the "X-PMFLAGS"-Headerline is delivered and shortly after that another copy of the same Mail appears including the "X-PMFLAGS:"... any suggestions what to do? I dont want to configure a header-filter deleting self-generated Mails.[/quote]

MercuryP generates the X-PMFLAGS when you tell the client not to delete the message, but to leave it on the server.

Two things I want you to check right off -
Have you ticked the DST Proof UID, setting in MercuryP?
Have you checked, mark succefully delivered emails as read?

When you change the setting to DST Proof UIDs all clients will download all previously downloaded messages again as a new algorithm is used for this purpose.

[quote user="Peter Strömblad"]

When you change the setting to DST Proof UIDs all clients will download all previously downloaded messages again as a new algorithm is used for this purpose.

[/quote]

[quote user="acrowley"]

+OK 14861 messages, 375867075 bytes

[/quote] 

[+o(] 

No, we don't use the 'Daylight Savings-proof' message IDs (I guess that's what Peter Strömblad means, isn't it?)

Our "Global POP3 Profile Settings":

1. Mark successfully-retrieved mail as 'Read'
2. POP3 deletions survive broken connections

And yes: our Clients are configured to leave messages on the server until deletion.

• The problem occurs since CNM-Files are not marked with a leading exclamation-mark anymore
• The X-PMFLAGS-Header is used now instead of changing the filename
• A new file "MERCURYP.CAC" - representing a "maildrop cache" has been generated in each users mailbox
• The new mechanism lets MercuryP deliver one "unmarked" message and after that the "marked" one
  

I'm going to reactivate version 4.52 now - my users are starting to riot... But that's not that easy: because of the new mechanisms every user will get the messages from the last day (since upgrade) now a third time (cnm-filenames without leading exclamation-mark)! Is there an easier way to prevent that from happening than deleting all those mailbox-contents of the last 24 hours?

[quote user="acrowley"]

Using the Win32-Version without Netware we did an Upgrade from 4.52 to 4.61 - every Mail sent via MercuryS and received via MercuryP will be duplicated now.

[quote user="acrowley"]

I consider it confirmed that the file holding the message in a users Mailbox
is changed by mercury after the first fetch of the mailclient.
A new header is added to the message and the changed file will be
delivered to the client on the second request.

Seems it has to do with how mercury marks files as "read"?

First of all: I appreciate your honest and competent answers very much - that's one of the reasons we decided to productively deploy Mercury in our company (Healthcare - and Yes, we purchased an unlimited license;).

I could reproduce the behaviour in our testenvironment: Win2003 Server, XP SP2 Clients - Mercury 4.52 upgraded to 4.61 - User-Mail-Directories and Mailqueue residing on separate partition (RAID-Mirror) - Clients configured to leave mails on the server until deletion. MercuryP default config (General: Port 110, Timeout 60, Refuse access without PWD - Global: First and last option checked).

Here session 1:

07:28:26.046: Connection from 192.168.xxx.xxx, Wed Jun 25 07:28:26 2008
07:28:26.046: << +OK <165577046.11839@xxx.co.at>, POP3 server ready.<cr><lf>
07:28:26.046: >> CAPA<cr><lf>
07:28:26.046: << +OK Capability list follows<cr><lf>
07:28:26.046: << USER<cr><lf>
07:28:26.046: << TOP<cr><lf>
07:28:26.046: << UIDL<cr><lf>
07:28:26.046: << EXPIRE NEVER<cr><lf>
07:28:26.046: << .<cr><lf>
07:28:26.062: >> USER xxx<cr><lf>
07:28:26.062: << +OK xxx is known here.<cr><lf>
07:28:26.062: >> PASS xxx<cr><lf>
07:28:26.062: << +OK Welcome! 1 messages (650 bytes)<cr><lf>
07:28:26.062: >> STAT<cr><lf>
07:28:26.062: << +OK 1 650<cr><lf>
07:28:26.062: >> LIST<cr><lf>
07:28:26.062: << +OK 1 messages, 650 bytes<cr><lf>
07:28:26.062: << 1 650<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.062: >> UIDL<cr><lf>
07:28:26.062: << +OK unique IDs follow...<cr><lf>
07:28:26.062: << 1 5RCTZ3R.CNM38D902B2<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.062: >> RETR 1<cr><lf>
07:28:26.062: << +OK Here it comes...<cr><lf>
07:28:26.062: << Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200<cr><lf>
07:28:26.062: << X-Envelope-To: <xxx@xxx.co.at><cr><lf>
07:28:26.062: << Return-path: <xxx@xxx.co.at> <cr><lf>
07:28:26.062: << Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;<cr><lf>
07:28:26.062: <<    25 Jun 2008 07:28:18 +0200<cr><lf>
07:28:26.062: << Message-ID: <4861D6CD.1000503@xxx.co.at><cr><lf>
07:28:26.062: << Date: Wed, 25 Jun 2008 07:25:33 +0200<cr><lf>
07:28:26.062: << From: XXX <xxx@xxx.co.at><cr><lf>
07:28:26.062: << User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)<cr><lf>
07:28:26.062: << MIME-Version: 1.0<cr><lf>
07:28:26.062: << To: XXX <xxx@xxx.co.at><cr><lf>
07:28:26.062: << Subject: test<cr><lf>
07:28:26.062: << Content-Type: text/plain; charset=ISO-8859-15; format=flowed<cr><lf>
07:28:26.062: << Content-Transfer-Encoding: 7bit<cr><lf>
07:28:26.062: << <cr><lf>
07:28:26.062: << this<cr><lf>
07:28:26.062: << .<cr><lf>
07:28:26.125: >> QUIT<cr><lf>
07:28:26.125: << +OK xxx.co.at Server closing down.<cr><lf>
07:28:26.125: --- Connection closed normally at Wed Jun 25 07:28:26 2008. ---
07:28:26.125:

and session 2:

07:28:29.546: Connection from 192.168.xxx.xxx, Wed Jun 25 07:28:29 2008
07:28:29.546: << +OK <165580546.6839@xxx.co.at>, POP3 server ready.<cr><lf>
07:28:29.546: >> CAPA<cr><lf>
07:28:29.546: << +OK Capability list follows<cr><lf>
07:28:29.546: << USER<cr><lf>
07:28:29.546: << TOP<cr><lf>
07:28:29.546: << UIDL<cr><lf>
07:28:29.546: << EXPIRE NEVER<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> USER xxx<cr><lf>
07:28:29.546: << +OK xxx is known here.<cr><lf>
07:28:29.546: >> PASS xxx<cr><lf>
07:28:29.546: << +OK Welcome! 1 messages (712 bytes)<cr><lf>
07:28:29.546: >> STAT<cr><lf>
07:28:29.546: << +OK 1 712<cr><lf>
07:28:29.546: >> LIST<cr><lf>
07:28:29.546: << +OK 1 messages, 712 bytes<cr><lf>
07:28:29.546: << 1 712<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> UIDL<cr><lf>
07:28:29.546: << +OK unique IDs follow...<cr><lf>
07:28:29.546: << 1 5RCTZ3R.CNM38D902B5<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.546: >> RETR 1<cr><lf>
07:28:29.546: << +OK Here it comes...<cr><lf>
07:28:29.546: << Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200<cr><lf>
07:28:29.546: << X-Envelope-To: <xxx@xxx.co.at><cr><lf>
07:28:29.546: << Return-path: <xxx@xxx.co.at> <cr><lf>
07:28:29.546: << Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;<cr><lf>
07:28:29.546: <<    25 Jun 2008 07:28:18 +0200<cr><lf>
07:28:29.546: << Message-ID: <4861D6CD.1000503@xxx.co.at><cr><lf>
07:28:29.546: << Date: Wed, 25 Jun 2008 07:25:33 +0200<cr><lf>
07:28:29.546: << From: XXX <xxx@xxx.co.at><cr><lf>
07:28:29.546: << User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)<cr><lf>
07:28:29.546: << MIME-Version: 1.0<cr><lf>
07:28:29.546: << To: XXX <xxx@xxx.co.at><cr><lf>
07:28:29.546: << Subject: test<cr><lf>
07:28:29.546: << Content-Type: text/plain; charset=ISO-8859-15; format=flowed<cr><lf>
07:28:29.546: << Content-Transfer-Encoding: 7bit<cr><lf>
07:28:29.546: << X-PMFLAGS: 34078848 0 0 Y5RCTZ3R.CNM                        <cr><lf>
07:28:29.546: << <cr><lf>
07:28:29.546: << this<cr><lf>
07:28:29.546: << .<cr><lf>
07:28:29.609: >> QUIT<cr><lf>
07:28:29.609: << +OK xxx.co.at Server closing down.<cr><lf>
07:28:29.609: --- Connection closed normally at Wed Jun 25 07:28:29 2008. ---
07:28:29.609:

This is the content of the MERCURYP.CAC (more than this one mail received...):

 # MercuryP maildrop cache, generated Wed, 25 Jun 2008 08:19:59 +0200

0,3,0
1,"Y51MXA2M.CNM","",1880,953747694,2,0,9946344
1,"Y57IR7R0.CNM","",1095,953746787,2,34078848,9946416
1,"Y5RCTZ3R.CNM","",712,953746101,2,34078848,9946405
1,"Y8NDMN3S.CNM","",3029,953747396,2,570950016,9946428
1,"YAUJKGO4.CNM","",1095,953746787,2,34078848,9946413
1,"YLV0KYIP.CNM","",1033,953747706,2,4194304,9946452
1,"YQXROT94.CNM","",1095,953747396,2,34078848,9946436
1,"YUBTC5JR.CNM","",1849478,953746787,2,570949760,9946412
1,"YXCZU9O0.CNM","",1095,953746787,2,34078848,9946420

This the content of the Y5RCTZ3R.CNM:

Received: from spooler by xxx.co.at (Mercury/32 v4.61); 25 Jun 2008 07:28:20 +0200
X-Envelope-To: <xxx@xxx.co.at>
Return-path: <xxx@xxx.co.at>
Received: from [192.168.xxx.xxx] (192.168.xxx.xxx) by xxx.co.at (Mercury/32 v4.61) with ESMTP ID MG001FEA;
   25 Jun 2008 07:28:18 +0200
Message-ID: <4861D6CD.1000503@xxx.co.at>
Date: Wed, 25 Jun 2008 07:25:33 +0200
From: XXX <xxx@xxx.co.at>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: XXX <xxx@xxx.co.at>
Subject: test
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
X-PMFLAGS: 34078848 0 0 Y5RCTZ3R.CNM                        

this

[quote user="acrowley"] I could reproduce the behaviour in our testenvironment: Win2003 Server, XP SP2 Clients - Mercury 4.52 upgraded to 4.61 - User-Mail-Directories and Mailqueue residing on separate partition (RAID-Mirror) - Clients configured to leave mails on the server until deletion. MercuryP default config (General: Port 110, Timeout 60, Refuse access without PWD - Global: First and last option checked). [/quote]

We have this setup too, and many users leaving the mail and no duplicates. I'm too very interested in seeing this solved. Now that you can duplicate the problem, and David can not - I'll contact you "off-forum", so we can solve this together.

OK, there clearly *is* a problem there: the UID for the message changes between sessions, which should not happen. The change is quite subtle (just 3 in what is effectively a date calculation), but significant.

Now that I have a clearer idea of what I'm looking for, I'll spend time today trying to narrow it down. If I can work out what's going on, I'll get a patch up as soon as possible.

Cheers!

-- David --

One way to get around the problem that all mail gets downloaded again would be to make a temporary folder and move the inbox-mail to it and then move them back after changing the DST Proof UIDs.

Works in small environments ;-) 

Doing this corrupts some imap clients, thinking the message has been deleted.