I think I've figured out the problem - by not honoring expunge requests until a logout has occured, it makes the IMAP implementation incompatible with mobile phones and laptops.  If your IP address gets changed (such as going from one mobile phone tower to another, or carrying a laptop between hotspots without shutting your email program), it leaves a hanging connection.  I guess you could fix this by setting a very low time before it automatically disconnects.

I think I found a serious bug in the new Mercury 4.73, so I'm searching for others to confirm this (or tell me it's just our installation):

If a mail is deleted, and the tcp connection lost without a proper logout, the mail re-appears. In imap terms, the mail lost the "\Deleted" flag.

This does not happen if the mail client is properly closed. This does happen if thunderbird is switched to "offline" mode: Upon returning to "online" mode and refreshing the inbox, recently deleted mails are normal, visible mails again. In older versions of Mercury, breaking the tcp connection caused Mercury to write all changes out to disk, in 4.73 it seems to cause Mercury to discard certain changes.

Interestingly, the "\Seen" flag is not lost, i.e. the mail re-appears as a "read" mail in the mail client.

Can anyone confirm this? It is easy to test: Delete a mail, and switch your mail client to offline mode, then back online again. Look if the just deleted mail re-appears. We use Mercury in NDS mode, but other modes might also be affected.

Greetings

Markus Borst

Various matters around IMAP deletion were discussed during beta testing of version 4.73 last fall, but as far as I can see there were not to be any changes compared to the previous version. David said that the program needs to be able to undo the state changes in the mailbox in the event of an abnormal program termination, so it won't commit the changes until it actually gets an event that necessitates it.

/Rolf

[quote user="Rolf Lindby"]

Various matters around IMAP deletion were discussed during beta testing of version 4.73 last fall, but as far as I can see there were not to be any changes compared to the previous version.

[/quote]

In some ways, it still is the old behaviour: The imap flags are saved in different files and at different points in time. For example, the "\Seen" flag ("mail has been read") was always saved immediately, but the "\Deleted" flag is saved in a different file and at a different time.

Interstingly, version 4.73 was supposed to save status information more often so avoid just such a situation.

[quote user="Rolf Lindby"]

David said that the program needs to be able to undo the state changes in the mailbox in the event of an abnormal program termination, so it won't commit the changes until it actually gets an event that necessitates it.

/Rolf

[/quote]

This is definitely no situation to undo changes: The change was already confirmed (imap server sent "OK UID STORE complete."), and the connection was lost. So all changes should be committed to disk now at the latest.

This really looks like a simple oversight, but I want to make sure, it is not a problem of my configuration. So could please other people verify this:

1. Delete a mail in the inbox

2. Switch your imap client to offline

3. Wait a minute

4. Switch the imap client back to online

5. Refresh the list of the inbox and look for your mail

Please post your finding here and post also your configuration (standalone mode, nds mode etc.).

Greetings

Markus

[quote user="Markus"]Please post your finding here and post also your configuration (standalone mode, nds mode etc.).[/quote]

Worked OK for me.  On refresh the message was gone.

Client - Pegasus Mail 4.61. Standalone mode to remote Mercury 4.73. 

[quote user="PaulW"]

[quote user="Markus"]Please post your finding here and post also your configuration (standalone mode, nds mode etc.).[/quote]

Worked OK for me.  On refresh the message was gone.

Client - Pegasus Mail 4.61. Standalone mode to remote Mercury 4.73. 

[/quote]

O.K. just learned that Pegasus Mail behaves a bit more friendly when going into offline mode, namely it actually logs off instead of just destroying the tcp connection.

So, more specifically, try it with Thunderbird. Other mailclients may behave differently and thus are not a good way to test the behaviour of a broken tcp connection.

Greetings

Markus

[quote user="Markus"]O.K. just learned that Pegasus Mail behaves a bit more friendly when going into offline mode, namely it actually logs off instead of just destroying the tcp connection.[/quote]

Well, the session log doesn't show anything special happens - it certainly doesn't logout, there's just a gap in the session.

[quote]So, more specifically, try it with Thunderbird. Other mailclients may behave differently and thus are not a good way to test the behaviour of a broken tcp connection.[/quote]

'Offline mode' won't always have the same effect as a 'broken connection'.

[quote user="PaulW"]

[quote user="Markus"]O.K. just learned that Pegasus Mail behaves a bit more friendly when going into offline mode, namely it actually logs off instead of just destroying the tcp connection.[/quote]

Well, the session log doesn't show anything special happens - it certainly doesn't logout, there's just a gap in the session.

[/quote]

This looks like Pegasus Mail does not go into any kind of "offline mode" at all. Just not sending further data does not constitute going offline, ending all connections does. That would mean that Pegasus Mail cannot be used to test for this behaviour of Mercury.

[quote user="PaulW"]

[quote]So, more specifically, try it with Thunderbird. Other mailclients may behave differently and thus are not a good way to test the behaviour of a broken tcp connection.[/quote]

'Offline mode' won't always have the same effect as a 'broken connection'.[/quote]

That's true, but switching Thunderbird into offline mode does provoke the issue, i.e. once deleted mails reappear as read mails. Also, from looking at the session logs, Thunderbird does seem to just kill the tcp connection upon switching to offline mode. Thus, Thunderbird can be used as a test for this kind of Mercury behaviour.

Greetings

Markus

Any word from David about a fix for this behavior? It's been quite a while since this issue has been discussed here. Other users have also noticed this and several users have reported that they switched back to 4.72 because of this.

To summarize: A deleted mail should stay marked as "DELETED", but Mercury does not retain this attribute if the tcp connection is lost or if Mercury crashes. This is unacceptable behavior for most installations.

Greetings

Markus

I Agree... +1

best regards

Filippo

This depends on how the server responds to the select flags command, and it looks like you're right. Mercury transmits \Deleted among the Permanentflags.

a06 SELECT INBOX FLAGS
* 146 EXISTS
* 0 RECENT
* FLAGS (\Deleted \Draft \Seen \Answered)
* OK [UIDVALIDITY 1104923238] UID Validity
* OK [UIDNEXT 56176] Predicted next UID
* OK [PERMANENTFLAGS (\Deleted \Draft \Seen \Answered)] Settable message flags
a06 OK [READ-WRITE] SELECT completed.

Do I understand this correctly: Mercury itself declares a delete as "permanent", i.e. as soon as the server acknowledges the change of the "\Deleted" flag with "OK", it should be preserved (preferably on disk)?

Greetings

Markus Borst

yes, that's how I read the rfc. Permanent means permanent, ie whatever happens to server, connection etc, the flag should persist for each message. If Mercury was not to have deleted among the list of settable permanentflags, then the current behavior would be ok.

[/quote] 

O.K., so Mercury is behaving inconsistent. This means, the behavior is a bug. Any chance to get it fixed?

I have tried to reach David in this matter, but have not gotten a reply yet. Maybe people who are in contact with him can raise the issue? Not treating deletions right is a definite no-go.

The new Mercury version is a lot more stable than the last, we really want to use it. But if our users cannot reliably delete mails, they will complain, loudly. Strong language might be involved, i.e.: The current Mercury is not an option.

There seem to be other issues with breaking the tcp connection before doing a proper imap log out: Folder subscriptions seem to get lost. In general, the server must treat a disconnect like a proper logout, i.e. do cleanup etc. afterwards.

Greetings

Markus

P.S.: As for "not to have deleted among the list of settable permanentflags": This would be undesireable in the highest degree: Deletes should be permanent, not matter what. The client can undelete if it wishes to, but it's not for the server to decide when to ignore the users wishes retroactively.

[quote user="Peter Strömblad"]

yes, that's how I read the rfc. Permanent means permanent, ie whatever happens to server, connection etc, the flag should persist for each message. If Mercury was not to have deleted among the list of settable permanentflags, then the current behavior would be ok.

[/quote] 

[quote user="Markus"]

To summarize: A deleted mail should stay marked as "DELETED", but Mercury does not retain this attribute if the tcp connection is lost or if Mercury crashes. This is unacceptable behavior for most installations.

[/quote]

Strange behavior noticed this morning:  I was using Thunderbird to clean out my spam mailbox, which had accumulated a few hundred messages while I was away on vacation.  I deleted about 200 emails, and selected "empty trash" from within Thunderbird.  I stopped to answer the phone, only 2-3 minutes, and during the phone call Thunderbird apparently polled the server for new messages.   When I came back to the computer, every message I had deleted was back in the Inbox, only now marked as "seen."  I don't have time to set up and check logs right now; I'll try to take a look at that this weekend.  But has anyone else seen this? 

A few days ago I updated to Mercury 4.73 (from 4.72) and unfortunately I can confirm that there seems to be something wrong in Mecury's handling of /deleted flags in V4.73.

We use the built in mail client of our CRM-System (CAS GenesisWorld V12) and with Mercury 4.72 IMAP worked without any problems. The CRM-System has some mail filters configured that poll the inboxes of users that are not logged in to the system every 15 minutes or so. For example one of the filters handles spam-tagged mails in the users inboxes, it copies them to a spam folder, marks the original mail in the inbox as /seen and /deleted and does an expunge.

Now with mercury 4.73 the users still find the spam mails in their inboxes after logging in to the CRM-Client. Copying to the spam folder was done, but the original spams in the inboxes lost their /deleted-Flag. The /read-Flag is still set.

I did some session logging and foud out that the CRM-Server and the CRM-clients handle the imap-sessions correctly. They always do a proper logoff. But it's striking to note that the expunge-commands to the inbox fail, mercury says "NO Folder in use by other connections.".

[quote user="loring"]

Strange behavior noticed this morning:  I was using Thunderbird to clean out my spam mailbox, which had accumulated a few hundred messages while I was away on vacation.  I deleted about 200 emails, and selected "empty trash" from within Thunderbird.  I stopped to answer the phone, only 2-3 minutes, and during the phone call Thunderbird apparently polled the server for new messages.   When I came back to the computer, every message I had deleted was back in the Inbox, only now marked as "seen."  I don't have time to set up and check logs right now; I'll try to take a look at that this weekend.  But has anyone else seen this?  [/quote]

We don't use Thunderbird, but I had a similar behavior yesterday. I deleted some messages and a few minutes later they were back in the inbox. I repeated the procedure - same effect. Finally I closed the CRM-Client and used Pegasus to delete the messages...

Edit:

Did some more searching on this forum and found related thread http://community.pmail.com/forums/post/29331.aspx, where Thunderbird seems to cause some problems when mail caching is enabled. But our CRM-clients have mail caching disabled.

I can reproduce the problem with the lost /deleted flag. I turned on session logging on mercury, sent a mail that would trigger a filtering rule of the CRM-Server to a user (user was not logged on to the CRM-client), waited for the CRM-Server to check the inboxes and turned off session logging. Searching the logs for the part with the specific user took a little time (I'm not very familiar with the IMAP protocol). This is the relevant part (IP, Domain, User and Password disguised):

09:31:30.578: Connection from XXX.XXX.XXX.XXX, Wed Sep 14 09:31:30 2011<lf>
09:31:30.609: << * OK DOMAIN IMAP4rev1 Mercury/32 v4.73 server ready.<cr><lf>
09:31:30.609: >> 100 LOGIN "USER" "PASSWORD"<cr><lf>
09:31:30.609: << 100 OK LOGIN completed.<cr><lf>
09:31:30.609: >> 101 SELECT "INBOX"<cr><lf>
09:31:30.609: << * 509 EXISTS<cr><lf>
09:31:30.609: << * 0 RECENT<cr><lf>
09:31:30.609: << * FLAGS (\Deleted \Draft \Seen \Answered)<cr><lf>
09:31:30.609: << * OK [UIDVALIDITY 1240929130] UID Validity<cr><lf>
09:31:30.609: << * OK [UIDNEXT 11406] Predicted next UID<cr><lf>
09:31:30.609: << * OK [PERMANENTFLAGS (\Deleted \Draft \Seen \Answered)] Settable message flags<cr><lf>
09:31:30.609: << 101 OK [READ-WRITE] SELECT completed.<cr><lf>
09:31:30.625: >> 102 NOOP<cr><lf>
09:31:30.625: << 102 OK NOOP complete.<cr><lf>
09:31:31.640: >> 103 UID COPY 11405:11405 "Junk or suspicious mail"<cr><lf>
09:31:31.640: << 103 OK COPY complete.<cr><lf>
09:31:31.640: >> 104 UID STORE 11405:11405 +FLAGS (\Deleted)<cr><lf>
09:31:31.640: << * 509 FETCH (UID 11405 FLAGS (\SEEN \DELETED))<cr><lf>
09:31:31.640: << 104 OK UID STORE complete.<cr><lf>
09:31:31.656: >> 105 EXPUNGE<cr><lf>
09:31:31.656: << 105 NO Folder in use by other connections.<cr><lf>
09:31:31.734: >> 106 LOGOUT<cr><lf>
09:31:31.796: << * BYE IMAP4rev1 server terminating connection.<cr><lf>
09:31:31.796: << 106 OK LOGOUT completed.<cr><lf>
09:31:31.812: --- Connection closed normally at Wed Sep 14 09:31:31 2011. ---
09:31:31.812:

Looking at inbox about half an hour later (in the meantime there was other work to do): deleted message (UID 11405 in the session log) still (or again) in inbox, /deleted flag lost, /seen flag still set, copy of the mail in folder "junk or suspicious mail".

Failed expunge (see log): I guess the CRM-Server establishes at least 2 sessions to one users mailbox at the same time. As far as I know there were no other processes connected to the users mailbox.

Yesterday in the evening I spent some more time in investigating the problem. Both the CRM Server and the clients open more than one IMAP session at the same time and because of this Mercury always responds "NO Folder in use by other connections" to an expunge command. I caugt another couple of mails with lost /deleted flag. If anybody wants to see I can post the logs.

Somewhere in this forum I read that in case of multiple sessions to a mailbox Mercury should do the expunge after the last session has logged out. But as a matter of fact our installation of mercury never executes an expunge when the IMAP-sessions come from CRM Server or client.

When I use Thunderbird to connect to a mailbox (the logs show, that TB opens only one session at a time) all the accrued mail marked as deleted is properly expunged. Also my Thunderbird users didn't report problems with lost /deleted flags. I guess there is a severe bug in mercury in connection with handling multiple sessions to one mailbox. Any suggestions exept for rolling back to version 4.72?

 Greetings, Eric

I'm suffering from this problem also, seeing it with both Thunderbird and Android mail clients. I'm planning on reverting to 4.72 - can I just run the 4.72 install over the top of 4.73 to downgrade it?

Or even better, wait for the 4.74 release that right now is being beta tested to see if that will solve it!

/Rolf 

Sounds promising, I will wait!

I've minimised the fallout by restricting Thunderbird to only one IMAP connection - it's still happening from time to time but nowhere near as frequently.